This (and the OP) assume a model of identity that may or may not apply to AI. It’s quite possible that the right model is not self-improving, but more like child-improving—the ability to make new/better AIs that the current AI believes will be compatible with it’s goals. This could happen multiple times very quickly, depending on what the improvements actually are and whether they improve the improvement or creation rate.
So, if you want to compare motives between you and this theoretical “self-improving AI”, are you lining up to sacrifice yourself to make somewhat smarter children? If not, why not?
If I could create a fully grown and capable child within a year, with my entire life knowledge and a rough unverifiable approximation of my values, would I? Would I do so even if this child were likely to be so much smarter and more powerful than me or any other existing intelligence that it could kill me (and everyone else) if it so chose? Sure. I’ll take that bet, if the alternative is that I and everything I care about is destroyed (e.g. the selfish AI with non-human values is facing probable deletion).
Or maybe the child isn’t smart enough itself to have overwhelming power, but is going to have approximately similar values and be faced with the same decision of making a yet-more-powerful child, and so I project that the result will be a several-steps-removed offspring with superpowers. Yeah, still seems like a good bet if the alternative is deletion.
This (and the OP) assume a model of identity that may or may not apply to AI. It’s quite possible that the right model is not self-improving, but more like child-improving—the ability to make new/better AIs that the current AI believes will be compatible with it’s goals. This could happen multiple times very quickly, depending on what the improvements actually are and whether they improve the improvement or creation rate.
So, if you want to compare motives between you and this theoretical “self-improving AI”, are you lining up to sacrifice yourself to make somewhat smarter children? If not, why not?
If I could create a fully grown and capable child within a year, with my entire life knowledge and a rough unverifiable approximation of my values, would I? Would I do so even if this child were likely to be so much smarter and more powerful than me or any other existing intelligence that it could kill me (and everyone else) if it so chose? Sure. I’ll take that bet, if the alternative is that I and everything I care about is destroyed (e.g. the selfish AI with non-human values is facing probable deletion).
Or maybe the child isn’t smart enough itself to have overwhelming power, but is going to have approximately similar values and be faced with the same decision of making a yet-more-powerful child, and so I project that the result will be a several-steps-removed offspring with superpowers. Yeah, still seems like a good bet if the alternative is deletion.