Don’t many/most people here want there to be posthumans, which may well cross the species-barrier? I don’t think there is an “essence of humanity” that carries over from humans to posthumans by virtue of descendance, so that case seems somewhat analogous to the wireheading AIs case already. And whether the AI would do wireheading or keep intact a preference architecture depends on what we/it values. If we do value complex preferences, and if we want to have many beings in the world that have them mostly fulfilled, I’d assume there would be more awesome or more effective ways of design than current humans However, if this view implies that killing is bad because it violates preferences, then replacement would, to some extent, be a bad thing and the AI might not do it.
Don’t many/most people here want there to be posthumans, which may well cross the species-barrier? I don’t think there is an “essence of humanity” that carries over from humans to posthumans by virtue of descendance, so that case seems somewhat analogous to the wireheading AIs case already. And whether the AI would do wireheading or keep intact a preference architecture depends on what we/it values. If we do value complex preferences, and if we want to have many beings in the world that have them mostly fulfilled, I’d assume there would be more awesome or more effective ways of design than current humans However, if this view implies that killing is bad because it violates preferences, then replacement would, to some extent, be a bad thing and the AI might not do it.