Suppose your AI’s goal was “preserve myself”. Ignoring any philosophical issues about denotation, here self-preservation is worthwhile even if the goal changed. If the AI, by changing itself into a paperclip maximizer, could maximize its chances of survival (say because of the threat of other Clippies) then it would do so. Because self-preservation is a instrumentally convergent goal, it would probably survive for quite a long time as a paperclipper—maybe much longer than as an enemy of Clippy.
To be the same and to have the same goals are two distinct, but equally possible kinds of sameness.
Most humans seem to care much more about the former (survival) then the later (that their goals be sustained in the universe)
Citing Woody Allen: “I don’t want to achieve immortality through my work. I want to achieve it through not dying.”
We do have distinct reasons to think machine intelligences would like to preserve their goals, and that for them, perhaps identity would feel more entangled with goals, however those reasons are far from unequivocal.
Note that self-preservation is really a sub-class of goal-content integrity, and is worthless without it.
This is a total nit pick, but:
Suppose your AI’s goal was “preserve myself”. Ignoring any philosophical issues about denotation, here self-preservation is worthwhile even if the goal changed. If the AI, by changing itself into a paperclip maximizer, could maximize its chances of survival (say because of the threat of other Clippies) then it would do so. Because self-preservation is a instrumentally convergent goal, it would probably survive for quite a long time as a paperclipper—maybe much longer than as an enemy of Clippy.
I take this to be false.
To be the same and to have the same goals are two distinct, but equally possible kinds of sameness.
Most humans seem to care much more about the former (survival) then the later (that their goals be sustained in the universe)
Citing Woody Allen: “I don’t want to achieve immortality through my work. I want to achieve it through not dying.”
We do have distinct reasons to think machine intelligences would like to preserve their goals, and that for them, perhaps identity would feel more entangled with goals, however those reasons are far from unequivocal.