To have the “pure rational” version serve the human one? Have it achieve goals that aren’t defined by ownership (e.g., ‘reduce total suffering’), because it’s better at those things? This sounds reasonable and I don’t see why we shouldn’t do this—assuming it’s easier or otherwise preferable to creating a GAI from scratch to achieve these goals.
Being modified like this is much the same as death. I would lose personhood and the modified remains would not be serving my original goals, but those of the winners in the lottery.
So this is equivalent to asking: would I enter a lottery with 0.01% chance of instant death and a prize for everything else? I might, depending on the prize. If the prize is service by the modified individuals, and they aren’t based on myself but on other people (so they’re not very well suited to advancing my personal goals), and I have to timeshare them with all the other winners (every winner receives 0.01 / 99.99 ~~ 10^-4 % service time) - that doesn’t seem worthwhile.
The transformation is at least partially self-undermining, if one of your values is conscious experience.
Can I get xeroxed and have the transformation performed on one randomly selected version of me before it wakes up?
To have the “pure rational” version serve the human one? Have it achieve goals that aren’t defined by ownership (e.g., ‘reduce total suffering’), because it’s better at those things? This sounds reasonable and I don’t see why we shouldn’t do this—assuming it’s easier or otherwise preferable to creating a GAI from scratch to achieve these goals.
Would you enter a lottery where a small portion (0.01%) of the entrants are selected as losers at random, and transformed without being copied?
Being modified like this is much the same as death. I would lose personhood and the modified remains would not be serving my original goals, but those of the winners in the lottery.
So this is equivalent to asking: would I enter a lottery with 0.01% chance of instant death and a prize for everything else? I might, depending on the prize. If the prize is service by the modified individuals, and they aren’t based on myself but on other people (so they’re not very well suited to advancing my personal goals), and I have to timeshare them with all the other winners (every winner receives 0.01 / 99.99 ~~ 10^-4 % service time) - that doesn’t seem worthwhile.
Modified question in place.
Modified answer in place...