(Unless you mind being simulated, in which case at least you’ll never know.)
If I paid you to extend the lives of cute puppies, and instead you bought video games with that money but still sent me very convincing pictures of cute puppies that I had “saved”, then you have still screwed me over. I wasn’t paying for the experience of feeling that I had saved cute puppies—I was paying for an increase in the probability of a world-state in which the cute puppies actually lived longer.
Tricking me into thinking that the utility of a world state that I inhabit is higher than it actually is isn’t Friendly at all.
I, on the other hand, (suspect) I don’t mind being simulated and living in a virtual environment. So can I get my MFAI before attempts to build true FAI kill the rest of you?
Possible compromise: Have there be some way of revealing the truth to those who want the truth badly enough.
I am extremely hostile to this idea. (I’m not dead-set against forced uploading or even hidden forced uploading (though I would prefer to keep the interface to the base universe open) but I cannot tolerate this end of interpersonal entanglement.
If I paid you to extend the lives of cute puppies, and instead you bought video games with that money but still sent me very convincing pictures of cute puppies that I had “saved”, then you have still screwed me over. I wasn’t paying for the experience of feeling that I had saved cute puppies—I was paying for an increase in the probability of a world-state in which the cute puppies actually lived longer.
Tricking me into thinking that the utility of a world state that I inhabit is higher than it actually is isn’t Friendly at all.
I, on the other hand, (suspect) I don’t mind being simulated and living in a virtual environment. So can I get my MFAI before attempts to build true FAI kill the rest of you?
Possible compromise: Have there be some way of revealing the truth to those who want the truth badly enough.
I am extremely hostile to this idea. (I’m not dead-set against forced uploading or even hidden forced uploading (though I would prefer to keep the interface to the base universe open) but I cannot tolerate this end of interpersonal entanglement.