It seems like most other commenters so far don’t share my opinion, but I view the above scenario as basically equivalent to wireheading, and consequently see it as only very slightly better than the destruction of all earth-originating intelligence (assuming the AI doesn’t do anything else interesting). “Affecting-the-real-world” is actually the one value I would not want to trade off (well, obviously, I’d still trade it off, but only at a prohibitively expensive rate).
I’m much more open to trading off other things, however. For example, if we could get Failed Utopia #4-2 much more easily than the successful utopia, I’d say we should go for it. What specific values are the best to throw away in the pursuit of getting something workable isn’t really clear, though. While I don’t agree that if we lose one, we lose them all, I’m also not sure that anything in particular can be meaningfully isolated.
Perhaps the best (meta-)value that we could trade off is “optimality”—we should consider that if we see a way to design something stable that’s clearly not the best we can do, we should nonetheless go with it if it’s considerably easier than better options. For example, if you see a way to specify a particular pretty good future and have the AI build that without going into some failure mode, it might be better to just use that future instead of trying to have the AI design the best possible future.
From an instrumental viewpoint, I hope you plan to figure out how to make everyone sitting around on a higher level credibly precommit to not messing with the power plug on your experience machine, otherwise it probably won’t last very long. (Other than that, I see no problems with us not sharing some terminal values.)
I can’t do much about scenarios in which it is optimal to kill humans. We’re probably all screwed in such a case. “Kill some humans according to these criteria” is a much smaller target than vast swathes of futures that simply kill us all.
figure out how to make everyone sitting around on a higher level credibly precommit to not messing with the power plug
That’s MFAI’s job. Living on the “highest level” also has the same problem, you have to protect your region of the universe from anything that could “de-optimize” it, and FAI will (attempt to) make sure this doesn’t happen.
“Affecting-the-real-world” is actually the one value I would not want to trade off
How are you defining “real world”? Which traits separate something real and meaningful from something you don’t value? Is it the simulation? The separation from other beings? The possibility that the AI is deceiving you? Something I’m missing entirely?
(Personally I’m not at all bothered by the simulation, moderately bothered by the separation, and unsure how I feel about the deception.)
The salient difference for me is that the real one has maximal impact. Many actions in it can affect anyone in a lower world, but not vice versa. I’d like my decisions to have as much effect as possible as a general principle, I think (not the only principle, but the one that dominates in this scenario).
This is pretty much why I’m comfortable calling being in the highest possible world a terminal value—it’s really not about the simulation (wouldn’t be bothered if it turned out current world is a simulation, although I’d like to go higher), not especially about separation, and only slightly about deception (certainly losing all impactfulness becomes more irreversible if the AI is lying).
My own view: -Separation is very, very bad. I’d be somewhat OK with reality becoming subjective but with some kind of interface between people but this whole scenario as stated is approaching the collapse civilization so we can’t FOOM level. My personal reaction to seeing this described as better than status quo was somewhat similar to playing Mass Effect and listening to Reapers talk about ‘salvation through destruction’ and ascension in the form of perpetually genocidal robo-squids. I mean seriously? ‘All your friends are actually p-zombies?’ Are you kidding me? /rant
Living in highest possible world for me is not a value but having access or interface or something to the highest possible world is. (Not particularly high.) But knowing the truth definitely is and having my friends actually be people also is. Would prefer just being separated from friends and given Verthandi (I.e. sentient people, but optimized) like in Failed Utopia 2-4.
It seems like most other commenters so far don’t share my opinion, but I view the above scenario as basically equivalent to wireheading, and consequently see it as only very slightly better than the destruction of all earth-originating intelligence (assuming the AI doesn’t do anything else interesting). “Affecting-the-real-world” is actually the one value I would not want to trade off (well, obviously, I’d still trade it off, but only at a prohibitively expensive rate).
I’m much more open to trading off other things, however. For example, if we could get Failed Utopia #4-2 much more easily than the successful utopia, I’d say we should go for it. What specific values are the best to throw away in the pursuit of getting something workable isn’t really clear, though. While I don’t agree that if we lose one, we lose them all, I’m also not sure that anything in particular can be meaningfully isolated.
Perhaps the best (meta-)value that we could trade off is “optimality”—we should consider that if we see a way to design something stable that’s clearly not the best we can do, we should nonetheless go with it if it’s considerably easier than better options. For example, if you see a way to specify a particular pretty good future and have the AI build that without going into some failure mode, it might be better to just use that future instead of trying to have the AI design the best possible future.
If believing you inhabit the highest level floats your boat be my guest, just don’t mess with the power plug on my experience machine.
From an instrumental viewpoint, I hope you plan to figure out how to make everyone sitting around on a higher level credibly precommit to not messing with the power plug on your experience machine, otherwise it probably won’t last very long. (Other than that, I see no problems with us not sharing some terminal values.)
I just have to ensure that the inequality (Amount of damage I cause if outside my experience machine>Cost of running my experience machine) holds.
Translating that back into English, I get “unplug me from the Matrix and I’ll do my best to help Skynet kill you all”.
Also that killing you outright isn’t optimal.
I can’t do much about scenarios in which it is optimal to kill humans. We’re probably all screwed in such a case. “Kill some humans according to these criteria” is a much smaller target than vast swathes of futures that simply kill us all.
That’s MFAI’s job. Living on the “highest level” also has the same problem, you have to protect your region of the universe from anything that could “de-optimize” it, and FAI will (attempt to) make sure this doesn’t happen.
How are you defining “real world”? Which traits separate something real and meaningful from something you don’t value? Is it the simulation? The separation from other beings? The possibility that the AI is deceiving you? Something I’m missing entirely?
(Personally I’m not at all bothered by the simulation, moderately bothered by the separation, and unsure how I feel about the deception.)
The salient difference for me is that the real one has maximal impact. Many actions in it can affect anyone in a lower world, but not vice versa. I’d like my decisions to have as much effect as possible as a general principle, I think (not the only principle, but the one that dominates in this scenario).
This is pretty much why I’m comfortable calling being in the highest possible world a terminal value—it’s really not about the simulation (wouldn’t be bothered if it turned out current world is a simulation, although I’d like to go higher), not especially about separation, and only slightly about deception (certainly losing all impactfulness becomes more irreversible if the AI is lying).
Hmmmm..
My own view: -Separation is very, very bad. I’d be somewhat OK with reality becoming subjective but with some kind of interface between people but this whole scenario as stated is approaching the collapse civilization so we can’t FOOM level. My personal reaction to seeing this described as better than status quo was somewhat similar to playing Mass Effect and listening to Reapers talk about ‘salvation through destruction’ and ascension in the form of perpetually genocidal robo-squids. I mean seriously? ‘All your friends are actually p-zombies?’ Are you kidding me? /rant
Living in highest possible world for me is not a value but having access or interface or something to the highest possible world is. (Not particularly high.) But knowing the truth definitely is and having my friends actually be people also is. Would prefer just being separated from friends and given Verthandi (I.e. sentient people, but optimized) like in Failed Utopia 2-4.