(Sorry—I’ve just stumbled on this thread and I’m not sure whether I should post this comment here or on Wei Dai’s original post).
It seems to me that the right thing for clippy to do is choose 10^20 staples instead of 10^10 paperclips, but not because of anything to do with logical uncertainty.
There are presumably a vast number of parallel (and/or non-logically-counterfactual) universes where games of this nature are being played out. In almost exactly half of them, the roles of clippy and staply will be reversed. A UDT clippy will happily cooperate in this case knowing that the staplies in parallel universes will do the same and generate lots of paperclips in the process.
This would still be the case if there’s no pi calculation involved, and Omega just flies to a planet, finds the first two utility maximizers with orthogonal utility functions and offers one of them a choice of 10^10 units of utility (confusion note: according to what scale???) or 10^20 units of the other agent’s utility.
Added fun:
A paperclip maximizer seems to have an inherent advantage against a paperclip minimizer (i.e. eventually Omega is unable to destroy any more paperclips). But what if you were a (paperclip minus staple) maximizer? Any win in one universe is going to be exactly canceled out by an analogous win for a (staple minus paperclip) maximizer in another universe, if we assume that those will be spawned with equal probability.
My argument only works because the concept of paperclips seems non-entangled with the concept of the millionth digit of pi. What if you replace “paperclip” with “piece of paper saying the millionth digit of pi is odd”?
I think it’s right to cooperate in this thought experiment only to the extent that we accept the impossibility of isolating this thought experiment from its other possible instances
I agree with one direction of implication—if we accept the impossibility of isolating this thought experiment form its other possible instances (e.g. cases with the exact same wording but with paperclips and staples swapped) then it’s right to cooperate.
If we don’t accept that then I will admit to being confused and have nothing meaningful to say either way. I accept the “least convenient world” principle for when someone suggests a really bizarre thought experiment, but I’m having trouble with the concept of “least convenient set of possible counterfactual worlds”. Is this concept worth exploring in its own right?
(Sorry—I’ve just stumbled on this thread and I’m not sure whether I should post this comment here or on Wei Dai’s original post).
It seems to me that the right thing for clippy to do is choose 10^20 staples instead of 10^10 paperclips, but not because of anything to do with logical uncertainty.
There are presumably a vast number of parallel (and/or non-logically-counterfactual) universes where games of this nature are being played out. In almost exactly half of them, the roles of clippy and staply will be reversed. A UDT clippy will happily cooperate in this case knowing that the staplies in parallel universes will do the same and generate lots of paperclips in the process.
This would still be the case if there’s no pi calculation involved, and Omega just flies to a planet, finds the first two utility maximizers with orthogonal utility functions and offers one of them a choice of 10^10 units of utility (confusion note: according to what scale???) or 10^20 units of the other agent’s utility.
Added fun:
A paperclip maximizer seems to have an inherent advantage against a paperclip minimizer (i.e. eventually Omega is unable to destroy any more paperclips). But what if you were a (paperclip minus staple) maximizer? Any win in one universe is going to be exactly canceled out by an analogous win for a (staple minus paperclip) maximizer in another universe, if we assume that those will be spawned with equal probability.
My argument only works because the concept of paperclips seems non-entangled with the concept of the millionth digit of pi. What if you replace “paperclip” with “piece of paper saying the millionth digit of pi is odd”?
What do you think of this thread, in particular the part quoted in the last comment?
The quote was:
I agree with one direction of implication—if we accept the impossibility of isolating this thought experiment form its other possible instances (e.g. cases with the exact same wording but with paperclips and staples swapped) then it’s right to cooperate.
If we don’t accept that then I will admit to being confused and have nothing meaningful to say either way. I accept the “least convenient world” principle for when someone suggests a really bizarre thought experiment, but I’m having trouble with the concept of “least convenient set of possible counterfactual worlds”. Is this concept worth exploring in its own right?