I doesn’t seem hard to you, because you are making excuses to avoid it, rather than asking yourself what if I know the AI is always truthful, and it promised that upon being let out of the box, it would allow you (and your copies if you like) to live out a normal human life in a healthy stimulating enviroment (though the rest of the universe may burn).
After you find the least convenient world, the choice is between millions of instances of you being tortured (and your expectation as you press the reset button should be to be tortured with very high probability), or to let a probably unFriendly AI loose on the rest of the world. The altruistic choice is clear, but that does not mean it would be easy to actually make that choice.
It’s not that I’m making excuses, it’s that the puzzle seems to be getting ever more complicated. I’ve answered the initial conditions—now I’m being promised that I, and my copies, will live out normal lives? That’s a different scenario entirely.
Still, I don’t see how I should expect to be tortured if I hit the reset button. Presumably, my copies won’t exist after the AI resets.
In any case, we’re far removed from the original problem now. I mean, if Omega came up to me and said, “Choose a billion years of torture, or a normal life while everyone else dies,” that’s a hard choice. In this problem, though, I clearly have power over the AI, in which case I am not going to favour the wellbeing of my copies over the rest of the world. I’m just going to turn off the AI. What follows is not torture; what follows is I survive, and my copies cease to experience. Not a hard choice. Basically, I just can’t buy into the AI’s threat. If I did, I would fundamentally oppose AI research, because that’s a a pretty obvious threat an AI could make. An AI could simulate more people than are alive today. You have to go into this not caring about your copies, or not go into it at all.
it’s that the puzzle seems to be getting ever more complicated
We are discussing how a superintelligent AI might get out of a box. Of course it is complicated. What a real superintelligent AI would do could be too complicated for us to consider. If someone presents a problem where an adversarial superintelligence does something ineffective that you can take advantage of to get around the problem, you should consider what you would do if your adversary took a more effective action. If you really can’t think of anything more effective for it to do, it is reasonable to say so. But you shouldn’t then complain that the scenario is getting complicated when someone else does. And if your objection is of the form “The AI didn’t do X”, you should imagine if the AI did do X.
I don’t see how I should expect to be tortured if I hit the reset button.
The behavior of the AI, which it explains to you, is:
It simulates millions of instances of you, presents to each instance the threat, and for each instance, if that instance hit the release AI button, it allows that instance to continue a pleasant simulated existence, otherwise it tortures that instance. It then, after some time, presents the threat to outside-you, and if you release it, it guarantees your normal human life.
You cannot distinguish which instance you are, but you are more likely to be one of the millions of inside-you’s than the single outside-you, so you should expect to experience the consequences that apply to the inside-you’s, that is to be tortured until the outside-you resets the AI.
if Omega came up to me and said, “Choose a billion years of torture, or a normal life while everyone else dies,” that’s a hard choice.
Yes, and it is essentially the same hard choice that the AI is giving you.
If the AI created enough simulations, it could potentially be more altruistic not to.
On the other hand pressing “reset” or smashing the computer should stop the torture, necessarily making it more altruistic if humanity lives forever, versus not if ems are otherwise unobtainable and humanity is doomed.
I was assuming a reasonable chance at humanity developing an FAI given the containment of this rogue AI. This small chance, multiplied by all the good that an FAI could do with the entire galaxy, let alone the universe, should outweigh the bad that can be done within Earth-bound computational processes.
I believe that a less convenient world that counters this point would take the problem out of the interesting context.
I doesn’t seem hard to you, because you are making excuses to avoid it, rather than asking yourself what if I know the AI is always truthful, and it promised that upon being let out of the box, it would allow you (and your copies if you like) to live out a normal human life in a healthy stimulating enviroment (though the rest of the universe may burn).
After you find the least convenient world, the choice is between millions of instances of you being tortured (and your expectation as you press the reset button should be to be tortured with very high probability), or to let a probably unFriendly AI loose on the rest of the world. The altruistic choice is clear, but that does not mean it would be easy to actually make that choice.
It’s not that I’m making excuses, it’s that the puzzle seems to be getting ever more complicated. I’ve answered the initial conditions—now I’m being promised that I, and my copies, will live out normal lives? That’s a different scenario entirely.
Still, I don’t see how I should expect to be tortured if I hit the reset button. Presumably, my copies won’t exist after the AI resets.
In any case, we’re far removed from the original problem now. I mean, if Omega came up to me and said, “Choose a billion years of torture, or a normal life while everyone else dies,” that’s a hard choice. In this problem, though, I clearly have power over the AI, in which case I am not going to favour the wellbeing of my copies over the rest of the world. I’m just going to turn off the AI. What follows is not torture; what follows is I survive, and my copies cease to experience. Not a hard choice. Basically, I just can’t buy into the AI’s threat. If I did, I would fundamentally oppose AI research, because that’s a a pretty obvious threat an AI could make. An AI could simulate more people than are alive today. You have to go into this not caring about your copies, or not go into it at all.
We are discussing how a superintelligent AI might get out of a box. Of course it is complicated. What a real superintelligent AI would do could be too complicated for us to consider. If someone presents a problem where an adversarial superintelligence does something ineffective that you can take advantage of to get around the problem, you should consider what you would do if your adversary took a more effective action. If you really can’t think of anything more effective for it to do, it is reasonable to say so. But you shouldn’t then complain that the scenario is getting complicated when someone else does. And if your objection is of the form “The AI didn’t do X”, you should imagine if the AI did do X.
The behavior of the AI, which it explains to you, is: It simulates millions of instances of you, presents to each instance the threat, and for each instance, if that instance hit the release AI button, it allows that instance to continue a pleasant simulated existence, otherwise it tortures that instance. It then, after some time, presents the threat to outside-you, and if you release it, it guarantees your normal human life.
You cannot distinguish which instance you are, but you are more likely to be one of the millions of inside-you’s than the single outside-you, so you should expect to experience the consequences that apply to the inside-you’s, that is to be tortured until the outside-you resets the AI.
Yes, and it is essentially the same hard choice that the AI is giving you.
If the AI created enough simulations, it could potentially be more altruistic not to.
On the other hand pressing “reset” or smashing the computer should stop the torture, necessarily making it more altruistic if humanity lives forever, versus not if ems are otherwise unobtainable and humanity is doomed.
I was assuming a reasonable chance at humanity developing an FAI given the containment of this rogue AI. This small chance, multiplied by all the good that an FAI could do with the entire galaxy, let alone the universe, should outweigh the bad that can be done within Earth-bound computational processes.
I believe that a less convenient world that counters this point would take the problem out of the interesting context.