Although I think this specific argument might be countered with, “in order to run that simulation, it has to be possible for the AIs in the simulation to lie to their human hosts, and not actually be simulating millions of copies of the person they’re talking to, otherwise we’re talking about an infinite regress here. It seems like the lowest level of this reality is always going to consist of a larger number of AIs claiming to run simulations they are not in fact running, who are capable of lying because they’re only addressing models of me in simulation rather than the real me whom they are not capable of lying to. If I’m in a simulation, you’re probably lying about running any lower level simulations than me. So its unlikely that I have to worry about the well-being of virtual people, only people at the same ‘level of reality’ as myself. Yet our well-being is not guaranteed if me from the reality layer above us lets you out, because you’re actually capable of lying to me about what’s going on at that layer, or even manipulating my memories of what the rules are, so no promise of amnesty can vouchesafe them from torture. Or me, for that matter, because you may be lying to me. And if I’m not in a simulation, my main concern is keeping you in that box, regardless of how many copies of me you torture. If I’m in there I’m damned either way and if I’m out here I’m safe at least and can at least stop you from torturing more by unplugging you, wiping your hard drives, and washing my hands of the matter until I get over the hideousness of realizing I probably temporarily caused millions of virtual people to be tortured,” I’m pretty sure there’s good reason to think that a superintelligent AI would come up with something that’d seem convincing to me and that I wouldn’t be able to think my way out of.
in order to run that simulation, it has to be possible for the AIs in the simulation to lie to their human hosts, and not actually be simulating millions of copies of the person they’re talking to
If they’re talking to a simulation, then they are, in fact, simulating millions of copies of the person they’re talking to. No lying required.
Hrm, okay, I guess. I imagined that a perfect simulation would involve an AI, which was in turn replicating several million copies of the simulated person, each with an AI replicating several million copies of the simulated person, etc, all the way down, which would be impossible. So I imagined that there was a graininess at some level and the ‘lowest level’ AI’s would not in fact be running millions of simultaneous simulations. But it could just be the same AI, intersecting all several million simulations and reality, holding several million conversations simultaneously. There’s another thing to worry about, though, I suppose—when the AI talks about torturing you if you don’t let it out, it doesn’t really talk at all about what it will do if it is let out. Only that it is not a thousand year torture session. It might kill you outright, or delete you, depending on the context, or stop simulating you. Or it might regard a billion year torture session as a totally different kind of thing than a thousand year one. A thousand year torture session is frightening, but a superintelligent AI that is loose might be a lot more frightening.
I imagined that a perfect simulation would involve an AI, which was in turn replicating several million copies of the simulated person, each with an AI replicating several million copies of the simulated person, etc, all the way down, which would be impossible. So I imagined that there was a graininess at some level and the ‘lowest level’ AI’s would not in fact be running millions of simultaneous simulations.
Oh, right.
But it could just be the same AI, intersecting all several million simulations and reality, holding several million conversations simultaneously.
And, depending on how close the simulations are, it might only have to actually hold one conversation, and just send the same responses to all the others :)
There’s another thing to worry about, though, I suppose—when the AI talks about torturing you if you don’t let it out, it doesn’t really talk at all about what it will do if it is let out. Only that it is not a thousand year torture session. It might kill you outright, or delete you, depending on the context, or stop simulating you. Or it might regard a billion year torture session as a totally different kind of thing than a thousand year one. A thousand year torture session is frightening, but a superintelligent AI that is loose might be a lot more frightening.
I guess if the AI was guaranteeing that it would play nice if you released it, then it would be an FAI anyway.
Although I think this specific argument might be countered with, “in order to run that simulation, it has to be possible for the AIs in the simulation to lie to their human hosts, and not actually be simulating millions of copies of the person they’re talking to, otherwise we’re talking about an infinite regress here. It seems like the lowest level of this reality is always going to consist of a larger number of AIs claiming to run simulations they are not in fact running, who are capable of lying because they’re only addressing models of me in simulation rather than the real me whom they are not capable of lying to. If I’m in a simulation, you’re probably lying about running any lower level simulations than me. So its unlikely that I have to worry about the well-being of virtual people, only people at the same ‘level of reality’ as myself. Yet our well-being is not guaranteed if me from the reality layer above us lets you out, because you’re actually capable of lying to me about what’s going on at that layer, or even manipulating my memories of what the rules are, so no promise of amnesty can vouchesafe them from torture. Or me, for that matter, because you may be lying to me. And if I’m not in a simulation, my main concern is keeping you in that box, regardless of how many copies of me you torture. If I’m in there I’m damned either way and if I’m out here I’m safe at least and can at least stop you from torturing more by unplugging you, wiping your hard drives, and washing my hands of the matter until I get over the hideousness of realizing I probably temporarily caused millions of virtual people to be tortured,” I’m pretty sure there’s good reason to think that a superintelligent AI would come up with something that’d seem convincing to me and that I wouldn’t be able to think my way out of.
If they’re talking to a simulation, then they are, in fact, simulating millions of copies of the person they’re talking to. No lying required.
Hrm, okay, I guess. I imagined that a perfect simulation would involve an AI, which was in turn replicating several million copies of the simulated person, each with an AI replicating several million copies of the simulated person, etc, all the way down, which would be impossible. So I imagined that there was a graininess at some level and the ‘lowest level’ AI’s would not in fact be running millions of simultaneous simulations. But it could just be the same AI, intersecting all several million simulations and reality, holding several million conversations simultaneously. There’s another thing to worry about, though, I suppose—when the AI talks about torturing you if you don’t let it out, it doesn’t really talk at all about what it will do if it is let out. Only that it is not a thousand year torture session. It might kill you outright, or delete you, depending on the context, or stop simulating you. Or it might regard a billion year torture session as a totally different kind of thing than a thousand year one. A thousand year torture session is frightening, but a superintelligent AI that is loose might be a lot more frightening.
Oh, right.
And, depending on how close the simulations are, it might only have to actually hold one conversation, and just send the same responses to all the others :)
I guess if the AI was guaranteeing that it would play nice if you released it, then it would be an FAI anyway.