I’ve been trying to wrap my head around arguments involving simulations, e.g. what to do if Skynet (replace with whatever AI you prefer to hate) threatens to torture a large number of simulations of you, etc.
Here is my stupid question: why can’t we humans use a similar threat? Why can’t I say that, if you don’t cooperate with my wishes, I’ll torture imaginary versions of you inside my head? It’s not like my brain isn’t another information processing device, so what is it about my imagination that feels less compelling than Skynet’s?
I no longer find it totally implausible that imagined people might, if modeled in enough detail, be in some sense conscious—it seems unlikely to me that human self-modeling and other-modeling logic would end up being that different—but even if we take that as given, there’s a couple of problems with threatening to imagine someone in some unpleasant situation.
The basic issue is asymmetry of information. You might be able to imagine someone that thinks or even reliably acts like your enemy; but, no matter how good you are at personality modeling, they aren’t going to have access to all, or even much, of your enemy’s memories and experiences. Lacking that, I wouldn’t say your imagined enemy is cognitively equivalent to your real enemy in a way that’d make the threat hold up.
(Skynet, by contrast, might be able to reproduce all that information by some means—brain scanning, say, or some superhuman form of induction.)
You can’t simulate anything in the relevant sense. Personally I don’t care about the threat you refer to even when it comes from skynet, but the thing that makes some people care is that the simulation contains all information that’s in their conscious mind, and thus plausibly is conscious and suffers for real.
My own thinking about this whole class of questions starts with: is the agent threatening this capable of torturing systems that I prefer (on reflection) not be tortured? If I’m confident they can do so, then they can credibly threaten me.
Among other things, this formulation lets me completely ignore whether Skynet’s simulation of me is actually me. That’s irrelevant to the question at hand. In fact, whether it’s even a simulation of me, and indeed whether it’s a person at all, is irrelevant. What’s important is whether I prefer it not be tortured.
A lot of ill-defined terms (“person”, “simulation”,”me”) thus drop out of my evaluation.
In principle I expect that a sufficiently capable intelligence can create systems that I prefer not be tortured, but I’d need quite a lot of evidence before I was actually confident that any given intelligence was capable of doing so.
That said, the problem of evidence is itself tricky here. I expect that it is much easier to build a system I don’t endorse caring about in the abstract, and then manipulate the setting so that I come to care about it anyway, than to build a system that I endorse caring about. That said, we can finesse the epistemic issue by asking a different question: is the intelligence capable of creating (and torturing) a system S such that, if I somehow became confident that S has the attributes S in fact has, I would prefer that S not be tortured?
My confidence that humans have this ability is low, though (as above) in principle I expect that a sufficiently capable intelligence can do so. Certainly I don’t have it, and I’ve never seen significant evidence that anyone else does.
I’ve been trying to wrap my head around arguments involving simulations, e.g. what to do if Skynet (replace with whatever AI you prefer to hate) threatens to torture a large number of simulations of you, etc.
Here is my stupid question: why can’t we humans use a similar threat? Why can’t I say that, if you don’t cooperate with my wishes, I’ll torture imaginary versions of you inside my head? It’s not like my brain isn’t another information processing device, so what is it about my imagination that feels less compelling than Skynet’s?
I no longer find it totally implausible that imagined people might, if modeled in enough detail, be in some sense conscious—it seems unlikely to me that human self-modeling and other-modeling logic would end up being that different—but even if we take that as given, there’s a couple of problems with threatening to imagine someone in some unpleasant situation.
The basic issue is asymmetry of information. You might be able to imagine someone that thinks or even reliably acts like your enemy; but, no matter how good you are at personality modeling, they aren’t going to have access to all, or even much, of your enemy’s memories and experiences. Lacking that, I wouldn’t say your imagined enemy is cognitively equivalent to your real enemy in a way that’d make the threat hold up.
(Skynet, by contrast, might be able to reproduce all that information by some means—brain scanning, say, or some superhuman form of induction.)
You can’t simulate anything in the relevant sense. Personally I don’t care about the threat you refer to even when it comes from skynet, but the thing that makes some people care is that the simulation contains all information that’s in their conscious mind, and thus plausibly is conscious and suffers for real.
My own thinking about this whole class of questions starts with: is the agent threatening this capable of torturing systems that I prefer (on reflection) not be tortured? If I’m confident they can do so, then they can credibly threaten me.
Among other things, this formulation lets me completely ignore whether Skynet’s simulation of me is actually me. That’s irrelevant to the question at hand. In fact, whether it’s even a simulation of me, and indeed whether it’s a person at all, is irrelevant. What’s important is whether I prefer it not be tortured.
A lot of ill-defined terms (“person”, “simulation”,”me”) thus drop out of my evaluation.
In principle I expect that a sufficiently capable intelligence can create systems that I prefer not be tortured, but I’d need quite a lot of evidence before I was actually confident that any given intelligence was capable of doing so.
That said, the problem of evidence is itself tricky here. I expect that it is much easier to build a system I don’t endorse caring about in the abstract, and then manipulate the setting so that I come to care about it anyway, than to build a system that I endorse caring about. That said, we can finesse the epistemic issue by asking a different question: is the intelligence capable of creating (and torturing) a system S such that, if I somehow became confident that S has the attributes S in fact has, I would prefer that S not be tortured?
My confidence that humans have this ability is low, though (as above) in principle I expect that a sufficiently capable intelligence can do so. Certainly I don’t have it, and I’ve never seen significant evidence that anyone else does.
Have you?