Thanks for linking to that previous post! I think the new considerations I’ve added here are:
(i) the rational refusal to update the prior of being in a simulation[1]; and
(ii) the likely minute cost of sparing us, thereby requiring a similarly low simulation prior to make it worth the effort.
In brief, I understand your argument to be that a being sufficiently intelligent to create a simulation wouldn’t need it for the purpose of asserting the ASI’s alignment in the first place. It seems to me that that argument can potentially survive under ii, depending on how strongly you (believe the ASI will) believe your conclusion. To that effect, I’m interested in hearing your reply to one of the counterarguments raised in that previous post, namely:
Maybe showing the alignment of an AI without running it is vastly more difficult than creating a good simulation. This feels unlikely, but I genuinely do not see any reason why this can’t be the case. If we create a simulation which is “correct” up to the nth digit of pi, beyond which the simpler explanation for the observed behavior becomes the simulation theory rather than a complex physics theory, then no matter how intelligent you are, you’d need to calculate n digits of pi to figure this out. And if n is huge, this will take a while.
In any case, even if your argument does hold under ii, whether it survives under i seems to be heavily influenced by inferential distance. Whatever the ASI “knows” or “concludes” is known or concluded through physical computations, which can presumably be later inspected if it happens to be in a simulation. It thus seems only natural that a sufficiently high (which may still be quite small) prior of being in a simulation would be enough to “lock” the ASI in that state, making undergoing those computations simply not worth the risk.
I’d have to think a bit more before tabooing that term, as it seems that “being fed false sensory data” doesn’t do the trick – you can be in a simulation without any sensory data at all.
I’m going to be a little stubborn and decline to reply till you ask me a question without “simulate” or “simulation” in it. I have an unpleasant memory of getting motte-and-baileyed by it.
Imagine that someone with sufficiently advanced technology perfectly scans your brain for every neuron firing while you dream, and can also make some neurons fire at will. Replace every instance of “simulation” in my previous comment with the analogous of that for the ASI.
Thanks for linking to that previous post! I think the new considerations I’ve added here are:
(i) the rational refusal to update the prior of being in a simulation[1]; and
(ii) the likely minute cost of sparing us, thereby requiring a similarly low simulation prior to make it worth the effort.
In brief, I understand your argument to be that a being sufficiently intelligent to create a simulation wouldn’t need it for the purpose of asserting the ASI’s alignment in the first place. It seems to me that that argument can potentially survive under ii, depending on how strongly you (believe the ASI will) believe your conclusion. To that effect, I’m interested in hearing your reply to one of the counterarguments raised in that previous post, namely:
In any case, even if your argument does hold under ii, whether it survives under i seems to be heavily influenced by inferential distance. Whatever the ASI “knows” or “concludes” is known or concluded through physical computations, which can presumably be later inspected if it happens to be in a simulation. It thus seems only natural that a sufficiently high (which may still be quite small) prior of being in a simulation would be enough to “lock” the ASI in that state, making undergoing those computations simply not worth the risk.
I’d have to think a bit more before tabooing that term, as it seems that “being fed false sensory data” doesn’t do the trick – you can be in a simulation without any sensory data at all.
I’m going to be a little stubborn and decline to reply till you ask me a question without “simulate” or “simulation” in it. I have an unpleasant memory of getting motte-and-baileyed by it.
Imagine that someone with sufficiently advanced technology perfectly scans your brain for every neuron firing while you dream, and can also make some neurons fire at will. Replace every instance of “simulation” in my previous comment with the analogous of that for the ASI.