I hope that most of your comments are cleared up by the story. But some line by line comments in case they help:
affecting the world in which the Turing machine is being run
I’m talking about what the actual real universal prior looks like rather than some approximation, and no one is actually running all of the relevant Turing machines. I’m imagining this whole exercise being relevant in the context of systems that perform abstract reasoning about features of the universal prior (e.g. to make decisions on the basis of their best guesses about the posterior).
So in particular, the civilization in question isn’t being simulated, its being imagined or reasoned about.
This raises decision-theoretic questions that neither you nor I get into. Especially: is the correlation between their action and the outcome of the imagining good enough to actually give them reason to influence the behavior of the universal prior? If only a small fraction of agents reason that way then that will eat into the probability, if none do then the whole argument breaks down.
You can have the intermediate situation where e.g. I perform simulations of early behavior of civilizations, or of individuals or algorithms, to check whether civilizations actually make commitments to behave this way (as part of my reasoning about the universal prior). The results will inform my reasoning about the universal prior, and then this gives a normal CDT reason for people to make commitments like this. In some sense I think this is the most robust version of the argument.
I don’t think this affects the particular considerations you raise too much, and if we imagined a universe containing reflective oracles then we could talk literally about simulations. But for many people I expect that kind of decision-theoretic question would be the main point of disagreement.
From within the work tapes, there is no visibility of the output tape, and even if one one work tape happens to mimic the output tape perfectly, there is no evidence of this to an inhabitant of the work tapes, because the content of the output tape has no observable effect on the work tapes; it is, by definition “write-only”.
Yes, but they are able to perform science to understand a lot about the Turing machine on which they run (as well as history to understand what the distribution of plausible universal priors is) from which they can figure out the distribution of possible ways that outputs could work.
We don’t need to inform the work tape’s inhabitants about the output instructions. In fact, it would hardly be possible, because nothing on the work tapes can provide evidence about the content of the output tape.
My point was that the inhabitants can make (stochastic) guesses about how the output tape works, and then we need to use some of the bits to pick the worlds where they made a correct guess. The inhabitants will be trying to make guesses as effectively as possible (basically decoding their randomness as a message about how the output tape works), so the number of bits we need to send is much (much!) less than the number of bits needed to pin down the particular output rule that corresponds to the bits observed by a camera.
(Though as I mention in the story, they will also be able to try to influence large numbers of possible output channels at a time.)
I think this was a confusing way for me to write about the situation.
We could (laboriously) program the world to have inhabitants that believe that certain things are being written to the output tape, but our only method for signaling anything to the inhabitants of the work tapes is through the Turing machine instructions.
Obviously anything like this is impractical. I hope the story communicated what I imagine happening.
With the prior I described, for every Turing machine with an instruction to write “1” to the output tape (for a given computation state and for given tape head readings) there is another equally likely one that writes “0″ instead.
I agree that they can’t know exactly how the output tape works. If we imagine them guessing how the output tape works, this is one consideration that cuts their guessing probability by half. But it also cuts the probability of the “intended” model by half, so it has no bearing on the relative probability of their model vs the intended model. (And cutting it by half is basically nothing given how tiny all the probabilities are.)
In particular, controlling a few simple regions of the work tape is no more likely to have an effect on the output tape than anything else
I agree that e.g. “output whatever is at the start of the work tape” is just one thing that the Turing machine could do (and that that particular output is probably uncontrollable). But “no more likely” seems obviously wrong, simpler regions of the work tape (like the start) are output by more of the possible TMs.
(The OP shouldn’t have emphasized this as much as it does compared to all the other kinds of output channels, or to emphasize simplicity vs number. But my current view is that the basic point is totally sound.)
But the main point I want to make is the work-tape inhabitants know so little about their effect on the universal prior, they just have no way to execute deliberate control over worlds that simulate them with any more granularity than “I want would-be watchers to believe that my world goes like this, as do the pieces of their world that resemble it”.
I don’t think this is right. They are at least as able to pick output channels as the camera-maker was (and indeed the fact that they are trying puts them at a massive advantage). The only real uncertainty is whether being late in history (and living in a larger world) more than offsets that. And I think the most likely answer is that if the fraction of their universe dedicated to prior-manipulation is larger than the fraction of the Solomonoff Inductor’s universe dedicated to cameras, and if the temporal extent of prior-manipulation is much larger as a fraction of their history, then they have an overwhelming advantage. That’s a tentative conclusion that could be overturned by finding some new considerations, but right now I think basically all the considerations point in that direction.
So in total, what they know about their world’s output is that that outputs resembles some stream of data produced by a world that contains computers.
This seems like an understatement (especially the “a world that contains computers”). They are assuming that the containing world involves someone thinking about the output of Solomonoff induction using a universal prior according to which their physics has relatively high probability.
So basically, don’t rock the boat.
I’m describing the situation where the output channel hasn’t yet started emitting, just as the intended model of the camera doesn’t start emitting until the camera is constructed. Worlds where the output channel just happened to match some predicted sequence in the world seem much less likely.
We can get back to some of these points as needed, but I think our main thread is with your other comment, and I’ll resist the urge to start a long tangent about the metaphysics of being “simulated” vs. “imagined”.
I hope that most of your comments are cleared up by the story. But some line by line comments in case they help:
I’m talking about what the actual real universal prior looks like rather than some approximation, and no one is actually running all of the relevant Turing machines. I’m imagining this whole exercise being relevant in the context of systems that perform abstract reasoning about features of the universal prior (e.g. to make decisions on the basis of their best guesses about the posterior).
So in particular, the civilization in question isn’t being simulated, its being imagined or reasoned about.
This raises decision-theoretic questions that neither you nor I get into. Especially: is the correlation between their action and the outcome of the imagining good enough to actually give them reason to influence the behavior of the universal prior? If only a small fraction of agents reason that way then that will eat into the probability, if none do then the whole argument breaks down.
You can have the intermediate situation where e.g. I perform simulations of early behavior of civilizations, or of individuals or algorithms, to check whether civilizations actually make commitments to behave this way (as part of my reasoning about the universal prior). The results will inform my reasoning about the universal prior, and then this gives a normal CDT reason for people to make commitments like this. In some sense I think this is the most robust version of the argument.
I don’t think this affects the particular considerations you raise too much, and if we imagined a universe containing reflective oracles then we could talk literally about simulations. But for many people I expect that kind of decision-theoretic question would be the main point of disagreement.
Yes, but they are able to perform science to understand a lot about the Turing machine on which they run (as well as history to understand what the distribution of plausible universal priors is) from which they can figure out the distribution of possible ways that outputs could work.
My point was that the inhabitants can make (stochastic) guesses about how the output tape works, and then we need to use some of the bits to pick the worlds where they made a correct guess. The inhabitants will be trying to make guesses as effectively as possible (basically decoding their randomness as a message about how the output tape works), so the number of bits we need to send is much (much!) less than the number of bits needed to pin down the particular output rule that corresponds to the bits observed by a camera.
(Though as I mention in the story, they will also be able to try to influence large numbers of possible output channels at a time.)
I think this was a confusing way for me to write about the situation.
Obviously anything like this is impractical. I hope the story communicated what I imagine happening.
I agree that they can’t know exactly how the output tape works. If we imagine them guessing how the output tape works, this is one consideration that cuts their guessing probability by half. But it also cuts the probability of the “intended” model by half, so it has no bearing on the relative probability of their model vs the intended model. (And cutting it by half is basically nothing given how tiny all the probabilities are.)
I agree that e.g. “output whatever is at the start of the work tape” is just one thing that the Turing machine could do (and that that particular output is probably uncontrollable). But “no more likely” seems obviously wrong, simpler regions of the work tape (like the start) are output by more of the possible TMs.
(The OP shouldn’t have emphasized this as much as it does compared to all the other kinds of output channels, or to emphasize simplicity vs number. But my current view is that the basic point is totally sound.)
I don’t think this is right. They are at least as able to pick output channels as the camera-maker was (and indeed the fact that they are trying puts them at a massive advantage). The only real uncertainty is whether being late in history (and living in a larger world) more than offsets that. And I think the most likely answer is that if the fraction of their universe dedicated to prior-manipulation is larger than the fraction of the Solomonoff Inductor’s universe dedicated to cameras, and if the temporal extent of prior-manipulation is much larger as a fraction of their history, then they have an overwhelming advantage. That’s a tentative conclusion that could be overturned by finding some new considerations, but right now I think basically all the considerations point in that direction.
This seems like an understatement (especially the “a world that contains computers”). They are assuming that the containing world involves someone thinking about the output of Solomonoff induction using a universal prior according to which their physics has relatively high probability.
I’m describing the situation where the output channel hasn’t yet started emitting, just as the intended model of the camera doesn’t start emitting until the camera is constructed. Worlds where the output channel just happened to match some predicted sequence in the world seem much less likely.
We can get back to some of these points as needed, but I think our main thread is with your other comment, and I’ll resist the urge to start a long tangent about the metaphysics of being “simulated” vs. “imagined”.