I feel like this story has run aground on an impossibility result. If a random variable’s value is unknowable (but its distribution is known) and an intelligent agent wants to act on its value, and they randomize their actions, the expected log probability of them acting on the true value cannot exceed the entropy of the distribution, no matter their intelligence.
I think that’s right (other than the fact that they can win simultaneously for many different output rules, but I’m happy ignoring that for now). But I don’t see why it contradicts the story at all. In the story the best case is that we know the true distribution of output rules, and then we do the utility-maximizing thing, and that results in our sequence having way more probability than some random camera on old earth.
If you want to talk about the information theory, and ignore the fact that we can do multiple things, then we control the single output channel with maximal probability, while the camera is just some random output channel (presumably with some much smaller probability).
The information theory isn’t very helpful, because actually all of the action is about which output channels are controllable. If you restrict to some subset of “controllable” channels, and believe that any output rule that outputs the camera is controllable, then the conclusion still holds. So the only way it fails is when the camera is higher probability than the best controllable output channels.
To express my confusion more precisely:
I think that’s right (other than the fact that they can win simultaneously for many different output rules, but I’m happy ignoring that for now). But I don’t see why it contradicts the story at all. In the story the best case is that we know the true distribution of output rules, and then we do the utility-maximizing thing, and that results in our sequence having way more probability than some random camera on old earth.
If you want to talk about the information theory, and ignore the fact that we can do multiple things, then we control the single output channel with maximal probability, while the camera is just some random output channel (presumably with some much smaller probability).
The information theory isn’t very helpful, because actually all of the action is about which output channels are controllable. If you restrict to some subset of “controllable” channels, and believe that any output rule that outputs the camera is controllable, then the conclusion still holds. So the only way it fails is when the camera is higher probability than the best controllable output channels.