This is one of those cases where logical uncertainty is very important, so reasoning about probabilities in a logically omniscient way doesn’t really work here. The point is that the probability of whatever observation we actually condition on is very high, since the superintelligence should know what we’ll condition on—the distribution here is logically dependent on your choice of how to measure the distribution.
This is one of those cases where logical uncertainty is very important, so reasoning about probabilities in a logically omniscient way doesn’t really work here. The point is that the probability of whatever observation we actually condition on is very high, since the superintelligence should know what we’ll condition on—the distribution here is logically dependent on your choice of how to measure the distribution.