...if the malign superintelligence knows what observations we would condition on, it can likely arrange to make the world match those observations, making the probability of our observations given a malign superintelligence roughly one
The probability of any observation given the existence of a malign superintelligence is 1? So P(observation | malign superintelligence) adds up to like a gajillion?
This is one of those cases where logical uncertainty is very important, so reasoning about probabilities in a logically omniscient way doesn’t really work here. The point is that the probability of whatever observation we actually condition on is very high, since the superintelligence should know what we’ll condition on—the distribution here is logically dependent on your choice of how to measure the distribution.
The probability of any observation given the existence of a malign superintelligence is 1? So P(observation | malign superintelligence) adds up to like a gajillion?
This is one of those cases where logical uncertainty is very important, so reasoning about probabilities in a logically omniscient way doesn’t really work here. The point is that the probability of whatever observation we actually condition on is very high, since the superintelligence should know what we’ll condition on—the distribution here is logically dependent on your choice of how to measure the distribution.