you should expect to update in the direction of the truth as the evidence comes in
I think this was addressed later on, but this is not at all true. With the waterfall example, every mile that passes without a waterfall you update downwards, but if there’s a waterfall at the very end you’ve been updating against the truth the whole time.
Another case. Suppose you’re trying to predict the outcome of a Gaussian random walk. Each step, you update whatever direction the walk took, but 50% of these steps are “away” from the truth.
you probably shouldn’t be able to predict that this pattern will happen to you in the future.
Again addressed later on, but one can easily come up with stories in which one predictably updates either “in favor of” or “against” AI doom.
Suppose you think there’s a 1% chance of AI doom every year, and AI Doom will arrive by 2050 or never. Then you predictably update downwards every year (unless Doom occurs).
Suppose on the other hand that you expect AI to stagnate at some level below AGI, but if AGI is developed then Doom occurs with 100% certainty. Then each year AI fails to stagnate you update upwards (until AI actually stagnates).
I think this was addressed later on, but this is not at all true. With the waterfall example, every mile that passes without a waterfall you update downwards, but if there’s a waterfall at the very end you’ve been updating against the truth the whole time.
Another case. Suppose you’re trying to predict the outcome of a Gaussian random walk. Each step, you update whatever direction the walk took, but 50% of these steps are “away” from the truth.
Again addressed later on, but one can easily come up with stories in which one predictably updates either “in favor of” or “against” AI doom.
Suppose you think there’s a 1% chance of AI doom every year, and AI Doom will arrive by 2050 or never. Then you predictably update downwards every year (unless Doom occurs).
Suppose on the other hand that you expect AI to stagnate at some level below AGI, but if AGI is developed then Doom occurs with 100% certainty. Then each year AI fails to stagnate you update upwards (until AI actually stagnates).