Don’t mind me; just trying to summarize some of the stuff I just processed.
If you’re choosing a strategy of predicting the future based on how accurate it turns out to be, the strategy who’s output influences the future in ways that make its prediction more likely will outperform a strategy that doesn’t (all else being equal). Thus, one might think that the strategy you chose will be the strategy that most effectively balances its prediction between a) how accurate that prediction (unconditioned on the prediction being given) and b) how much the prediction itself improves the accuracy of the prediction (conditioning on the prediction). Because of this, the intern predicts that the world will be made more predictable than it would be normally.
In short, you’ll tend to choose the prediction strategies that give self-fulfilling predictions when possible over those that don’t.
However, choosing the strategy that predicts the future most accurately is also equivalent to throwing away every strategy that doesn’t predict the future the best. In the same way that self-fulfilling predictions are good for prediction strategies because they enhance accuracy of the strategy in question, self-fulfilling predictions that seem generally surprising to outside observers are even better because they lower the accuracy of competing strategies. The established prediction strategy thus systematically causes the kinds of events in the world that no other method could predict to further establish itself. Because of this, the engineer predicts that the world will become less predictable than it would be normally.
In short, you’ll tend to choose the prediction strategy that give self-fulfilling predictions which fulfill in maximally surprising ways relative to the other prediction strategies you are considering.
I’m actually trying to be somewhat agnostic about the right conclusion here. I could have easily added another chapter discussing why the maximizing-surprise idea is not quite right. The moral is that the questions are quite complicated, and thinking vaguely about ‘optimization processes’ is quite far from adequate to understand this. Furthermore, it’ll depend quite a bit on the actual details of a training procedure!
Don’t mind me; just trying to summarize some of the stuff I just processed.
If you’re choosing a strategy of predicting the future based on how accurate it turns out to be, the strategy who’s output influences the future in ways that make its prediction more likely will outperform a strategy that doesn’t (all else being equal). Thus, one might think that the strategy you chose will be the strategy that most effectively balances its prediction between a) how accurate that prediction (unconditioned on the prediction being given) and b) how much the prediction itself improves the accuracy of the prediction (conditioning on the prediction). Because of this, the intern predicts that the world will be made more predictable than it would be normally.
In short, you’ll tend to choose the prediction strategies that give self-fulfilling predictions when possible over those that don’t.
However, choosing the strategy that predicts the future most accurately is also equivalent to throwing away every strategy that doesn’t predict the future the best. In the same way that self-fulfilling predictions are good for prediction strategies because they enhance accuracy of the strategy in question, self-fulfilling predictions that seem generally surprising to outside observers are even better because they lower the accuracy of competing strategies. The established prediction strategy thus systematically causes the kinds of events in the world that no other method could predict to further establish itself. Because of this, the engineer predicts that the world will become less predictable than it would be normally.
In short, you’ll tend to choose the prediction strategy that give self-fulfilling predictions which fulfill in maximally surprising ways relative to the other prediction strategies you are considering.
Oh god...
I’m actually trying to be somewhat agnostic about the right conclusion here. I could have easily added another chapter discussing why the maximizing-surprise idea is not quite right. The moral is that the questions are quite complicated, and thinking vaguely about ‘optimization processes’ is quite far from adequate to understand this. Furthermore, it’ll depend quite a bit on the actual details of a training procedure!