I think I’m missing something about your perverse donor example. What makes your number a prediction rather than just a preference? If they’re going to erase the (meaningless) 1M and give you P-1, you just maximize P, right? A prediction is just a stated belief, and if it’s not paying rent in conditional future experience, it’s probably not worth having.
More generally, is the self-confirming prediction just the same as a conditional probability on a not-revealed-by-the-oracle condition? In what cases will the oracle NOT want to reveal the condition? In this case, the nature of adversarial goals needs to be examined—why wouldn’t the oracle just falsify the prediction in addition to hiding the conditions?
Also, I’m not sure where the “continuous” requirement comes from. Your example isn’t continuous, only whole pennies are allowed. Even if only prime multiples of 3 were allowed, it would seem the same lesson holds.
Separately (and minor), I’m not enjoying the “can be arbitrarily bad” titles. They don’t convey information, and confuse me into thinking the posts are about something more fundamental than they seem to be. _ANY_ arbitrary scenario can be arbitrarily bad, why are these topics special on that front?
The donor example was to show how such a predictor could end up moving you far in the positive or negative direction. If you were optimising for income rather than accuracy, the choice is obvious.
The £(P±1) is a continuous model of a discontinuous reality. The model has a self-confirming prediction, and it turns out “reality” (the discretised version) has one too. Unless derivatives get extremely high, a continuous model implies a self-confirming prediction implies a close-to-self-confirming prediction in the discretised model.
I think I’m still confused—a naive sequence predictor is _OF COURSE_ broken by perverse or adversarial unmodelled (because of the naivety of the predictor) behaviors. And such a predictor cannot unlock new corners of strategy space, or generate self-reinforcing predictions, because the past sequence on which it’s trained won’t have those features.
And such a predictor cannot unlock new corners of strategy space, or generate self-reinforcing predictions, because the past sequence on which it’s trained won’t have those features.
See my last paragraph above; I don’t think we can rely on predictors not unlocking new corners of strategy space, because it may be able to learn gradually how to do so.
I think I’m missing something about your perverse donor example. What makes your number a prediction rather than just a preference? If they’re going to erase the (meaningless) 1M and give you P-1, you just maximize P, right? A prediction is just a stated belief, and if it’s not paying rent in conditional future experience, it’s probably not worth having.
More generally, is the self-confirming prediction just the same as a conditional probability on a not-revealed-by-the-oracle condition? In what cases will the oracle NOT want to reveal the condition? In this case, the nature of adversarial goals needs to be examined—why wouldn’t the oracle just falsify the prediction in addition to hiding the conditions?
Also, I’m not sure where the “continuous” requirement comes from. Your example isn’t continuous, only whole pennies are allowed. Even if only prime multiples of 3 were allowed, it would seem the same lesson holds.
Separately (and minor), I’m not enjoying the “can be arbitrarily bad” titles. They don’t convey information, and confuse me into thinking the posts are about something more fundamental than they seem to be. _ANY_ arbitrary scenario can be arbitrarily bad, why are these topics special on that front?
A self-confirming prediction is what an oracle that was a naive sequence predictor (or that was rewarded on results) would give. https://www.lesswrong.com/posts/i2dNFgbjnqZBfeitT/oracles-sequence-predictors-and-self-confirming-predictions
The donor example was to show how such a predictor could end up moving you far in the positive or negative direction. If you were optimising for income rather than accuracy, the choice is obvious.
The £(P±1) is a continuous model of a discontinuous reality. The model has a self-confirming prediction, and it turns out “reality” (the discretised version) has one too. Unless derivatives get extremely high, a continuous model implies a self-confirming prediction implies a close-to-self-confirming prediction in the discretised model.
I think I’m still confused—a naive sequence predictor is _OF COURSE_ broken by perverse or adversarial unmodelled (because of the naivety of the predictor) behaviors. And such a predictor cannot unlock new corners of strategy space, or generate self-reinforcing predictions, because the past sequence on which it’s trained won’t have those features.
See my last paragraph above; I don’t think we can rely on predictors not unlocking new corners of strategy space, because it may be able to learn gradually how to do so.