As we evaluate predictions for accuracy, one thing we should all be hyper-aware of is that predictions can affect behavior. It is always at least a little bit suspect to evaluate a prediction simply for accuracy, when its goal might very well have been more than accuracy.
If I bet you $100 on even odds that I’m going to lose 10 lbs this month, it doesn’t necessarily indicate that I think the probability of it happening is > 50%. Perhaps this bet increases the probability of it happening from 10% to 40%, and maybe that 30% increase in probability is worth the expected $20 cost. More generally, even with no money on the table, such a bet might motivate me, and so it cannot be inferred from the fact that I made the bet that I actually believe the numbers in the bet.
Or let’s talk about the planning fallacy. Instrumentally, it might make sense to hold the belief that you’ll get your project done before the deadline, if that sort of thinking motivates you to finish it earlier than you would if you were completely resigned to finishing it late from the get-go. It might even be detrimental to your project-finishing goals to become enlightened about the reality.
And of course, predictions can affect other people than the predicter. It is ludicrous to look at the public predictions of the Fed chairman and call him a fool when he turns out to be wrong.
Sometimes a prediction is a prediction. But this is definitely something to keep in mind. And gwern, given that you have all this data now, you might find it interesting to see if there are any systematic differences between predictions that do and don’t have any significant effect on behavior.
As we evaluate predictions for accuracy, one thing we should all be hyper-aware of is that predictions can affect behavior. It is always at least a little bit suspect to evaluate a prediction simply for accuracy, when its goal might very well have been more than accuracy.
If I bet you $100 on even odds that I’m going to lose 10 lbs this month, it doesn’t necessarily indicate that I think the probability of it happening is > 50%. Perhaps this bet increases the probability of it happening from 10% to 40%, and maybe that 30% increase in probability is worth the expected $20 cost. More generally, even with no money on the table, such a bet might motivate me, and so it cannot be inferred from the fact that I made the bet that I actually believe the numbers in the bet.
Or let’s talk about the planning fallacy. Instrumentally, it might make sense to hold the belief that you’ll get your project done before the deadline, if that sort of thinking motivates you to finish it earlier than you would if you were completely resigned to finishing it late from the get-go. It might even be detrimental to your project-finishing goals to become enlightened about the reality.
And of course, predictions can affect other people than the predicter. It is ludicrous to look at the public predictions of the Fed chairman and call him a fool when he turns out to be wrong.
Sometimes a prediction is a prediction. But this is definitely something to keep in mind. And gwern, given that you have all this data now, you might find it interesting to see if there are any systematic differences between predictions that do and don’t have any significant effect on behavior.