I spent years trading in prediction markets so I can offer some perspective.
If you step back and think about it, the question ‘How well can the long-term future be forecasted?’ doesn’t really have an answer. The reason for this is that it completely depends on the domain of the forecasts. Like, consider all facts about the universe. Some facts are very, very predictable. In 10 years, I predict the Sun will exist with 99.99%+ probability. Some facts are very, very unpredictable. In 10 years, I have no clue whether the coin you flip will come up heads or tails. As a result, you cannot really say the future is predictable or not predictable. It depends on which aspect of the future you are predicting. And even if you say, ok sure it depends, but like what’s the average answer—even then, the only the way to arrive at some unbiased global sense of whether the future is predictable is to come up with some way of enumerating and weighing all possible facts about the future universe… which is an impossible problem. So we’re left with the unsatisfying truth that the future is neither predictable or unpredictable—it depends on which features of the future you are considering.
So when you show the plot above, you have to realize it doesn’t generalize very well to other domains. For example, if the questions were about certain things—e.g., will the sun exist in 10 years—it would look high and flat. If the questions were about fundamentally uncertain things—e.g., what will the coin flip be 10 years from now—it would look low and flat. The slope we observe in that plot is less a property of how well the future can be predicted and more a property of the limited set of questions that were asked. If the questions were about uncertain near-term geopolitical events, then that graph shows the rate that information came in to the market consensus. It doesn’t really tell us about the bigger picture of predicting the future.
Incidentally, this was my biggest gripe with Tetlock and Gardner’s Superforecasting book. They spent a lot of time talking about how Superforecasters could predict the future, but almost no time talking about how the questions were selected and how if you choose different sets of counterfactual questions you can get totally different results (e.g., experts cannot predict the future vs rando smart people can predict the future). I don’t really fault them for this, because it’s a slippery thorny issue to discuss. I hope I have given you some flavor of it here.
I agree with most of what you’re saying, but this part seems like giving up way too easily: “And even if you say, ok sure it depends, but like what’s the average answer—even then, the only the way to arrive at some unbiased global sense of whether the future is predictable is to come up with some way of enumerating and weighing all possible facts about the future universe… which is an impossible problem. So we’re left with the unsatisfying truth that the future is neither predictable or unpredictable—it depends on which features of the future you are considering.”
The only way to say something useful about this is literally enumerating all possible facts? Sounds needlessly pessimistic.
On the contrary, I think it could be tremendously interesting and useful to start building some kind of categorization of prediction domains that allows saying something about their respective predictability. Obviously this is a hard problem, obviously the universe is very complex and the categorization will miss a lot of the intricacy, but the same is true about many other domains of knowledge (probably, nearly all of them). Despite the complexity of the universe (i) we should keep looking for (extremely) simplified models that capture a lot of what we actually care about (ii) having even an extremely simplified model is often much better than no model at all (iii) the model will keep evolving over time (which is to say, it feels more like a potential new science than a single problem that can be stated and solved in a relatively short time frame).
Thanks for this. The work I really want to see from more forecasting projects is an analysis of how much things that typically impact people’s lives can be predicted. Things like health, home-ownership, relationships, career, etc. Specifically, people’s levels of cooperate/defect against their future self seems really inconsistent. i.e. people work really hard for their future selves along certain dimensions and then defect along lots of others. This is mostly just mimetic momentum, but still. Even rigorous research figuring out exactly what actuaries know that can be applied practically by people would be good. After all, actuaries have really good life outcomes along lots of dimensions, which means that most aren’t taking advantage of the insights there.
My hope had been that 80k hours would have evolved to do more in this area but they’ve specialized narrower than that AFAICT.
I’ve been thinking similar things about predictability recently. Different variables have different levels of predictability, it seems very clear. I’m also under the impression that the examples in the Superforecasting study were quite specific. It seems likely that similar problems to what they studied have essentially low predictability 5-10 years out (and that is interesting information!), but this has limited relevance on other possible interesting questions.
the only the way to arrive at some unbiased global sense of whether the future is predictable is to come up with some way of enumerating and weighing all possible facts about the future universe… which is an impossible problem. So we’re left with the unsatisfying truth that the future is neither predictable or unpredictable—it depends on which features of the future you are considering.
While I agree with the specifics, I don’t think that the answer to a question like, “What is the average predictability of all possible statements” would be all that interesting. We generally care about a very small subset of “all possible statements.” It seems pretty reasonable to me that we could learn about the predictability of the kinds of things we’re interested. That said, i feel like we can get most of the benefits of this by just having calibrated forecasters try predicting all of these things, and seeing what their resolution numbers are. So I don’t think we need to do a huge amount of work running tests for the sole purpose of better understanding long-term predictability.
I spent years trading in prediction markets so I can offer some perspective.
If you step back and think about it, the question ‘How well can the long-term future be forecasted?’ doesn’t really have an answer. The reason for this is that it completely depends on the domain of the forecasts. Like, consider all facts about the universe. Some facts are very, very predictable. In 10 years, I predict the Sun will exist with 99.99%+ probability. Some facts are very, very unpredictable. In 10 years, I have no clue whether the coin you flip will come up heads or tails. As a result, you cannot really say the future is predictable or not predictable. It depends on which aspect of the future you are predicting. And even if you say, ok sure it depends, but like what’s the average answer—even then, the only the way to arrive at some unbiased global sense of whether the future is predictable is to come up with some way of enumerating and weighing all possible facts about the future universe… which is an impossible problem. So we’re left with the unsatisfying truth that the future is neither predictable or unpredictable—it depends on which features of the future you are considering.
So when you show the plot above, you have to realize it doesn’t generalize very well to other domains. For example, if the questions were about certain things—e.g., will the sun exist in 10 years—it would look high and flat. If the questions were about fundamentally uncertain things—e.g., what will the coin flip be 10 years from now—it would look low and flat. The slope we observe in that plot is less a property of how well the future can be predicted and more a property of the limited set of questions that were asked. If the questions were about uncertain near-term geopolitical events, then that graph shows the rate that information came in to the market consensus. It doesn’t really tell us about the bigger picture of predicting the future.
Incidentally, this was my biggest gripe with Tetlock and Gardner’s Superforecasting book. They spent a lot of time talking about how Superforecasters could predict the future, but almost no time talking about how the questions were selected and how if you choose different sets of counterfactual questions you can get totally different results (e.g., experts cannot predict the future vs rando smart people can predict the future). I don’t really fault them for this, because it’s a slippery thorny issue to discuss. I hope I have given you some flavor of it here.
I agree with most of what you’re saying, but this part seems like giving up way too easily: “And even if you say, ok sure it depends, but like what’s the average answer—even then, the only the way to arrive at some unbiased global sense of whether the future is predictable is to come up with some way of enumerating and weighing all possible facts about the future universe… which is an impossible problem. So we’re left with the unsatisfying truth that the future is neither predictable or unpredictable—it depends on which features of the future you are considering.”
The only way to say something useful about this is literally enumerating all possible facts? Sounds needlessly pessimistic.
On the contrary, I think it could be tremendously interesting and useful to start building some kind of categorization of prediction domains that allows saying something about their respective predictability. Obviously this is a hard problem, obviously the universe is very complex and the categorization will miss a lot of the intricacy, but the same is true about many other domains of knowledge (probably, nearly all of them). Despite the complexity of the universe (i) we should keep looking for (extremely) simplified models that capture a lot of what we actually care about (ii) having even an extremely simplified model is often much better than no model at all (iii) the model will keep evolving over time (which is to say, it feels more like a potential new science than a single problem that can be stated and solved in a relatively short time frame).
Thanks for this. The work I really want to see from more forecasting projects is an analysis of how much things that typically impact people’s lives can be predicted. Things like health, home-ownership, relationships, career, etc. Specifically, people’s levels of cooperate/defect against their future self seems really inconsistent. i.e. people work really hard for their future selves along certain dimensions and then defect along lots of others. This is mostly just mimetic momentum, but still. Even rigorous research figuring out exactly what actuaries know that can be applied practically by people would be good. After all, actuaries have really good life outcomes along lots of dimensions, which means that most aren’t taking advantage of the insights there.
My hope had been that 80k hours would have evolved to do more in this area but they’ve specialized narrower than that AFAICT.
I’ve been thinking similar things about predictability recently. Different variables have different levels of predictability, it seems very clear. I’m also under the impression that the examples in the Superforecasting study were quite specific. It seems likely that similar problems to what they studied have essentially low predictability 5-10 years out (and that is interesting information!), but this has limited relevance on other possible interesting questions.
While I agree with the specifics, I don’t think that the answer to a question like, “What is the average predictability of all possible statements” would be all that interesting. We generally care about a very small subset of “all possible statements.” It seems pretty reasonable to me that we could learn about the predictability of the kinds of things we’re interested. That said, i feel like we can get most of the benefits of this by just having calibrated forecasters try predicting all of these things, and seeing what their resolution numbers are. So I don’t think we need to do a huge amount of work running tests for the sole purpose of better understanding long-term predictability.
I left some longer comments in the EA Forum Post discussion.