Higher-order forecasting could be a useful concept for prediction markets and forecasting systems more broadly.
The core idea is straightforward: Nth-order forecasts are forecasts about (N-1)th order forecasts.
Abstract representation of 2nd-order forecasts. Dall-E 3
Examples
Here are some examples:
0-Order Forecasting (i.e., the ground truth)
Biden won the 2020 U.S. presidential election
The US GDP in 2023 was $27 trillion
1st-Order Forecasting (i.e., regular forecasting)
What is the chance that Trump will win the 2024 U.S. presidential election?
What will be the GDP of the US in 2024?
2nd-Order Forecasting
How much will the forecasts for US GDP in 2024 and 2025 be correlated over the next year?
How many forecasts will the question “What will be the GDP of the US in 2024?” receive in total?
If the question “What is the chance that a Republican will win the 2028 Presidential Election?” was posted to Manifold, with a subsidy of 100k Mana, what would the prediction be, after 1 month?”
3rd-Order Forecasting
How much will the forecasts, [How much will the forecasts for US GDP in 2024 and 2025 be correlated over the next year?] and [How many forecasts will the question “What will be the GDP of the US in 2024?” receive in total?], be correlated, from now until 2024?
How valuable were all the forecasts for the question, [‘How many forecasts will the question “What will be the GDP of the US in 2024?” receive in total?’]
As forecasting systems mature, higher-order forecasts could play a role analogous to financial derivatives in markets. Derivatives allow for more efficient pricing, risk transfer, and information aggregation by letting market participants express views on the relationships between assets. Similarly, higher-order forecasts could allow forecasters to express views on the relationships between predictions, leading to a more efficient and informative overall forecasting ecosystem.
Benefits
Some potential benefits of higher-order forecasting include:
Identify Overconfidence
Improve the accuracy of forecasts by having participants directly predict and get rewarded for estimating overconfidence or poor calibration in other forecasts.
“How overconfident is [forecast/forecaster] X”
Prioritize Questions
Prioritize the most important and decision-relevant questions by forecasting the value of information from different predictions.
“How valuable is the information from forecasting question X?”
Surface Relationships
Surface key drivers and correlations between events by letting forecasters predict how different questions relate to each other.
“How correlated will the forecasts for questions X and Y be over [time period]?”
Faster Information Aggregation
Enable faster aggregation of information by allowing forecasts on future forecast values, which may update more frequently than the underlying events.
“What will the forecast for question X be on [future date], conditional on [other forecasts or events]?”
Leverage Existing Infrastructure
Leverage the existing infrastructure and resolution processes of prediction platforms, which are already designed to handle large numbers of forecasting questions.
We’ve already seen some early examples of higher-order forecasts on platforms like Manifold Markets. For example, with the recent questions:
Of course, there are also challenges and risks to consider with higher-order forecasts:
The accuracy of higher-order forecasts depends on the accuracy of the lower-order forecasts they build on. If the underlying forecasts are poorly calibrated or noisy, that will limit the value of higher-order forecasts.
Higher-order forecasts inherently add complexity to forecasting systems, which could create challenges for participation, interpretation, and managing systemic risks.
A substantial base of lower-order forecasting questions is needed before higher-order forecasts can be productively created on top of them.
Alternative Names
I considered a few options for names, asked around a bit, and settled on “higher-order” for this term. Here are some other options I considered:
Derivatives: In the financial market, “markets about markets” are called derivatives. However, “derivative” is often understood as a term specific to markets, which could make it more confusing for forecasting.
Meta-forecasts: I used this term before. It’s a catchy term, but it doesn’t differentiate between layers easily. There’s no straightforward way to refer to “meta-layer 1.”
Higher-Layer: Similar to “higher-order,” but less precise.
If there’s contention on this later, it could be useful to have some formal discussion, to make sure that we share consistent terminology. Right now, I doubt many people care about it.
Conclusion
Over time, I expect higher-order forecasts to go from a niche idea to a key component of mature forecasting systems. Just as financial markets would be far less efficient without derivatives, forecasting platforms could see substantial accuracy and liquidity gains from higher-order forecasts.
Higher-Order Forecasts
Higher-order forecasting could be a useful concept for prediction markets and forecasting systems more broadly.
The core idea is straightforward:
Nth-order forecasts are forecasts about (N-1)th order forecasts.
Examples
Here are some examples:
0-Order Forecasting (i.e., the ground truth)
Biden won the 2020 U.S. presidential election
The US GDP in 2023 was $27 trillion
1st-Order Forecasting (i.e., regular forecasting)
What is the chance that Trump will win the 2024 U.S. presidential election?
What will be the GDP of the US in 2024?
2nd-Order Forecasting
How much will the forecasts for US GDP in 2024 and 2025 be correlated over the next year?
How many forecasts will the question “What will be the GDP of the US in 2024?” receive in total?
If the question “What is the chance that a Republican will win the 2028 Presidential Election?” was posted to Manifold, with a subsidy of 100k Mana, what would the prediction be, after 1 month?”
3rd-Order Forecasting
How much will the forecasts, [How much will the forecasts for US GDP in 2024 and 2025 be correlated over the next year?] and [How many forecasts will the question “What will be the GDP of the US in 2024?” receive in total?], be correlated, from now until 2024?
How valuable were all the forecasts for the question, [‘How many forecasts will the question “What will be the GDP of the US in 2024?” receive in total?’]
As forecasting systems mature, higher-order forecasts could play a role analogous to financial derivatives in markets. Derivatives allow for more efficient pricing, risk transfer, and information aggregation by letting market participants express views on the relationships between assets. Similarly, higher-order forecasts could allow forecasters to express views on the relationships between predictions, leading to a more efficient and informative overall forecasting ecosystem.
Benefits
Some potential benefits of higher-order forecasting include:
Identify Overconfidence
Improve the accuracy of forecasts by having participants directly predict and get rewarded for estimating overconfidence or poor calibration in other forecasts.
“How overconfident is [forecast/forecaster] X”
Prioritize Questions
Prioritize the most important and decision-relevant questions by forecasting the value of information from different predictions.
“How valuable is the information from forecasting question X?”
Surface Relationships
Surface key drivers and correlations between events by letting forecasters predict how different questions relate to each other.
“How correlated will the forecasts for questions X and Y be over [time period]?”
Faster Information Aggregation
Enable faster aggregation of information by allowing forecasts on future forecast values, which may update more frequently than the underlying events.
“What will the forecast for question X be on [future date], conditional on [other forecasts or events]?”
Leverage Existing Infrastructure
Leverage the existing infrastructure and resolution processes of prediction platforms, which are already designed to handle large numbers of forecasting questions.
We’ve already seen some early examples of higher-order forecasts on platforms like Manifold Markets. For example, with the recent questions:
Will there be a Manifold bot that makes profitable bets on random 1-month markets by December 2024? (Ṁ3,000 subsidy!)
Manifold Top Traders Leaderboard Ranking Prediction (2024)
If Manifold begins allowing real-money withdrawals, will its accuracy improve?
Is Manifold’s P(Doom) by 2050 currently between 10% and 90%? [Resolves to Poll]
Will Manifold be more accurate than real-money markets in forecasting the 2024 election?
Challenges
Of course, there are also challenges and risks to consider with higher-order forecasts:
The accuracy of higher-order forecasts depends on the accuracy of the lower-order forecasts they build on. If the underlying forecasts are poorly calibrated or noisy, that will limit the value of higher-order forecasts.
Higher-order forecasts inherently add complexity to forecasting systems, which could create challenges for participation, interpretation, and managing systemic risks.
A substantial base of lower-order forecasting questions is needed before higher-order forecasts can be productively created on top of them.
Alternative Names
I considered a few options for names, asked around a bit, and settled on “higher-order” for this term. Here are some other options I considered:
Derivatives: In the financial market, “markets about markets” are called derivatives. However, “derivative” is often understood as a term specific to markets, which could make it more confusing for forecasting.
Meta-forecasts: I used this term before. It’s a catchy term, but it doesn’t differentiate between layers easily. There’s no straightforward way to refer to “meta-layer 1.”
Higher-Layer: Similar to “higher-order,” but less precise.
If there’s contention on this later, it could be useful to have some formal discussion, to make sure that we share consistent terminology. Right now, I doubt many people care about it.
Conclusion
Over time, I expect higher-order forecasts to go from a niche idea to a key component of mature forecasting systems. Just as financial markets would be far less efficient without derivatives, forecasting platforms could see substantial accuracy and liquidity gains from higher-order forecasts.