There are a bunch of different metrics which you could look at on a prediction market / prediction platform to gauge how “uncertain” the forecast is:
Volatility—if the forecast is moving around quite a bit, there are two reasons:
Lots of new information arriving and people updating efficiently
There is little conviction around “fair value” so traders can move the price with little capital
Liquidity—if the market is 49.9 / 50.1 in millions of dollars, then you can be fairly confident that 50% is the “right” price. If the market is 40 / 60 with $1 on the bid and $0.50 on the offer, I probably wouldn’t be confident the probability lies between 40 and 60, let along “50% is the right number”. (The equivalent on prediction platforms might be number of forecasters, although CharlesD has done some research on this which suggests there’s little addition value being added by large numbers of forecasters)
“Spread of forecasts”—on Metaculus (for example) you can see a distribution of people’s forecasts. If everyone is tightly clustered around 50% that (usually) gives me more confidence that 50% is the right number than if they are widely spread out
All these indicators are definitely useful for a market observer. And betting on these indicators would make for an interesting derivatives market—especially on higher volume questions. The issue I was referring to is that all these indicators are still only based on traders who felt certain enough to bet on the market.
Say 100 people who have researched East-Asian geopolitics saw the question “Will China invade Taiwan this year?”. 20 did not feel confident enough to place a bet. Of the remaining 80 people, 20 bet small amounts because of their lack of certainty.
The market and most of the indicators you mentioned would be dominated by the 60 that placed large bets. A LOT of information about uncertainty would be lost. And this would have been fairly useful information about an event.
The goal would be to capture the uncertainty signal of the 40 that did not place bets, or placed small bets. One way to do that would be to make “uncertainty” itself a bettable property of the question. And one way to accomplish that would be to bet on what percentage of bets are on “uncertainty” vs. a prediction.
And one way to accomplish that would be to bet on what percentage of bets are on “uncertainty” vs. a prediction.
How do you plan on incentivising people to bet on “uncertainty”? All the ways I can think of lead to people either gaming the index, or turning uncertainty into a KBC.
The market and most of the indicators you mentioned would be dominated by the 60 that placed large bets
I disagree with this. Volatility, liquidity, # predictors, spread of forecasts will all be affected by the fact that 20 people aren’t willing to get involved. I’m not sure what information you think is being lost by people stepping away? (I guess the difference between “the market is wrong” and “the market is uninteresting”?)
What is being lost is related to your intuition in the earlier comment:
if the market is 49.9 / 50.1 in millions of dollars, then you can be fairly confident that 50% is the “right” price.
Without knowing how many people of the “I’ve studied this subject, and still don’t think a reasonable prediction is possible” variety didn’t participate in the market, it’s very hard to place any trust in it being the “right” price.
This is similar to the “pundit” problem where you are only hearing from the most opinionated people. If 60 nutritionist are on TV and writing papers saying eating fats is bad, you may try to draw the “wrong” conclusion from that.; because unknown to you, 40 nutritionists believe “we just don’t know yet”. And these 40 are provided no incentives to say so.
Take the Russia-Kiev question on Metaculus which had a large number of participants. It hovered at 8% for a long time. If prediction markets are to be useful beyond just pure speculation, that market didn’t tell me how many knowledgable people thought an opinion was simply not possible.
The ontological skepticism signal is missing—people saying there is no right or wrong that “exists”—we just don’t know. So be skeptical of what this market says.
As for KBC—most markets allow you to change/sell your bet before the event happens; especially for longer-term events. So my guess is that this is already happening. In fact, the uncertainty index would seperate out much of the “What do other people think?” element into it’s own question.
For locked in markets like ACX where the suggestion is to leave your prediction blank if you don’t know, imagine every question being paired with “What percentage of people will leave this prediction blank?”
There are a bunch of different metrics which you could look at on a prediction market / prediction platform to gauge how “uncertain” the forecast is:
Volatility—if the forecast is moving around quite a bit, there are two reasons:
Lots of new information arriving and people updating efficiently
There is little conviction around “fair value” so traders can move the price with little capital
Liquidity—if the market is 49.9 / 50.1 in millions of dollars, then you can be fairly confident that 50% is the “right” price. If the market is 40 / 60 with $1 on the bid and $0.50 on the offer, I probably wouldn’t be confident the probability lies between 40 and 60, let along “50% is the right number”. (The equivalent on prediction platforms might be number of forecasters, although CharlesD has done some research on this which suggests there’s little addition value being added by large numbers of forecasters)
“Spread of forecasts”—on Metaculus (for example) you can see a distribution of people’s forecasts. If everyone is tightly clustered around 50% that (usually) gives me more confidence that 50% is the right number than if they are widely spread out
All these indicators are definitely useful for a market observer. And betting on these indicators would make for an interesting derivatives market—especially on higher volume questions. The issue I was referring to is that all these indicators are still only based on traders who felt certain enough to bet on the market.
Say 100 people who have researched East-Asian geopolitics saw the question “Will China invade Taiwan this year?”. 20 did not feel confident enough to place a bet. Of the remaining 80 people, 20 bet small amounts because of their lack of certainty.
The market and most of the indicators you mentioned would be dominated by the 60 that placed large bets. A LOT of information about uncertainty would be lost. And this would have been fairly useful information about an event.
The goal would be to capture the uncertainty signal of the 40 that did not place bets, or placed small bets. One way to do that would be to make “uncertainty” itself a bettable property of the question. And one way to accomplish that would be to bet on what percentage of bets are on “uncertainty” vs. a prediction.
How do you plan on incentivising people to bet on “uncertainty”? All the ways I can think of lead to people either gaming the index, or turning uncertainty into a KBC.
I disagree with this. Volatility, liquidity, # predictors, spread of forecasts will all be affected by the fact that 20 people aren’t willing to get involved. I’m not sure what information you think is being lost by people stepping away? (I guess the difference between “the market is wrong” and “the market is uninteresting”?)
What is being lost is related to your intuition in the earlier comment:
Without knowing how many people of the “I’ve studied this subject, and still don’t think a reasonable prediction is possible” variety didn’t participate in the market, it’s very hard to place any trust in it being the “right” price.
This is similar to the “pundit” problem where you are only hearing from the most opinionated people. If 60 nutritionist are on TV and writing papers saying eating fats is bad, you may try to draw the “wrong” conclusion from that.; because unknown to you, 40 nutritionists believe “we just don’t know yet”. And these 40 are provided no incentives to say so.
Take the Russia-Kiev question on Metaculus which had a large number of participants. It hovered at 8% for a long time. If prediction markets are to be useful beyond just pure speculation, that market didn’t tell me how many knowledgable people thought an opinion was simply not possible.
The ontological skepticism signal is missing—people saying there is no right or wrong that “exists”—we just don’t know. So be skeptical of what this market says.
As for KBC—most markets allow you to change/sell your bet before the event happens; especially for longer-term events. So my guess is that this is already happening. In fact, the uncertainty index would seperate out much of the “What do other people think?” element into it’s own question.
For locked in markets like ACX where the suggestion is to leave your prediction blank if you don’t know, imagine every question being paired with “What percentage of people will leave this prediction blank?”