Why would we expect an online prediction market to work in this instance? The people with the most information aren’t going to be participating, either because they are in a war zone or because of restrictions on acting on classified data.
Judging by the Metaculus’ track record, it is surprisingly good in most cases.
In particular, it gave a >50% probability of the invasion more than 1 month before it has started. And 4 days before the invasion, it gave 75% (source).
Of course, the invasion plans were classified and carefully guarded, even from some members of the Russian gov. But publicly accessible data were sufficient to give a good forecast anyway.
I’m closely monitoring the war, and from my understanding of the situation, Metaculus is right: both a ceasefire and a capture of Kyiv are unlikely to happen in the next few months. Thus, the war is unlikely to stop anytime soon.
If the information doesn’t get to the forecasters, a typical person shouldn’t expect to be able to obtain that data themselves, so the forecast can still be your best option. If you do have information that others don’t, you can always adjust the forecasts you see accordingly.
Tetlock showed that superforcasters can outperform people with access to classified data. You might want to read his book if you are interested in the details.
Yes, but in a highly dynamic, rapidly evolving situation where the classified data is extremely asymmetric with respect to open sources? A CIA analyst will have daily or even hourly satellite photo updates, real-time access to Russian and Ukrainian communications and transmitter positions, the non-propaganda numbers for functioning military equipment and losses, not to mention all sorts of human intel on the ground. Someone outside of that world and not on the ground in Ukraine has very limited access to information—mostly propaganda from both sides. It’s really hard to believe you can extract a better signal from that.
I can believe that prediction markets can outperform a handful of analysts with access to classified data in certain contexts. But this situation appears to me to be maximally weighted in favor of the analysts.
After screwing up predictions about Iraqi WMDs the CIA believed that they have a problem. That lead to IARPA making a tournament for different approaches to making predictions. Tedlock’s Good Judgement project won that tournament.
I think you can make an argument that Metaculus community predictions aren’t really reflecting superforcasting predictions as there are plenty of people who are making predictions without that skillset but a well-functioning prediction market wouldn’t have that problem.
Why would we expect an online prediction market to work in this instance? The people with the most information aren’t going to be participating, either because they are in a war zone or because of restrictions on acting on classified data.
Judging by the Metaculus’ track record, it is surprisingly good in most cases.
In particular, it gave a >50% probability of the invasion more than 1 month before it has started. And 4 days before the invasion, it gave 75% (source).
Of course, the invasion plans were classified and carefully guarded, even from some members of the Russian gov. But publicly accessible data were sufficient to give a good forecast anyway.
I’m closely monitoring the war, and from my understanding of the situation, Metaculus is right: both a ceasefire and a capture of Kyiv are unlikely to happen in the next few months. Thus, the war is unlikely to stop anytime soon.
If the information doesn’t get to the forecasters, a typical person shouldn’t expect to be able to obtain that data themselves, so the forecast can still be your best option. If you do have information that others don’t, you can always adjust the forecasts you see accordingly.
Tetlock showed that superforcasters can outperform people with access to classified data. You might want to read his book if you are interested in the details.
Yes, but in a highly dynamic, rapidly evolving situation where the classified data is extremely asymmetric with respect to open sources? A CIA analyst will have daily or even hourly satellite photo updates, real-time access to Russian and Ukrainian communications and transmitter positions, the non-propaganda numbers for functioning military equipment and losses, not to mention all sorts of human intel on the ground. Someone outside of that world and not on the ground in Ukraine has very limited access to information—mostly propaganda from both sides. It’s really hard to believe you can extract a better signal from that.
I can believe that prediction markets can outperform a handful of analysts with access to classified data in certain contexts. But this situation appears to me to be maximally weighted in favor of the analysts.
After screwing up predictions about Iraqi WMDs the CIA believed that they have a problem. That lead to IARPA making a tournament for different approaches to making predictions. Tedlock’s Good Judgement project won that tournament.
I think you can make an argument that Metaculus community predictions aren’t really reflecting superforcasting predictions as there are plenty of people who are making predictions without that skillset but a well-functioning prediction market wouldn’t have that problem.