Metaculus had only a 8% chance of “Russian Troops in Kyiv in 2022” as late as Feb 11. (It’s now at 97%.) Why did everyone do so badly on this prediction?
Speaking from ignorance: this prediction failure seems (from my ignorant perspective) similar to forecasting failures in Brexit and in the Trump 2016 election, in that it’s a case where some force whose motives are unlike Western academia/elites was surprising to them/us. If so, the moral might be to study the perspectives, motives, and capabilities of forces outside the Western elite on their own terms / by cobbling together an inside view from primary sources, rather than looking via Western experts/media (though this is much harder).
To what extent are there people who visibly made good predictions here beforehand? It seems worth compiling them. I appreciate cmessinger’s comment.
In this vein, here’s someone calling the situation we’re seeing shortly after Russia’s annexation of Crimea in 2014. Basic premise being that great power competition is alive and well, and the US and European leaders trying to tell themselves it’s dead are consistently confused. Title aside I would not describe his talk as assigning blame—more that the US had a great hand after the collapse of the USSR and has played it poorly. I found a few talks from this guy, and they’ve all aged remarkably well.
Even the honest-to-god financial markets did badly on this prediction. The moment Russia started invading, MOEX and oil prices had wild shocks, as if something unexpected had happened, even though we already had abundant ahead-of-time warnings about their attack from U.S. intelligence. My suspicion is that everyone, including market analysts, expected Russia to simply annex the seperatist eastern terrority of Donbas. They expected the war to end quickly and with little bloodshed, and for the Ukrainian government to capitulate. The presumption was that they’d be fighting over a small separatist province, and it would have made sense—Putin gets what he wants, which is to look powerful, and he can come back again in 5-10 years for another piece.
Instead, Putin has chosen to do something insane—he is forcing his armed forces to attack and at least temporarily occupy the entirety of Ukraine, including Kyiv, Lziz, and Odessa. These are centers of the Ukrainian resistance that nobody, not even a brainwashed Russian populace, could possibly believe are secretly interested in unification. Maybe he underestimated the response or maybe he doesn’t care, but now western countries are thinking of kicking Russia out of their banking systems entirely, and the Ukrainian leadership has decided to face death at the hands of their oppressors instead of desert their country. There’s going to be a total war—perhaps a quick one, but many Ukrainians and Russians will die. I wonder if Vladimir Putin actually anticipated this would be the consequences of his actions. It gestures suggestively to a God complex that might be undermining the decision making of Russia’s governor and his coconspirators.
I wonder if in the aftermath we will be able to decipher the scope of the increase in risk, financially, between the separatist-territories and full-invasion scenarios. I imagine this is questions like $ of assets destroyed or put offline, in addition to the question of sanctions.
Or perhaps Putin has a more accurate world-model and correctly predicted that the world will just sit and watch? Could change any moment of course, but at least thus far it seems he’s right about Western threats of severe sanctions being hollow.
1. Mistakes about the magnitude of the power centralization in Russia.
I don’t know how it was perceived in other countries but in Russia there were many debates about how many powers belongs to Putin himself. Real process of decision-making was hidden and people had different hypotheses about it. Many people thought about Putin as an arbiter between oligarchs/other forces or as first among peers.
As far as I understand last week broadcast from Russian Security Council was very surprising for many people. Openly Putin practically humiliated some high officials (especially Chief of Intelligence Service).
2. Mistakes about Putin’s motives.
I think a problem is a changing in Putin. In the beginnings of his rule Putin demonstrated that he is very pragmatic. He said words which sounds very reasonably. He very rare did something non-reversible. I think all missed a moment when this was changed.
3. I suppose people underestimate the magnitude of Russian intelligence service degradation.
I think in Russian defense agencies people were used to say things which their superiors wants to hear. I think Putin understands Ukrainians very inadequately because he read many reports which only confirm his point of view.
My baseline assumption for this is that everyone always does badly on these kinds of predictions. This frequently includes highly trained defense analysts with access to privileged information sources.
I suspect, at a gut level, that this is mostly because markets and crowds work on publicly available information and national security is pretty much the foremost subject for inside-view information being kept from the public.
I saw people discussing forecasting success of this on twitter and people were saying that the intelligence agencies actually called this right. Does anyone know an easy link to what those agencies were saying?
My baseline assumption for this is that everyone always does badly on these kinds of predictions. This frequently includes highly trained defense analysts with access to privileged information sources.
I don’t think this answers the question. 8% is an awfully strong prediction. Your meta-point is public info. People who honor your meta-point could make a killing in markets as badly miscalibrated as this one. That would make it profitable for people to correct the markets.
How can we expect well-calibrated markets on questions with so few events? It appears to me there is a ceiling on how good we can expect calibration to be here, and it is much lower than the average Metaculus question.
Of course, reflecting on this suggests to me we should be able to take a serious crack at the answer after this one, with more to follow afterward.
Off the top of my head, maybe it’s because Metaculus’ presents medians, and the median user neither investigates the issue much, nor trusts those who do (Matt Y, Scott A) and just roughly follows base rates. I also feel there was some wishful thinking, and that to some extent, the fullness of the invasion was at least somewhat intrinsically surprising.
Another possibility is that most people were reluctant to read, summarise, or internalise Putin’s writing on Ukraine due to finding it repugnant, because they aren’t decouplers.
Welp, I was wrong: the Feb 11 shows a 45% likely invasion before 2023, which is not consistent with the won’t-invade scenario I proposed.
It is consistent with a moderate expectation of another Crimea level event, but not a full-scale invasion. Do we have a way of determining if the same participants are betting across multiple of these questions, or if the answerers are mostly unique?
Metaculus had only a 8% chance of “Russian Troops in Kyiv in 2022” as late as Feb 11. (It’s now at 97%.) Why did everyone do so badly on this prediction?
Speaking from ignorance: this prediction failure seems (from my ignorant perspective) similar to forecasting failures in Brexit and in the Trump 2016 election, in that it’s a case where some force whose motives are unlike Western academia/elites was surprising to them/us. If so, the moral might be to study the perspectives, motives, and capabilities of forces outside the Western elite on their own terms / by cobbling together an inside view from primary sources, rather than looking via Western experts/media (though this is much harder).
To what extent are there people who visibly made good predictions here beforehand? It seems worth compiling them. I appreciate cmessinger’s comment.
In this vein, here’s someone calling the situation we’re seeing shortly after Russia’s annexation of Crimea in 2014. Basic premise being that great power competition is alive and well, and the US and European leaders trying to tell themselves it’s dead are consistently confused. Title aside I would not describe his talk as assigning blame—more that the US had a great hand after the collapse of the USSR and has played it poorly. I found a few talks from this guy, and they’ve all aged remarkably well.
A LW post predicted a Feb invasion back in Dec: https://www.lesswrong.com/posts/QEsqKFabffwKXAPso/
Years ago, many politicians and professors of international relations predicted a strong Russian response to talk of admitting Ukraine into NATO: https://nitter.net/RnaudBertrand/status/1498491107902062592#m
Even the honest-to-god financial markets did badly on this prediction. The moment Russia started invading, MOEX and oil prices had wild shocks, as if something unexpected had happened, even though we already had abundant ahead-of-time warnings about their attack from U.S. intelligence. My suspicion is that everyone, including market analysts, expected Russia to simply annex the seperatist eastern terrority of Donbas. They expected the war to end quickly and with little bloodshed, and for the Ukrainian government to capitulate. The presumption was that they’d be fighting over a small separatist province, and it would have made sense—Putin gets what he wants, which is to look powerful, and he can come back again in 5-10 years for another piece.
Instead, Putin has chosen to do something insane—he is forcing his armed forces to attack and at least temporarily occupy the entirety of Ukraine, including Kyiv, Lziz, and Odessa. These are centers of the Ukrainian resistance that nobody, not even a brainwashed Russian populace, could possibly believe are secretly interested in unification. Maybe he underestimated the response or maybe he doesn’t care, but now western countries are thinking of kicking Russia out of their banking systems entirely, and the Ukrainian leadership has decided to face death at the hands of their oppressors instead of desert their country. There’s going to be a total war—perhaps a quick one, but many Ukrainians and Russians will die. I wonder if Vladimir Putin actually anticipated this would be the consequences of his actions. It gestures suggestively to a God complex that might be undermining the decision making of Russia’s governor and his coconspirators.
I wonder if in the aftermath we will be able to decipher the scope of the increase in risk, financially, between the separatist-territories and full-invasion scenarios. I imagine this is questions like $ of assets destroyed or put offline, in addition to the question of sanctions.
Or perhaps Putin has a more accurate world-model and correctly predicted that the world will just sit and watch? Could change any moment of course, but at least thus far it seems he’s right about Western threats of severe sanctions being hollow.
I think about several reasons:
1. Mistakes about the magnitude of the power centralization in Russia.
I don’t know how it was perceived in other countries but in Russia there were many debates about how many powers belongs to Putin himself. Real process of decision-making was hidden and people had different hypotheses about it. Many people thought about Putin as an arbiter between oligarchs/other forces or as first among peers.
As far as I understand last week broadcast from Russian Security Council was very surprising for many people. Openly Putin practically humiliated some high officials (especially Chief of Intelligence Service).
2. Mistakes about Putin’s motives.
I think a problem is a changing in Putin. In the beginnings of his rule Putin demonstrated that he is very pragmatic. He said words which sounds very reasonably. He very rare did something non-reversible. I think all missed a moment when this was changed.
3. I suppose people underestimate the magnitude of Russian intelligence service degradation.
I think in Russian defense agencies people were used to say things which their superiors wants to hear. I think Putin understands Ukrainians very inadequately because he read many reports which only confirm his point of view.
My baseline assumption for this is that everyone always does badly on these kinds of predictions. This frequently includes highly trained defense analysts with access to privileged information sources.
I suspect, at a gut level, that this is mostly because markets and crowds work on publicly available information and national security is pretty much the foremost subject for inside-view information being kept from the public.
I saw people discussing forecasting success of this on twitter and people were saying that the intelligence agencies actually called this right. Does anyone know an easy link to what those agencies were saying?
Context: https://twitter.com/ClayGraubard/status/1496699988801433602?s=20&t=mQ8sAzMRppI8Pr44O38M3w
https://twitter.com/ClayGraubard/status/1496866236973658112?s=20&t=mQ8sAzMRppI8Pr44O38M3w
The thing about intelligence agencies is that they are really good at insider trading.
A broken clock is right twice a day. If you always predict war in a crisis, you’ll eventually be right.
They do not often predict war this publicly.
I don’t think this answers the question. 8% is an awfully strong prediction. Your meta-point is public info. People who honor your meta-point could make a killing in markets as badly miscalibrated as this one. That would make it profitable for people to correct the markets.
So the question stands:
WTF?
How can we expect well-calibrated markets on questions with so few events? It appears to me there is a ceiling on how good we can expect calibration to be here, and it is much lower than the average Metaculus question.
Of course, reflecting on this suggests to me we should be able to take a serious crack at the answer after this one, with more to follow afterward.
Off the top of my head, maybe it’s because Metaculus’ presents medians, and the median user neither investigates the issue much, nor trusts those who do (Matt Y, Scott A) and just roughly follows base rates. I also feel there was some wishful thinking, and that to some extent, the fullness of the invasion was at least somewhat intrinsically surprising.
Another possibility is that most people were reluctant to read, summarise, or internalise Putin’s writing on Ukraine due to finding it repugnant, because they aren’t decouplers.
A second observation:
How is this prediction inconsistent with a combination of these two beliefs:
The Russians are unlikely to invade, and in that case the probability of Russian troops in the capital is ~0
In the counterfactual case where the Russians do invade, the probability of Russian troops in the capital is ~1
For example, Metaculus has a 97% chance of Russians in Kyiv, but a Russian invasion at all before 2023 is at 96%:
Welp, I was wrong: the Feb 11 shows a 45% likely invasion before 2023, which is not consistent with the won’t-invade scenario I proposed.
It is consistent with a moderate expectation of another Crimea level event, but not a full-scale invasion. Do we have a way of determining if the same participants are betting across multiple of these questions, or if the answerers are mostly unique?