The DAGGRE project is based on just that, decomposition of forecasts. This PDF explains how it works. It’s an interesting approach, and the reason I mentioned in an aside, in part 1, that I might have liked to join that team.
The GJP, on the other hand, uses different tools—as I understand it some teams have “survey” type interfaces, where we enter just a probability and our reasoning, other teams have “prediction market” interfaces.
I don’t personally find it very useful (yet?) to explicitly decompose my forecasts.
For instance a recent question was “Will the sentence of any of the three members of the band Pussy Riot who were convicted of hooliganism be reduced, nullified, or suspended before 1 December 2012?” It’s not clear how you’d decompose that:
chance that each individual girl member of PR would have her sentence reduced
chance for each possible grounds for a sentence reduction
chance for each possible political influence on sentencing (public opinion, Putin, Medvedev)
ISTM that making a fine-grained forecast on any of the above is to presume way too much of my detailed knowledge of the situation. Maybe someone close to the case might have predicted that Yekaterina would walk while the other two would serve a full sentence. The reason given was “because she was thrown out of the cathedral by guards before she could remove her guitar from its case and take part in the performance.” I only learned about that just now, looking at news reports on the appeal result; this was never mentioned previously.
So, I don’t know how I feel about decomposition. What I’m reminded of is the distinction between “fox” and “hedgehog” approaches that originated with Tetlock and which Silver discusses in his book: “Hedgehogs know one big thing, while foxes know many little things.”
Silver says that a “fox” usually does better because they approach different predictions in different ways and bring a variety of perspectives to each, whereas the “hedgehog” tends to be more ideological, to insist that there is One True Way to tackle every forecasst. The decomposition approach strikes me as less fox-y and more hedgehog-gy.
The results of the questionnaire I filled when I joined GJP identified me as more of a hedgehog: 4.5 on a 1-7 scale, compared to a mean of 3.81, SD .52. I’m pretty sure that my actual forecasting behavior, at least this year, is foxier.
I also scored slightly on the hedgehog scale. I think people who like to “think about thinking” are already slightly hedgehog. True foxes don’t believe in such grand theories.
The decomposition approach seems more foxy to me because it applies more specific details and approaches different aspects of the problems in different ways. The analogy I see is that the decomposition approach is similar to using a Taylor series to approximate the slope of a function in calculus, whereas the other approach is much less fine tuned. Except it’s more like using multiple Taylor series’ in a piece wise function than just a vanilla Taylor series, so the analogy isn’t really great.
With Pussy Riot, I would just say that you can identify the macro factors which effect the probability of conviction, and then determine the probability that those macro factors exist by estimating the probability that they change. So, for instance, you could learn who their judge would be and what that judge’s rulings have been like in the past. You could try to get a rough understanding of that judge’s thought process. You could do the same with lawyers. You could check what time or what month the trial was scheduled for, and then apply data on how judges tend to be kinder when it’s the beginning of the day or how when the weather is nice (this latter factor is more of a guess but it seems to me like it would be accurate).
Those specific factors I mentioned are actually somewhat weak and wouldn’t have much overall effect. But they’re intended as examples of what can be done in a more general sense. Although I agree that the Pussy Riot conviction specifically would be difficult to predict, and although I’m doing a bad job of explaining this specific case, I think that’s more because I haven’t really been following that news story at all, because it seemed boring to me. If I had done more reading on it, even if I didn’t have many more specific details, then I think I would be able to identify more powerful factors that a decomposition approach could use.
You mention that you often don’t have specific information, but I think that there’s general information which is available that can be applied to specific aspects of the decomposition, and that this would improve the accuracy of the prediction.
To use a different example that I feel more comfortable with, let’s say we had to evaluate the probability that the current government of China collapses by 2035. Economic trends are going to be a huge factor in this. We can break that down further. Here are some major things that will determine the economic health of China: 1. Demographics, what China does with its aging population and what the economic effects of the one-child policy and lack of females are. 2. Environmental health, the environment is the foundation of the economy. 3. Infrastructure is important.
So, looking at 1, we can say that there’s a high probability that China will at least initially attempt to support its aging population. However, the population is so old that I can’t see this working for long. I think China will lose lots of money on its elderly people, but then will just let them die (this would have social consequences, those should be put as inputs into a different section of our overall prediction). I also think that China is going to have much of its male population move overseas, and that it’s going to start encouraging foreign females to immigrate. These policies won’t be very successful because China is awful. Overall, China will still lose lots of money.
Looking at 2, China’s environment is trash, and their priority is on short term growth, so this will stay the same for the foreseeable future. Unless China gets some bold new leader who is so charismatic he can keep the people from revolting while their short term growth slows and who implements major environmental reforms, then China is doomed. I don’t think that any such leaders exist in China right now. However, another solution would arise if some fantastic new technology allowed China to change the basis of its production system. I don’t know what the probability of cheap and widespread nanotech or something similar by 2035 is, but I’m going to assume that it’s low and so this also won’t happen and so China’s environment is in trouble.
Looking at 3, China won’t be able to maintain infrastructure as a consequence of 1 and 2. This will aggravate underlying problems. Also, China’s already running into problems because other countries in the region are trying to take the cheap industrial production section of the market China used to have. China’s been trying to switch towards a more technologically advanced production economy, but their education is subpar. This will get worse during the aging crisis.
China is basically dependent on economic growth to keep its population from revolting, from what I’ve read.
So I conclude that unless something very important happens with China’s leadership or cultural/social practices, the economy of China will collapse by 2035 and China will then disintegrate into a civil war. I’m going to put about a 80% probability estimate on this one. This is what I actually believe, although this is sort of an incomplete description and also it oversimplifies and formalizes a lot of what is just raw intuition in my head.
This example also hopefully makes clear the risks of infinite regression. Stuff is complicated.
So I conclude that unless something very important happens with China’s leadership or cultural/social practices, the economy of China will collapse by 2035 and China will then disintegrate into a civil war. I’m going to put about a 80% probability estimate on this one.
The “unless something very important” hedge makes this prediction rather hard to judge: it’s vague enough that anyone might confidently predict “something very important happens with China’s leadership or cultural/social practices between now and 2035″, because something is bound to happen that arguably falls under this specification.
This is kind of a converse rule of thumb to the “prefer status quo”, which—I should maybe have said—is valid at fairly short timeframes. Over long enough timeframes, the reverse is true—pretty much everything we can specify is going to change. (Over long enough timeframes, continents will move, mountains will be eroded to sea level, and eventually the planet itself will no longer exist.)
Over a twenty year timeframe, if you phrase your prediction in terms of (say) “the rate of exchange between USD and CNY”, there is some chance that one of the currencies will cease to exist, or that the notion of “rate of exchange” will stop making sense.
there is some chance that one of the currencies will cease to exist
Nitpick: I don’t think this is much beyond epsilon. Even if a currency ‘ceases to exist’, the currency still exists in whatever physical embodiment. There is still an exchange rate from Saddam’s dinars to US dollars: it’s however many pieces of old paper $1 will buy you on eBay. The exchange rates for currencies that cease to exist can even go up over time (how much would a American Revolution-era shinplaster cost ya?).
Ooh. As long as I’m making predictions, I believe Romney will defy the polls and win. 65% chance I’m right.
The economy is doing poorly.
Obama is an incumbent.
Romney is taller.
I think polls (and 538) are overestimating what Democratic participation will be like.
These are the only factors I considered. I intentionally took a different approach to this prediction.
This prediction is very hedgehoggy, but I think that is the right tool for the job here. I feel comfortable doing that here but not in China because I believe macro trends work much better as predictors of things like human psychology and things like democratic votes. In contrast, I don’t think I can form a good reference class to put China (2012) in that will allow me to evaluate the probability it collapses in, so I chose to instead use speciifc data. Also, I’ve examined specific facts about China, but I don’t feel confident in my ability to do that when it comes to the elections, because of media bias and cultural biases.
It’s far too late to get in on IEM or InTrade action—with like 24 days to go, it’d take almost all of that time to deposit funds via check or get ID cleared.
That said, I’d be happy to wager with chaosmosis that Romney will lose and Obama will win- say his $30 against my $20?
I’m confident, but not meta-level confident, because I haven’t been doing this for a while and I’m using new techniques and the experts are against me. I also have low risk tolerance for cash losses.
Additionally, you’re probably putting the terms of the bargain like that because I said 65% chance, but I expect that you’ve got an even higher chance in your mind that Obama would win. Obama winning is conventional wisdom, I would only accept a bet that rewarded me for being an outlier.
but I expect that you’ve got an even higher chance in your mind that Obama would win. Obama winning is conventional wisdom, I would only accept a bet that rewarded me for being an outlier.
Fine—you drive a hard bargain! $20:$20 or $30:$30?
Not gonna do it. Maybe that means I should revise my estimates, or that I have irrational feelings about bets in general. I’ll figure that out later.
I notice that I would be willing to bet on China, that’s because: 1. I’ve looked through the data more and taken a foxy approach to it. 2. I think that China’s fate is less subject to random events 3. I’m not really against conventional wisdom with China. 4. I feel more metalevel confident about my ability to predict world affairs, I’m not sure why though.
I think the most likely reason that I’m not wanting to bet is that I don’t know whether or not I’m safe in doubting conventional wisdom and current polls and most experts in this specific field, and that I actually suspect that I’m not. For now I’ll revise my estimates, I’ll still incorporate my move towards Romney from the mainstream but I’ll start at the mainstream, which would be about 80% Obama, and I’ll move towards Romney, to 60% for Obama.
Also, I might have been in backlash mode or trying to be TOO pessimistic when I made the initial prediction that Obama would lose. Either I’ve gotten deeper into the liberal echo chamber from redditing and whatnot and my perceptions are getting distorted, or else I was being too pessimistic before (because I tried to force factors like height into my model, and I assumed that substance was basically irrelevant to political success, but now I’m not so sure.) I think I wanted to see myself as pessimistic so I wrote off the mainstream opinion and maybe I overdid it.
Additionally, I’m not wanting to go through the hassle of getting a paypal or whatever might happen. I would like the information on this (whatever the standard betting venue or monetary exchange thingy on LessWrong would be, if there is one, even though I’m not going to be betting here). I think I might be scared that I’ll lose the bet and then decide to not pay you, also, and that’s something else that’s making me reluctant to bet. Now that I think about it, I should also get an Intrade account in advance now.
There are many complicating factors. Demands for money helped me identify some.
FWIW, my own Obama probability is more like 65-70%.
Additionally, I’m not wanting to go through the hassle of getting a paypal or whatever might happen. I would like the information on this (whatever the standard betting venue or monetary exchange thingy on LessWrong would be, if there is one, even though I’m not going to be betting here). I think I might be scared that I’ll lose the bet and then decide to not pay you, also, and that’s something else that’s making me reluctant to bet.
Bets are typically paid as Paypal, Bitcoin, or donations to specified charities.
Now that I think about it, I should also get an Intrade account in advance now.
I personally don’t use Intrade because the new fees make it very expensive for low-stakes long-term bets like most bets on LW (assuming the market even exists).
The DAGGRE project is based on just that, decomposition of forecasts. This PDF explains how it works. It’s an interesting approach, and the reason I mentioned in an aside, in part 1, that I might have liked to join that team.
The GJP, on the other hand, uses different tools—as I understand it some teams have “survey” type interfaces, where we enter just a probability and our reasoning, other teams have “prediction market” interfaces.
I don’t personally find it very useful (yet?) to explicitly decompose my forecasts.
For instance a recent question was “Will the sentence of any of the three members of the band Pussy Riot who were convicted of hooliganism be reduced, nullified, or suspended before 1 December 2012?” It’s not clear how you’d decompose that:
chance that each individual girl member of PR would have her sentence reduced
chance for each possible grounds for a sentence reduction
chance for each possible political influence on sentencing (public opinion, Putin, Medvedev)
ISTM that making a fine-grained forecast on any of the above is to presume way too much of my detailed knowledge of the situation. Maybe someone close to the case might have predicted that Yekaterina would walk while the other two would serve a full sentence. The reason given was “because she was thrown out of the cathedral by guards before she could remove her guitar from its case and take part in the performance.” I only learned about that just now, looking at news reports on the appeal result; this was never mentioned previously.
So, I don’t know how I feel about decomposition. What I’m reminded of is the distinction between “fox” and “hedgehog” approaches that originated with Tetlock and which Silver discusses in his book: “Hedgehogs know one big thing, while foxes know many little things.”
Silver says that a “fox” usually does better because they approach different predictions in different ways and bring a variety of perspectives to each, whereas the “hedgehog” tends to be more ideological, to insist that there is One True Way to tackle every forecasst. The decomposition approach strikes me as less fox-y and more hedgehog-gy.
The results of the questionnaire I filled when I joined GJP identified me as more of a hedgehog: 4.5 on a 1-7 scale, compared to a mean of 3.81, SD .52. I’m pretty sure that my actual forecasting behavior, at least this year, is foxier.
I also scored slightly on the hedgehog scale. I think people who like to “think about thinking” are already slightly hedgehog. True foxes don’t believe in such grand theories.
The decomposition approach seems more foxy to me because it applies more specific details and approaches different aspects of the problems in different ways. The analogy I see is that the decomposition approach is similar to using a Taylor series to approximate the slope of a function in calculus, whereas the other approach is much less fine tuned. Except it’s more like using multiple Taylor series’ in a piece wise function than just a vanilla Taylor series, so the analogy isn’t really great.
With Pussy Riot, I would just say that you can identify the macro factors which effect the probability of conviction, and then determine the probability that those macro factors exist by estimating the probability that they change. So, for instance, you could learn who their judge would be and what that judge’s rulings have been like in the past. You could try to get a rough understanding of that judge’s thought process. You could do the same with lawyers. You could check what time or what month the trial was scheduled for, and then apply data on how judges tend to be kinder when it’s the beginning of the day or how when the weather is nice (this latter factor is more of a guess but it seems to me like it would be accurate).
Those specific factors I mentioned are actually somewhat weak and wouldn’t have much overall effect. But they’re intended as examples of what can be done in a more general sense. Although I agree that the Pussy Riot conviction specifically would be difficult to predict, and although I’m doing a bad job of explaining this specific case, I think that’s more because I haven’t really been following that news story at all, because it seemed boring to me. If I had done more reading on it, even if I didn’t have many more specific details, then I think I would be able to identify more powerful factors that a decomposition approach could use.
You mention that you often don’t have specific information, but I think that there’s general information which is available that can be applied to specific aspects of the decomposition, and that this would improve the accuracy of the prediction.
To use a different example that I feel more comfortable with, let’s say we had to evaluate the probability that the current government of China collapses by 2035. Economic trends are going to be a huge factor in this. We can break that down further. Here are some major things that will determine the economic health of China: 1. Demographics, what China does with its aging population and what the economic effects of the one-child policy and lack of females are. 2. Environmental health, the environment is the foundation of the economy. 3. Infrastructure is important.
So, looking at 1, we can say that there’s a high probability that China will at least initially attempt to support its aging population. However, the population is so old that I can’t see this working for long. I think China will lose lots of money on its elderly people, but then will just let them die (this would have social consequences, those should be put as inputs into a different section of our overall prediction). I also think that China is going to have much of its male population move overseas, and that it’s going to start encouraging foreign females to immigrate. These policies won’t be very successful because China is awful. Overall, China will still lose lots of money.
Looking at 2, China’s environment is trash, and their priority is on short term growth, so this will stay the same for the foreseeable future. Unless China gets some bold new leader who is so charismatic he can keep the people from revolting while their short term growth slows and who implements major environmental reforms, then China is doomed. I don’t think that any such leaders exist in China right now. However, another solution would arise if some fantastic new technology allowed China to change the basis of its production system. I don’t know what the probability of cheap and widespread nanotech or something similar by 2035 is, but I’m going to assume that it’s low and so this also won’t happen and so China’s environment is in trouble.
Looking at 3, China won’t be able to maintain infrastructure as a consequence of 1 and 2. This will aggravate underlying problems. Also, China’s already running into problems because other countries in the region are trying to take the cheap industrial production section of the market China used to have. China’s been trying to switch towards a more technologically advanced production economy, but their education is subpar. This will get worse during the aging crisis.
China is basically dependent on economic growth to keep its population from revolting, from what I’ve read.
So I conclude that unless something very important happens with China’s leadership or cultural/social practices, the economy of China will collapse by 2035 and China will then disintegrate into a civil war. I’m going to put about a 80% probability estimate on this one. This is what I actually believe, although this is sort of an incomplete description and also it oversimplifies and formalizes a lot of what is just raw intuition in my head.
This example also hopefully makes clear the risks of infinite regression. Stuff is complicated.
The “unless something very important” hedge makes this prediction rather hard to judge: it’s vague enough that anyone might confidently predict “something very important happens with China’s leadership or cultural/social practices between now and 2035″, because something is bound to happen that arguably falls under this specification.
This is kind of a converse rule of thumb to the “prefer status quo”, which—I should maybe have said—is valid at fairly short timeframes. Over long enough timeframes, the reverse is true—pretty much everything we can specify is going to change. (Over long enough timeframes, continents will move, mountains will be eroded to sea level, and eventually the planet itself will no longer exist.)
Over a twenty year timeframe, if you phrase your prediction in terms of (say) “the rate of exchange between USD and CNY”, there is some chance that one of the currencies will cease to exist, or that the notion of “rate of exchange” will stop making sense.
Nitpick: I don’t think this is much beyond epsilon. Even if a currency ‘ceases to exist’, the currency still exists in whatever physical embodiment. There is still an exchange rate from Saddam’s dinars to US dollars: it’s however many pieces of old paper $1 will buy you on eBay. The exchange rates for currencies that cease to exist can even go up over time (how much would a American Revolution-era shinplaster cost ya?).
If something important happens, it is not an excuse. My prediction will still have been wrong.
Ooh. As long as I’m making predictions, I believe Romney will defy the polls and win. 65% chance I’m right.
The economy is doing poorly.
Obama is an incumbent.
Romney is taller.
I think polls (and 538) are overestimating what Democratic participation will be like.
These are the only factors I considered. I intentionally took a different approach to this prediction.
This prediction is very hedgehoggy, but I think that is the right tool for the job here. I feel comfortable doing that here but not in China because I believe macro trends work much better as predictors of things like human psychology and things like democratic votes. In contrast, I don’t think I can form a good reference class to put China (2012) in that will allow me to evaluate the probability it collapses in, so I chose to instead use speciifc data. Also, I’ve examined specific facts about China, but I don’t feel confident in my ability to do that when it comes to the elections, because of media bias and cultural biases.
How confident are you? IEM and InTrade would like to know.
It’s far too late to get in on IEM or InTrade action—with like 24 days to go, it’d take almost all of that time to deposit funds via check or get ID cleared.
That said, I’d be happy to wager with chaosmosis that Romney will lose and Obama will win- say his $30 against my $20?
Nope, but thanks.
I’m confident, but not meta-level confident, because I haven’t been doing this for a while and I’m using new techniques and the experts are against me. I also have low risk tolerance for cash losses.
Additionally, you’re probably putting the terms of the bargain like that because I said 65% chance, but I expect that you’ve got an even higher chance in your mind that Obama would win. Obama winning is conventional wisdom, I would only accept a bet that rewarded me for being an outlier.
Fine—you drive a hard bargain! $20:$20 or $30:$30?
Not gonna do it. Maybe that means I should revise my estimates, or that I have irrational feelings about bets in general. I’ll figure that out later.
I notice that I would be willing to bet on China, that’s because: 1. I’ve looked through the data more and taken a foxy approach to it. 2. I think that China’s fate is less subject to random events 3. I’m not really against conventional wisdom with China. 4. I feel more metalevel confident about my ability to predict world affairs, I’m not sure why though.
I think the most likely reason that I’m not wanting to bet is that I don’t know whether or not I’m safe in doubting conventional wisdom and current polls and most experts in this specific field, and that I actually suspect that I’m not. For now I’ll revise my estimates, I’ll still incorporate my move towards Romney from the mainstream but I’ll start at the mainstream, which would be about 80% Obama, and I’ll move towards Romney, to 60% for Obama.
Also, I might have been in backlash mode or trying to be TOO pessimistic when I made the initial prediction that Obama would lose. Either I’ve gotten deeper into the liberal echo chamber from redditing and whatnot and my perceptions are getting distorted, or else I was being too pessimistic before (because I tried to force factors like height into my model, and I assumed that substance was basically irrelevant to political success, but now I’m not so sure.) I think I wanted to see myself as pessimistic so I wrote off the mainstream opinion and maybe I overdid it.
Additionally, I’m not wanting to go through the hassle of getting a paypal or whatever might happen. I would like the information on this (whatever the standard betting venue or monetary exchange thingy on LessWrong would be, if there is one, even though I’m not going to be betting here). I think I might be scared that I’ll lose the bet and then decide to not pay you, also, and that’s something else that’s making me reluctant to bet. Now that I think about it, I should also get an Intrade account in advance now.
There are many complicating factors. Demands for money helped me identify some.
FWIW, my own Obama probability is more like 65-70%.
Bets are typically paid as Paypal, Bitcoin, or donations to specified charities.
I personally don’t use Intrade because the new fees make it very expensive for low-stakes long-term bets like most bets on LW (assuming the market even exists).
I was tired when I wrote this. I’m going to bed now, and I’m not going to fix the grammar or the colloquialisms or the general disorganization.