Maybe a similar rule in forecasting is “home ground advantage plus personal or professional stakes, or don’t bother.”
On 16 questions currently scored, I’ve done better than the team average at 15. Two of the questions where I outperformed the team by a large margin where the Syrian refugee question, basically a matter of extrapolating a trend and predicting status quo with respect to the conflict, and the Kismayo question, basically a matter of knowing my loss function. I had zero home ground advantage on either question.
Some of my wins resulted purely from general knowledge rather than from having any idea of the specifics of the situation: for instance, in mid-August I answered 40% to “Will Kuwait commence parliamentary elections before 1 October 2012?”, reflecting only status quo bias in that a date for the election had not yet been announced. However, early in September I downgraded this to 10%, because I know that as a rule of thumb it takes at least one month to convene an election. The week before, I went to 5% (and even that was quite a generous margin), while several of my teammates made predictions, after I published mine, of 15%, 19%, 33% and even 51% (!).
This felt like entering a poker tournament where people routinely raise pre-flop with a “beer hand” (seven and two—when you play this, either you’ve had too many beers, or it’s time you have one). Elections aren’t a mysterious thing, we participate in one every so often. You need to print ballots, set up voting booths, audit voter registration records, give people time to campaign on national media, all very mundane stuff. Even dictatorships make at least a half-hearted attempt at this, and it’s not like anyone in Kuwait had any particular interest in meeting an October deadline, this was strictly an internal-to-GJP deadline.
So while this question had to do, ostensibly, with something happening in Kuwait, all you needed to make a call at least as good as mine was background knowledge about extremely mundane, practical stuff that, if I had any hint that you wouldn’t factor that in when making a close-to-home prediction, I wouldn’t trust you with organizing so much as the PTA president election. Maybe a birthday party.
I wouldn’t go so far as to claim that “skill at forecasting macro trends transfer to microeconomic moves”.
But I’d take a stand on “demonstrated incompetence at the most elementary moves of forecasting, in a macro domain, is a strong indicator of likely incompetence at forecasting in any micro domain, other than the few narrow ones you might happen to be good at”.
They compute your Brier score for each day that the question is open, according to what your forecast is on that day, and average over all days.
Suppose you start at 80%, six days pass, you switch to 40% three days before the deadline, and the event doesn’t happen, your score is (6*(0.8)^2+3*(0.4)^2)/9 = .48, which is a so-so score—but an improvement over the .64 that you’d get if you didn’t change your mind.
Some of my wins resulted purely from general knowledge rather than from having any idea of the specifics of the situation: for instance, in mid-August I answered 40% to “Will Kuwait commence parliamentary elections before 1 October 2012?”, reflecting only status quo bias in that a date for the election had not yet been announced. However, early in September I downgraded this to 10%, because I know that as a rule of thumb it takes at least one month to convene an election. The week before, I went to 5% (and even that was quite a generous margin), while several of my teammates made predictions, after I published mine, of 15%, 19%, 33% and even 51% (!).
Yeah. Answering “1%” that “there will be a major earthquake in California during $time_period” a month before the end of $time_period kind-of felt like cheating to me.
On 16 questions currently scored, I’ve done better than the team average at 15. Two of the questions where I outperformed the team by a large margin where the Syrian refugee question, basically a matter of extrapolating a trend and predicting status quo with respect to the conflict, and the Kismayo question, basically a matter of knowing my loss function. I had zero home ground advantage on either question.
Some of my wins resulted purely from general knowledge rather than from having any idea of the specifics of the situation: for instance, in mid-August I answered 40% to “Will Kuwait commence parliamentary elections before 1 October 2012?”, reflecting only status quo bias in that a date for the election had not yet been announced. However, early in September I downgraded this to 10%, because I know that as a rule of thumb it takes at least one month to convene an election. The week before, I went to 5% (and even that was quite a generous margin), while several of my teammates made predictions, after I published mine, of 15%, 19%, 33% and even 51% (!).
This felt like entering a poker tournament where people routinely raise pre-flop with a “beer hand” (seven and two—when you play this, either you’ve had too many beers, or it’s time you have one). Elections aren’t a mysterious thing, we participate in one every so often. You need to print ballots, set up voting booths, audit voter registration records, give people time to campaign on national media, all very mundane stuff. Even dictatorships make at least a half-hearted attempt at this, and it’s not like anyone in Kuwait had any particular interest in meeting an October deadline, this was strictly an internal-to-GJP deadline.
So while this question had to do, ostensibly, with something happening in Kuwait, all you needed to make a call at least as good as mine was background knowledge about extremely mundane, practical stuff that, if I had any hint that you wouldn’t factor that in when making a close-to-home prediction, I wouldn’t trust you with organizing so much as the PTA president election. Maybe a birthday party.
I wouldn’t go so far as to claim that “skill at forecasting macro trends transfer to microeconomic moves”.
But I’d take a stand on “demonstrated incompetence at the most elementary moves of forecasting, in a macro domain, is a strong indicator of likely incompetence at forecasting in any micro domain, other than the few narrow ones you might happen to be good at”.
How does GJP score predictions that change over time?
They compute your Brier score for each day that the question is open, according to what your forecast is on that day, and average over all days.
Suppose you start at 80%, six days pass, you switch to 40% three days before the deadline, and the event doesn’t happen, your score is (6*(0.8)^2+3*(0.4)^2)/9 = .48, which is a so-so score—but an improvement over the .64 that you’d get if you didn’t change your mind.
Yeah. Answering “1%” that “there will be a major earthquake in California during $time_period” a month before the end of $time_period kind-of felt like cheating to me.