I expect it will be easier to get Metaculus users to make forecasts on pundits’ questions than to get pundits to make forecasts on each other’s questions.
Suggested variant (with dates for concreteness):
Dec 1: deadline for pundits to submit their questions Dec 10: metaculus announces the final version of all the questions they’re using, but does not open markets Dec 20: deadline for pundits & anyone else to privately submit their forecasts (maybe hashed), and metaculus markets open Dec 31: current metaculus consensus becomes the official metaculus forecast for the questions, and pundits (& anyone else) can publicize the forecasts that they made by Dec 20
Contestants (anyone who submitted forecasts by Dec 20) mainly get judged based on how they did relative to the Dec 31 metaculus forecast. I expect that they will mostly be pundits making forecasts on their own questions, plus forecasting aficionados.
(We want contestants & metaculus to make their forecasts simultaneously, with neither having access to the other’s forecasts, which is tricky since metaculus is a public platform. That’s why I have the separate deadlines on Dec 20 & Dec 31, with contestants’ forecasts initially private—hopefully that’s a short enough time period so that not much new information should arise, and long enough for people to have time to make forecasts.)
With only a small sample size of questions, it may be more meaningful to evaluate contestants based on how close they came to the official metaculus forecast rather than on how accurate they were (there’s a bias-variance tradeoff). As a contestant does more questions (this year or over multiple years), the comparison with what actually happened becomes more meaningful.
I expect it will be easier to get Metaculus users to make forecasts on pundits’ questions than to get pundits to make forecasts on each other’s questions.
Suggested variant (with dates for concreteness):
Dec 1: deadline for pundits to submit their questions
Dec 10: metaculus announces the final version of all the questions they’re using, but does not open markets
Dec 20: deadline for pundits & anyone else to privately submit their forecasts (maybe hashed), and metaculus markets open
Dec 31: current metaculus consensus becomes the official metaculus forecast for the questions, and pundits (& anyone else) can publicize the forecasts that they made by Dec 20
Contestants (anyone who submitted forecasts by Dec 20) mainly get judged based on how they did relative to the Dec 31 metaculus forecast. I expect that they will mostly be pundits making forecasts on their own questions, plus forecasting aficionados.
(We want contestants & metaculus to make their forecasts simultaneously, with neither having access to the other’s forecasts, which is tricky since metaculus is a public platform. That’s why I have the separate deadlines on Dec 20 & Dec 31, with contestants’ forecasts initially private—hopefully that’s a short enough time period so that not much new information should arise, and long enough for people to have time to make forecasts.)
With only a small sample size of questions, it may be more meaningful to evaluate contestants based on how close they came to the official metaculus forecast rather than on how accurate they were (there’s a bias-variance tradeoff). As a contestant does more questions (this year or over multiple years), the comparison with what actually happened becomes more meaningful.