So, to “win” I need to participate in every possible market, regardless of my own knowledge, as long as I can make a prediction with positive value regardless of outcome (or at least a hugely favorable spread for going with the consensus?
That sounds like a flaw.
Yes, but it doesn’t take much time to just predict the community median when you don’t have a clue about a question and don’t want to take the time for getting into it. However, as another commenter points out, this means that Metaculus is rewarding a combination of time put in + prediction skills, rather than just prediction skills.
What are you hoping to “win”? This isn’t a market—you don’t need your relative performance to be better than someone else’s to have done well. And giving people points for guessing the community prediction is valuable, since it provides evidence that they don’t have marginal information that causes them to believe something different. If people only predict when they are convinced they know significantly more than others, there would be far fewer predictions.
The wording here makes me worry we’re Goodharting on quantity of predictions. And the best way to predict the community prediction is to (of course) wait for others to predict first, then match them...
If the user is interested in getting into the top ranks, this strategy won’t be anything like enough. And if not, but they want to maximize their score, the scoring system is still incentive compatible—they are better off reporting their true estimate on any given question. And for the worst (but still self-aware) predictors, this should be the metaculus prediction anyways—so they can still come away with a positive number of points, but not many. Anything much worse than that, yes, people could have negative overall scores—which, if they’ve predicted on a decent number of questions, is pretty strong evidence that they really suck at forecasting.
Looking at my track record, for questions resolved in the last 3 months, evaluated at all times, here’s how my log score looks compared to the community:
Binary questions (N=19): me: -.072 vs. community: -.045
Continuous questions (N=20): me: 2.35 vs. community: 2.33
So if anything, I’ve done a bit worse than the community overall, and am in 5th by virtue of predicting on all questions. It’s likely that the predictors significantly in front of me are that far ahead in part due to having predicted on (a) questions that have resolved recently but closed before I was active and (b) a longer portion of the lifespan for questions that were open before I became active.
Edit:
I discovered that the question set changes when I evaluate at “resolve time” and filter for the past 3 months, not sure why exactly. Numbers at resolve time:
Binary questions (N=102): me: .598 vs. community: .566
Continuous questions (N=92): me: 2.95 vs. community: 2.86
I think this weakens my case substantially, though I still think a bot that just predicts the community as soon as it becomes visible and updates every day would currently be at least top 10.
Would a bot that just predicted the community be at least top 10 on Metaculus over the last 3 months, as of Feb 12 2021?
Anything much worse than that, yes, people could have negative overall scores—which, if they’ve predicted on a decent number of questions, is pretty strong evidence that they really suck at forecasting
I agree that this should have some effect of being less welcoming to newcomers, but I’m curious to what extent. I have seen plenty of people with worse brier scores than the median continuing to predict on GJO rather than being demoralized and quitting (disclaimer: survivorship bias).
So, to “win” I need to participate in every possible market, regardless of my own knowledge, as long as I can make a prediction with positive value regardless of outcome (or at least a hugely favorable spread for going with the consensus? That sounds like a flaw.
Yes, but it doesn’t take much time to just predict the community median when you don’t have a clue about a question and don’t want to take the time for getting into it. However, as another commenter points out, this means that Metaculus is rewarding a combination of time put in + prediction skills, rather than just prediction skills.
What are you hoping to “win”? This isn’t a market—you don’t need your relative performance to be better than someone else’s to have done well. And giving people points for guessing the community prediction is valuable, since it provides evidence that they don’t have marginal information that causes them to believe something different. If people only predict when they are convinced they know significantly more than others, there would be far fewer predictions.
The wording here makes me worry we’re Goodharting on quantity of predictions. And the best way to predict the community prediction is to (of course) wait for others to predict first, then match them...
If the user is interested in getting into the top ranks, this strategy won’t be anything like enough. And if not, but they want to maximize their score, the scoring system is still incentive compatible—they are better off reporting their true estimate on any given question. And for the worst (but still self-aware) predictors, this should be the metaculus prediction anyways—so they can still come away with a positive number of points, but not many. Anything much worse than that, yes, people could have negative overall scores—which, if they’ve predicted on a decent number of questions, is pretty strong evidence that they really suck at forecasting.
I think this isn’t true empirically for a reasonable interpretation of top ranks. For example, I’m ranked 5th on questions that have resolved in the past 3 months due to predicting on almost every question.
Looking at my track record, for questions resolved in the last 3 months, evaluated at all times, here’s how my log score looks compared to the community:
Binary questions (N=19): me: -.072 vs. community: -.045
Continuous questions (N=20): me: 2.35 vs. community: 2.33
So if anything, I’ve done a bit worse than the community overall, and am in 5th by virtue of predicting on all questions. It’s likely that the predictors significantly in front of me are that far ahead in part due to having predicted on (a) questions that have resolved recently but closed before I was active and (b) a longer portion of the lifespan for questions that were open before I became active.
Edit:
I discovered that the question set changes when I evaluate at “resolve time” and filter for the past 3 months, not sure why exactly. Numbers at resolve time:
Binary questions (N=102): me: .598 vs. community: .566
Continuous questions (N=92): me: 2.95 vs. community: 2.86
I think this weakens my case substantially, though I still think a bot that just predicts the community as soon as it becomes visible and updates every day would currently be at least top 10.
I agree that this should have some effect of being less welcoming to newcomers, but I’m curious to what extent. I have seen plenty of people with worse brier scores than the median continuing to predict on GJO rather than being demoralized and quitting (disclaimer: survivorship bias).
I think you get more points for earlier predictions.