Thanks for this great piece! A few thoughts with my Metaculus hat on:
We can think of a sort of “contest spectrum” where there is always a ranking, and there is a relationship between ranking and win probability. On one end of the spectrum the top N people win, and on the other end people are just paid in proportion to how well they predict. The latter end runs into legal problems, as it’s effectively then just a betting , while the former end runs into problems, as you say, if the number of questions in the contest is too low. Our current plan at Metaculus is to just make sure contests generally have enough questions (>20 is probably enough) to ensure that your chance of winning by taking extreme positions is vanishingly small; then we hope to implement other sorts of “bounties” that are not directly in proportion to predictive success (and are hence not betting). The probabilistic contest is a good fix for a contest with too few questions, but I’m not sure I love it in general.
In designing a scoring or reward system, it’s very tricky to find the right level of transparency/simplicity. Some transparency and simplicity is important as people need to know what the incentives are. But if it’s too simple and transparent, then the metric becomes the goal rather than measuring something else. The current Metaculus point system is complicated, but devised to incentivise certain things (lots of participation, frequent updates, and prediction of one’s true estimate of the probability) while being complicated enough that it’s kindof inscrutable and hence a pain to game. But there are lots of possible ways to do it and it would be quite interesting to think of more metrics for assessing and comparing predictors. In addition, there’s no reason for that Metaculus or anyone else to stick to a single metric (indeed the Metaculus aggregation does not work on the basis of points, and bounties — someday — probably won’t either).
We have a possible idea of “microteaming” here, but have not gotten much feedback on it so far. We definitely need more ways of rewarding information sharing and collaboration.
Thanks for this great piece! A few thoughts with my Metaculus hat on:
We can think of a sort of “contest spectrum” where there is always a ranking, and there is a relationship between ranking and win probability. On one end of the spectrum the top N people win, and on the other end people are just paid in proportion to how well they predict. The latter end runs into legal problems, as it’s effectively then just a betting , while the former end runs into problems, as you say, if the number of questions in the contest is too low. Our current plan at Metaculus is to just make sure contests generally have enough questions (>20 is probably enough) to ensure that your chance of winning by taking extreme positions is vanishingly small; then we hope to implement other sorts of “bounties” that are not directly in proportion to predictive success (and are hence not betting). The probabilistic contest is a good fix for a contest with too few questions, but I’m not sure I love it in general.
In designing a scoring or reward system, it’s very tricky to find the right level of transparency/simplicity. Some transparency and simplicity is important as people need to know what the incentives are. But if it’s too simple and transparent, then the metric becomes the goal rather than measuring something else. The current Metaculus point system is complicated, but devised to incentivise certain things (lots of participation, frequent updates, and prediction of one’s true estimate of the probability) while being complicated enough that it’s kindof inscrutable and hence a pain to game. But there are lots of possible ways to do it and it would be quite interesting to think of more metrics for assessing and comparing predictors. In addition, there’s no reason for that Metaculus or anyone else to stick to a single metric (indeed the Metaculus aggregation does not work on the basis of points, and bounties — someday — probably won’t either).
We have a possible idea of “microteaming” here, but have not gotten much feedback on it so far. We definitely need more ways of rewarding information sharing and collaboration.