What do you want to reward? Honest estimates or accurate predictions? Remember that probability (at least at this level) is in the map, not the territory—there IS no definition of “correct probability” after the result is known. In other words, things that happen are probability 1, things that don’t are probability 0.
If you care about prediction ability, go with the naive mechanism—reward whoever’s right. You can still get an overall probaility because people should predict based on their probability estimate AND the number of people they’ll share with if that side wins. So if I think there’s a 95% chance of “yes”, but 99⁄100 of other predictors are saying “yes”, I should predict “no”, because 5% of $50 is better than 99% of 1.01. This will move the aggregate closer to our average beliefs.
If you care about honesty, it’s trickier. You might be able to get there with some amount of hedging or quantity-wagering. Make the payout split among everyone, weighted by their credence in that outcome. So if “yes”, someone who picked 75% gets 50% more reward than someone who picked 50% (but if it’s “no”, they get half as much reward for 75% than 50%).
Weighted by credence means you’re scored on Probability*Prediction, which isn’t a fair rule. If I sincerely believe its 60:40 between two options, and I write that down, I expect 36+16=52 payout, but if I write 100:0 I expect 60+0=60 payout, putting more credence on higher probability outcomes gets me more money even in excess of my true beliefs.
What do you want to reward? Honest estimates or accurate predictions? Remember that probability (at least at this level) is in the map, not the territory—there IS no definition of “correct probability” after the result is known. In other words, things that happen are probability 1, things that don’t are probability 0.
If you care about prediction ability, go with the naive mechanism—reward whoever’s right. You can still get an overall probaility because people should predict based on their probability estimate AND the number of people they’ll share with if that side wins. So if I think there’s a 95% chance of “yes”, but 99⁄100 of other predictors are saying “yes”, I should predict “no”, because 5% of $50 is better than 99% of 1.01. This will move the aggregate closer to our average beliefs.
If you care about honesty, it’s trickier. You might be able to get there with some amount of hedging or quantity-wagering. Make the payout split among everyone, weighted by their credence in that outcome. So if “yes”, someone who picked 75% gets 50% more reward than someone who picked 50% (but if it’s “no”, they get half as much reward for 75% than 50%).
Weighted by credence means you’re scored on Probability*Prediction, which isn’t a fair rule.
If I sincerely believe its 60:40 between two options, and I write that down, I expect 36+16=52 payout, but if I write 100:0 I expect 60+0=60 payout, putting more credence on higher probability outcomes gets me more money even in excess of my true beliefs.