My heuristic for deciding what heuristic to use, when you’re going to do something quick-n-dirty: figure out what quantity you’re actually interested in, use means in its natural scale for point estimates, and transform back to your inputs.
How does this apply to some examples?
In your post, you’re talking quite a lot about bets. To a first approximation marginal utility is linear in marginal wealth, so usually this means the quantity we are actually interested in is linear in probability, and the correct heuristic is “arithmetic mean” (of probability).
In SimonM’s comment, we’re talking about probabilities directly. Forecasting. Usually that means what we care about is calibration or a proper scoring rule, so the natural scale is [0,1] or log-odds. Now the correct heuristic is “arithmetic mean” (of log-odds of probability).
What about your difficult examples?
Expert 1 gives you a probability distribution 50% A, 25% B, 25% C, expert 2 gives you a probability distribution 25% A, 50% B, 25% C, and expert 3 gives you a probability distribution 25% A, 25% B, 50% C.
We’re already doing quick-n-dirty things rather than anything rigorous. What I usually do when I want constraints and my summary statistics don’t satisfy them is to just go ahead and normalize, after which of course we get 1⁄3, 1⁄3, 1⁄3 by symmetry after any estimation.
that depends on the prior
Once we’re no longer just doing something quick-n-dirty, all, uh, bets, are off. But my heuristic is still to transform to the domain where what you care about is linear, do your somewhat-more-sophisticated dirty work, and transform back to get your point estimate.
In SimonM’s comment, we’re talking about probabilities directly. Forecasting. Usually that means what we care about is calibration or a proper scoring rule, so the natural scale is [0,1] or log-odds. Now the correct heuristic is “arithmetic mean” (of log-odds of probability).
Not sure what you mean by this. A proper scoring rule incentivizes the same results that deciding what odds you’d be indifferent to betting on at (against a gambler whose decisions carry no information about reality) does.
My heuristic for deciding what heuristic to use, when you’re going to do something quick-n-dirty: figure out what quantity you’re actually interested in, use means in its natural scale for point estimates, and transform back to your inputs.
How does this apply to some examples?
In your post, you’re talking quite a lot about bets. To a first approximation marginal utility is linear in marginal wealth, so usually this means the quantity we are actually interested in is linear in probability, and the correct heuristic is “arithmetic mean” (of probability).
In SimonM’s comment, we’re talking about probabilities directly. Forecasting. Usually that means what we care about is calibration or a proper scoring rule, so the natural scale is [0,1] or log-odds. Now the correct heuristic is “arithmetic mean” (of log-odds of probability).
What about your difficult examples?
We’re already doing quick-n-dirty things rather than anything rigorous. What I usually do when I want constraints and my summary statistics don’t satisfy them is to just go ahead and normalize, after which of course we get 1⁄3, 1⁄3, 1⁄3 by symmetry after any estimation.
Once we’re no longer just doing something quick-n-dirty, all, uh, bets, are off. But my heuristic is still to transform to the domain where what you care about is linear, do your somewhat-more-sophisticated dirty work, and transform back to get your point estimate.
Not sure what you mean by this. A proper scoring rule incentivizes the same results that deciding what odds you’d be indifferent to betting on at (against a gambler whose decisions carry no information about reality) does.