What’s the best way to have people indicate probability distributions? It has to be easy enough to not cut down on participation unduly.
My first guess would be asking people to state their 95% confidence intervals and pretend that the distribution is normal.
Other possibilities: ask people to bet on binary events at various odds; ask people to drag-and-drop a probability curve; ask a short sequence of frequentist-language questions to get at intuition (“if you tried this 10 times, how many times would you expect the positive outcome?” “if you tried this ten times, would you expect one example to reach a score of at least X?”)
I think we’ve had pretty decent success with Guesstimate, where users enter 90% confidence intervals which are fitted to lognormal distributions (toggle-able to normal distributions.)
I’d imagine here that really simple defaults are useful, but for the cases that users want more precision, they should be able to get it. Being asked a few separate questions is one way of inferring that.
It also of course depends a bit on the education of the audience.
Separately, I imagine that if algorithms would compete, they should just be able to enter the distribution directly.
What’s the best way to have people indicate probability distributions? It has to be easy enough to not cut down on participation unduly.
My first guess would be asking people to state their 95% confidence intervals and pretend that the distribution is normal. Other possibilities: ask people to bet on binary events at various odds; ask people to drag-and-drop a probability curve; ask a short sequence of frequentist-language questions to get at intuition (“if you tried this 10 times, how many times would you expect the positive outcome?” “if you tried this ten times, would you expect one example to reach a score of at least X?”)
It’s definitely an important question.
I think we’ve had pretty decent success with Guesstimate, where users enter 90% confidence intervals which are fitted to lognormal distributions (toggle-able to normal distributions.)
I’d imagine here that really simple defaults are useful, but for the cases that users want more precision, they should be able to get it. Being asked a few separate questions is one way of inferring that.
It also of course depends a bit on the education of the audience.
Separately, I imagine that if algorithms would compete, they should just be able to enter the distribution directly.