How did Less Wrong do by comparison? The average estimated probability of Amanda Knox’s guilt was 0.35 (thanks to Yvain for doing the calculation).
Yvain! How could you? What did the probabilities do to deserve that kind of abuse? (I strongly assert that averaging the probabilities is not a good way to combine such estimates.)
...and why do you assert that? If you have good reasons, I’d like to see a top level post on the subject, since this is my natural response to a bunch of probability estimates given by different people with the same information who are rational enough that I care what they think.
Not a top level post because I don’t think I have the definitive say on the matter, but I made an article in the wiki that illustrates why the mean of the log-odds makes more sense.
Unfortunately, the wiki appears to have issues with math right now, so the article has an ugly error message in it. The formula works fine in the Wikipedia sandbox. If anyone knows what is going on or has any other changes, feel free to edit.
That’s what I would do. If one person is almost certain (say 1/(10^10^10)) then the strength of their view would be represented. Of course if anyone gives an irrationally low or high answer, or puts <=0 or >=1, then it overweights their views/blows up.
I wonder what implications this has for the method of choosing priors I came up with that is “ask everybody in the world what they think the priors should be, normalize the invalid ones, and take the average of all of them”.
Yvain! How could you? What did the probabilities do to deserve that kind of abuse? (I strongly assert that averaging the probabilities is not a good way to combine such estimates.)
...and why do you assert that? If you have good reasons, I’d like to see a top level post on the subject, since this is my natural response to a bunch of probability estimates given by different people with the same information who are rational enough that I care what they think.
Not a top level post because I don’t think I have the definitive say on the matter, but I made an article in the wiki that illustrates why the mean of the log-odds makes more sense.
Unfortunately, the wiki appears to have issues with math right now, so the article has an ugly error message in it. The formula works fine in the Wikipedia sandbox. If anyone knows what is going on or has any other changes, feel free to edit.
Somebody with no information does not so effectively counterbalance ten people who can describe the positions of every atom on the planet.
I calculated an example involving 0.99, 0.5, 0.745 and (1-10^20) but then I noticed badger’s link beat me to it.
What would the right thing look like? Averaging the log-odds ratio?
That’s what I would do. If one person is almost certain (say 1/(10^10^10)) then the strength of their view would be represented. Of course if anyone gives an irrationally low or high answer, or puts <=0 or >=1, then it overweights their views/blows up.
I wonder what implications this has for the method of choosing priors I came up with that is “ask everybody in the world what they think the priors should be, normalize the invalid ones, and take the average of all of them”.