The book is pretty much exactly what the title says: it’s all about how to accurately get expert’s opinions, whatever those opinions might be (as opposed to trying to get experts to be accurate). Much probability/statistics theory is explained (especially Bayesianism) as well as a good deal of heuristics and biases material like anchoring-adjusting, affect heuristic + inside/outside view.
Some points:
A repeated point is that experts, notwithstanding their subject expertise, are often not trained in probability and probabilistic thinking such that they’re not very good by default at reporting estimates.
Part of this is most people are familiar with probability only in terms of repeatable, random events that are nicely covered by frequentist statistics and don’t know how to give subjective probability estimates well. (The book calls subjective probabilities “personal probabilities”.)
A suggested solution is giving experts appropriate training, calibration training, etc. in advance of trying to elicit their estimates.
There’s discussion of coherence (in the sense of conforming to the basic probability theorems). An interesting point is that while it’s easy to see if probabilities of mutually exclusive events add up to greater than 1, it can harder to see if several correlations one believes in are inconsistent (say, resulting in a covariance matrix that isn’t positive-definite). Each believed correlation on its own can seem fine to a person even though in aggregate they don’t work.
Another interesting point is the observation is that people are good at reporting the frequency of their own observation of thing, but bad at seeing or correcting for the fact that sampling biases can affect what they end up observing.
On the whole, kinda interesting stuff on how to actually get experts actual true beliefs, but nothing really specifically on the topic of getting consistent estimates. The closest thing to that seems to be the parts on getting coherent probability estimates from people, though generally, the book mixes between “accurately elicit expert’s beliefs” and “get experts to have accurate, unbiased beliefs.”
I had a look over Uncertain Judgements: Eliciting Experts’ Probabilities, mostly reading the through the table of contents and jumping around and reading bits which seemed relevant.
The book is pretty much exactly what the title says: it’s all about how to accurately get expert’s opinions, whatever those opinions might be (as opposed to trying to get experts to be accurate). Much probability/statistics theory is explained (especially Bayesianism) as well as a good deal of heuristics and biases material like anchoring-adjusting, affect heuristic + inside/outside view.
Some points:
A repeated point is that experts, notwithstanding their subject expertise, are often not trained in probability and probabilistic thinking such that they’re not very good by default at reporting estimates.
Part of this is most people are familiar with probability only in terms of repeatable, random events that are nicely covered by frequentist statistics and don’t know how to give subjective probability estimates well. (The book calls subjective probabilities “personal probabilities”.)
A suggested solution is giving experts appropriate training, calibration training, etc. in advance of trying to elicit their estimates.
There’s discussion of coherence (in the sense of conforming to the basic probability theorems). An interesting point is that while it’s easy to see if probabilities of mutually exclusive events add up to greater than 1, it can harder to see if several correlations one believes in are inconsistent (say, resulting in a covariance matrix that isn’t positive-definite). Each believed correlation on its own can seem fine to a person even though in aggregate they don’t work.
Another interesting point is the observation is that people are good at reporting the frequency of their own observation of thing, but bad at seeing or correcting for the fact that sampling biases can affect what they end up observing.
On the whole, kinda interesting stuff on how to actually get experts actual true beliefs, but nothing really specifically on the topic of getting consistent estimates. The closest thing to that seems to be the parts on getting coherent probability estimates from people, though generally, the book mixes between “accurately elicit expert’s beliefs” and “get experts to have accurate, unbiased beliefs.”