A Bayesian scores himself on total calibration, “number of times my 95% confidence interval includes the truth” is just a small part of it. You can generate an experiment that has a high chance (let’s say 99%) of making a Bayesian have a 20:1 likelihood ratio in favor of some hypothesis. By conservation of expected evidence, the same experiment might have 1% chance of generating close to a 2000:1 likelihood ratio against that same hypothesis. A frequentist could never be as sure of anything, this occasional 2000:1 confidence is the Bayesian’s reward.
Hold on. Let’s say I hire a Bayesian statistician to produce some estimate for me. I do not care about “scoring” or “reward”, all I care about is my estimate and how accurate it is. Now you are going to tell me that in 99% of the cases your estimate will be wrong and that’s fine because there is a slight chance that you’ll be really really sure of the opposite conclusion?
I’m running a Bayesian Casino in Vegas. Debrah Mayo comes to my casino every day with $31.
Why, that’s such a frequentist approach X-/
Let’s change the situation slightly. You are running the Bayesian Casino and Debrah Mayo comes to you casino once with, say, $1023 in her pocket. Will I lend you money to bet against her? No, I will not. The distribution matters beyond simple expected means.
van Nostrand: Of course. I remember each problem quite clearly. And I recall that on
each occasion I was quite thorough. I interrogated you in detail, determined your model
and prior and produced a coherent 95 percent interval for the quantity of interest.
Pennypacker: Yes indeed. We did this many times and I paid you quite handsomely.
van Nostrand: Well earned money I’d say. And it helped win you that Nobel.
Pennypacker: Well they retracted the Nobel and they took away my retirement savings.
…
van Nostrand: Whatever are you talking about?
Pennypacker: You see, physics has really advanced. All those quantities I estimated
have now been measured to great precision. Of those thousands of 95 percent intervals,
only 3 percent contained the true values! They concluded I was a fraud.
van Nostrand: Pennypacker you fool. I never said those intervals would contain the
truth 95 percent of the time. I guaranteed coherence not coverage!
Now you are going to tell me that in 99% of the cases your estimate will be wrong
No. Your calibration is still perfect if your priors are perfect. You can only get to that “99% chance of getting strong evidence for hypothesis” if you’re already very sure of that hypothesis math here
Hold on. Let’s say I hire a Bayesian statistician to produce some estimate for me. I do not care about “scoring” or “reward”, all I care about is my estimate and how accurate it is. Now you are going to tell me that in 99% of the cases your estimate will be wrong and that’s fine because there is a slight chance that you’ll be really really sure of the opposite conclusion?
Why, that’s such a frequentist approach X-/
Let’s change the situation slightly. You are running the Bayesian Casino and Debrah Mayo comes to you casino once with, say, $1023 in her pocket. Will I lend you money to bet against her? No, I will not. The distribution matters beyond simple expected means.
Reminds of this bit from a Wasserman paper http://ba.stat.cmu.edu/journal/2006/vol01/issue03/wasserman.pdf
No. Your calibration is still perfect if your priors are perfect. You can only get to that “99% chance of getting strong evidence for hypothesis” if you’re already very sure of that hypothesis math here