Welcome to the web site for the upcoming paper “The Fallacy of Placing Confidence in Confidence Intervals.” Here you will find a number of resources connected to the paper, including the itself, the supplement, teaching resources and in the future, links to discussion of the content.
The paper is accepted for publication in Psychonomic Bulletin & Review.
Interval estimates – estimates of parameters that include an allowance for sampling uncertainty
– have long been touted as a key component of statistical analyses. There are several kinds of
interval estimates, but the most popular are confidence intervals (CIs): intervals that contain
the true parameter value in some known proportion of repeated samples, on average. The width
of confidence intervals is thought to index the precision of an estimate; CIs are thought to be a
guide to which parameter values are plausible or reasonable; and the confidence coefficient of
the interval (e.g., 95%) is thought to index the plausibility that the true parameter is included
in the interval. We show in a number of examples that CIs do not necessarily have any of
these properties, and can lead to unjustified or arbitrary inferences. For this reason, we caution
against relying upon confidence interval theory to justify interval estimates, and suggest that
other theories of interval estimation should be used instead
The Fallacy of Placing Confidence in Confidence Intervals
I just read through this, and it sounds like they’re trying to squish a frequentist interpretation on a Bayesian tool. They keep saying how the confidence intervals don’t correspond with reality, but confidence intervals are supposed to be measuring degrees of belief. Am I missing something here?
I briefly skimmed the paper and don’t see how you are getting this impression. Confidence intervals are—if we force the dichotomy—considered a frequentist rather than Bayesian tool. They point out that others are trying to squish a Bayesian interpretation on a frequentist tool by treating confidence intervals as though they are credible intervals, and they state this quite explicitly (p.17–18, emphasis mine):
Finally, we believe that in science, the meaning of our inferences are important. Bayesian credible intervals support an interpretation of probability in terms of plausibility, thanks to the explicit use of a prior. Confidence intervals, on the other hand, are based on a philosophy that does not allow inferences about plausibility, and does not utilize prior information. Using confidence intervals as if they were credible intervals is an attempt to smuggle Bayesian meaning into frequentist statistics, without proper consideration of a prior. As they say, there is no such thing as a free lunch; one must choose. We suspect that researchers, given the choice, would rather specify priors and get the benefits that come from Bayesian theory. We should not pretend, however, that the choice need not be made. Confidence interval theory and Bayesian theory are not interchangeable, and should not be treated as so.
I see. Looking into this, it seems that the (mis)use of the phrase “confidence interval” to mean “credible interval” is endemic on LW. A Google search for “confidence interval” on LW yields more than 200 results, of which many—perhaps most—should say “credible interval” instead. The corresponding search for “credible interval” yields less than 20 results.
The Fallacy of Placing Confidence in Confidence Intervals
pdf
I just read through this, and it sounds like they’re trying to squish a frequentist interpretation on a Bayesian tool. They keep saying how the confidence intervals don’t correspond with reality, but confidence intervals are supposed to be measuring degrees of belief. Am I missing something here?
I briefly skimmed the paper and don’t see how you are getting this impression. Confidence intervals are—if we force the dichotomy—considered a frequentist rather than Bayesian tool. They point out that others are trying to squish a Bayesian interpretation on a frequentist tool by treating confidence intervals as though they are credible intervals, and they state this quite explicitly (p.17–18, emphasis mine):
Hmmm, yes, I suppose I was making the same mistake they were… I thought that what confidence intervals were are actually what credible intervals are.
I see. Looking into this, it seems that the (mis)use of the phrase “confidence interval” to mean “credible interval” is endemic on LW. A Google search for “confidence interval” on LW yields more than 200 results, of which many—perhaps most—should say “credible interval” instead. The corresponding search for “credible interval” yields less than 20 results.