Hence my question in another thread: Was that “exactly 95% confidence” or “at least 95% confidence”? However when researchers say “at a 95% confidence level” they typically mean “p < 0.05″, and reporting the actual p-values is often even explicitly discouraged (let’s not digress into whether it is justified).
Yet the mistake I had in mind (as opposed to other, less relevant, merely “a” mistakes) involves Type I and Type II error rates. Just because you are 95% (or more) confident of not making one type of error doesn’t guarantee you an automatic 5% chance of getting the other.
Hence my question in another thread: Was that “exactly 95% confidence” or “at least 95% confidence”? However when researchers say “at a 95% confidence level” they typically mean “p < 0.05″, and reporting the actual p-values is often even explicitly discouraged (let’s not digress into whether it is justified).
Yet the mistake I had in mind (as opposed to other, less relevant, merely “a” mistakes) involves Type I and Type II error rates. Just because you are 95% (or more) confident of not making one type of error doesn’t guarantee you an automatic 5% chance of getting the other.