There are two misconceptions that you must be aware of, as you will certainly hear these. The first is thinking that we calculate the probability of the null hypothesis being true or false. Whether the null hypothesis is true or false is not subject to chance; it either is true or it is false—there is no probability of one or the other.
So from this statement you conclude that frequentists think P(hypothesis) is meaningless? Bayesians assign degrees of belief to things that are actually true or false also. The coin really is either fair or not fair, but you will never find out with finite trials. This is a map/territory distinction, I am surprised you didn’t get it. This quote has nothing to do with B/F differences.
A Bayesian version of this quote would point out that it is a type error to confuse the truth value of the underlying thing, and the belief about this truth value.
You have successfully explained why it is irrational for frequentists to consider P(hypothesis) meaningless. And yet they do. They would say that probabilities can only be defined as limiting frequencies in repeated experiments, and that for a typical hypothesis there is no experiment you can rerun to get a sample for the truth of the hypothesis.
Yes, you’re right. Clearly many people who identify as frequentists do hold P(hypothesis) to be meaningful. There are statisticians all over the B/F spectrum as well as not on the spectrum at all. So when I said “frequentists believe …” I could never really be correct because various frequentists believe various different things.
Perhaps we could agree on the following statement: “Probabilities such as P(hypothesis) are never needed to do frequentist analysis.”
For example, the link you gave suggests the following as a characterisation of frequentism:
Goal of Frequentist Inference: Construct procedure with frequency guarantees. (For example, confidence intervals.)
Since frequency guarantees are typically of the form “for each possible true value of theta doing the construction blah on the data will, with probability at least 1-p, yield a result with property blah”. Then since this must hold true for each theta, the distribution for the true value of theta is irrelevant.
I could never really be correct because various frequentists believe various different things.
The interesting questions to me are: (a) “what is the steelman of the frequentist position?” (folks like Larry are useful here), and (b) “are there actually prominent frequentist statisticians who say stupid things?”
By (b) I mean “actually stupid under any reasonable interpretation.”
Clearly many people who identify as frequentists
Quote from the url I linked:
One thing that has harmed statistics — and harmed science — is identity statistics. By this I mean that some
people identify themselves as “Bayesians” or “Frequentists.” Once you attach a label to yourself, you have
painted yourself in a corner.
When I was a student, I took a seminar course from Art Dempster. He was the one who suggested to me that
it was silly to describe a person as being Bayesian of Frequentist. Instead, he suggested that we describe a
particular data analysis as being Bayesian of Frequentist. But we shouldn’t label a person that way.
I think Art’s advice was very wise.
“Keep your identity small”—advice familiar to a LW audience.
Perhaps we could agree on the following statement: “Probabilities such as P(hypothesis) are never needed to do
frequentist analysis.”
I guess you disagree with Larry’s take: B vs F is about goals not methods. I could do Bayesian looking things while having a frequentist interpretation in mind.
In the spirit of collaborative argumentation, can we agree on the following:
We have better things to do than engage in identity politics.
Could you give me something to read? Who are these frequentists, and where do they insist on this?
Let us take a common phrase from the original comment “the hypothesis is either true or false”. The first google hit:
So from this statement you conclude that frequentists think P(hypothesis) is meaningless? Bayesians assign degrees of belief to things that are actually true or false also. The coin really is either fair or not fair, but you will never find out with finite trials. This is a map/territory distinction, I am surprised you didn’t get it. This quote has nothing to do with B/F differences.
A Bayesian version of this quote would point out that it is a type error to confuse the truth value of the underlying thing, and the belief about this truth value.
You have successfully explained why it is irrational for frequentists to consider P(hypothesis) meaningless. And yet they do. They would say that probabilities can only be defined as limiting frequencies in repeated experiments, and that for a typical hypothesis there is no experiment you can rerun to get a sample for the truth of the hypothesis.
You guys need to stop assuming frequentists are morons. Here are posts by a frequentist:
http://normaldeviate.wordpress.com/2012/12/04/nate-silver-is-a-frequentist-review-of-the-signal-and-the-noise/
http://normaldeviate.wordpress.com/2012/11/17/what-is-bayesianfrequentist-inference/
Some of the comments are good as well.
Yes, you’re right. Clearly many people who identify as frequentists do hold P(hypothesis) to be meaningful. There are statisticians all over the B/F spectrum as well as not on the spectrum at all. So when I said “frequentists believe …” I could never really be correct because various frequentists believe various different things.
Perhaps we could agree on the following statement: “Probabilities such as P(hypothesis) are never needed to do frequentist analysis.”
For example, the link you gave suggests the following as a characterisation of frequentism:
Since frequency guarantees are typically of the form “for each possible true value of theta doing the construction blah on the data will, with probability at least 1-p, yield a result with property blah”. Then since this must hold true for each theta, the distribution for the true value of theta is irrelevant.
The interesting questions to me are: (a) “what is the steelman of the frequentist position?” (folks like Larry are useful here), and (b) “are there actually prominent frequentist statisticians who say stupid things?”
By (b) I mean “actually stupid under any reasonable interpretation.”
Quote from the url I linked:
“Keep your identity small”—advice familiar to a LW audience.
I guess you disagree with Larry’s take: B vs F is about goals not methods. I could do Bayesian looking things while having a frequentist interpretation in mind.
In the spirit of collaborative argumentation, can we agree on the following:
We have better things to do than engage in identity politics.