Nope. This was a good point by Jaynes. The truth may not exist in your hypothesis space. It may be (and often is) something you haven’t conceived of.
If the truth doesn’t exist in your hypothesis space then Bayesian methods are just as screwed as frequentist methods. In fact, Bayesian methods can grow increasingly confident that an incorrect hypothesis is true in this case. I don’t see how this is a weakness of Matt’s argument.
The details are hazy at this point, but by assigning a realistic probability to the “Something else” hypothesis, you avoid making over confident estimates of your other hypotheses in a multiple hypothesis testing scenario.
See Multiple Hypothesis Testing in Jaynes PTTLOS, starting pg. 98, and the punchline on pg. 105:
In summary, the role of our new hypothesis C was only to be held in abeyace until needed, like a fire extinguisher. In a normal testing situation, it is “dead”, playing no part in the inference because its probability remains far below that of the other hypotheses. But a dead hypothesis can be brought back to life by very unexpected data.
I think this is especially relevant to standard “null hypothesis” hypothesis testing because the likelihood of the data under the alternative hypothesis is never calculated, so you don’t even get a hint that your model might just suck, and instead conclude that the null hypothesis should be rejected.
What is the likelihood of the “something else” hypothesis? I don’t think this is really a general remedy.
Also, you can get the same thing in the hypothesis testing framework by doing two hypothesis tests, one of which is a comparison to the “something else” hypothesis and one of which is a comparison to the original null hypothesis.
Finally, while I forgot to mention this above, in most cases where hypothesis testing is applied, you actually are considering all possibilities, because you are doing something like P0 = “X ⇐ 0”, P1 = “X > 0″ and these really are logically the only possibilities =) [although I guess often you need to make some assumptions on the probabilistic dependencies among your samples to get good bounds].
Yes, you can say it in that framework. And you should. That’s part of the steelmanning exercise—putting in the things that are missing. If you steelman enough, you get to be a good bayesian.
P0 = “X ⇐ 0” and {All My other assumptions} NOT(P0) = NOT(“X ⇐ 0″) or NOT({All My other assumptions})
If the truth doesn’t exist in your hypothesis space then Bayesian methods are just as screwed as frequentist methods. In fact, Bayesian methods can grow increasingly confident that an incorrect hypothesis is true in this case. I don’t see how this is a weakness of Matt’s argument.
The details are hazy at this point, but by assigning a realistic probability to the “Something else” hypothesis, you avoid making over confident estimates of your other hypotheses in a multiple hypothesis testing scenario.
See Multiple Hypothesis Testing in Jaynes PTTLOS, starting pg. 98, and the punchline on pg. 105:
I think this is especially relevant to standard “null hypothesis” hypothesis testing because the likelihood of the data under the alternative hypothesis is never calculated, so you don’t even get a hint that your model might just suck, and instead conclude that the null hypothesis should be rejected.
What is the likelihood of the “something else” hypothesis? I don’t think this is really a general remedy.
Also, you can get the same thing in the hypothesis testing framework by doing two hypothesis tests, one of which is a comparison to the “something else” hypothesis and one of which is a comparison to the original null hypothesis.
Finally, while I forgot to mention this above, in most cases where hypothesis testing is applied, you actually are considering all possibilities, because you are doing something like P0 = “X ⇐ 0”, P1 = “X > 0″ and these really are logically the only possibilities =) [although I guess often you need to make some assumptions on the probabilistic dependencies among your samples to get good bounds].
Yes, you can say it in that framework. And you should. That’s part of the steelmanning exercise—putting in the things that are missing. If you steelman enough, you get to be a good bayesian.
P0 = “X ⇐ 0” and {All My other assumptions}
NOT(P0) = NOT(“X ⇐ 0″) or NOT({All My other assumptions})