Getting to an actual confidence level in the hypothesis requires having a prior.
All you’re saying is that studies should use Bayesian statistics. No medical journal articles use Bayesian statistics.
Given that the frequentist approach behind these tests is “correct”, the article’s claim is incorrect. The authors intended to use frequentist statistics, and so they made an error.
If a weak default complexity prior of 1 nat for 1 extra variable wipes out 84% confidence, that implies that many articles have incorrect conclusions, because 95% confidence might not be enough to account for a one-variable complexity penalty.
In any case, you are still incorrect, because your penalty cannot prove that the null hypothesis is correct. It can only make it harder to prove it’s incorrect. Failure to prove that it is incorrect is not proof that it is correct. Which is a key point of this post.
Nah, they’re welcome to use whichever statistics they like. We might point out interpretation errors, though, if they make any.
Under the assumptions I described, a p-value of 0.16 is about 0.99 nats of evidence which is essentially canceled by the 1 nat prior. A p-value of 0.05 under the same assumptions would be about 1.92 nats of evidence, so if there’s a lot of published science that matches those assumptions (which is dubious), then they’re merely weak evidence, not necessarily wrong.
It’s not the job of the complexity penalty to “prove the null hypothesis is correct”. Proving what’s right and what’s wrong is a job for evidence. The penalty was merely a cheap substitute for an informed prior.
All you’re saying is that studies should use Bayesian statistics. No medical journal articles use Bayesian statistics.
Given that the frequentist approach behind these tests is “correct”, the article’s claim is incorrect. The authors intended to use frequentist statistics, and so they made an error.
If a weak default complexity prior of 1 nat for 1 extra variable wipes out 84% confidence, that implies that many articles have incorrect conclusions, because 95% confidence might not be enough to account for a one-variable complexity penalty.
In any case, you are still incorrect, because your penalty cannot prove that the null hypothesis is correct. It can only make it harder to prove it’s incorrect. Failure to prove that it is incorrect is not proof that it is correct. Which is a key point of this post.
Nah, they’re welcome to use whichever statistics they like. We might point out interpretation errors, though, if they make any.
Under the assumptions I described, a p-value of 0.16 is about 0.99 nats of evidence which is essentially canceled by the 1 nat prior. A p-value of 0.05 under the same assumptions would be about 1.92 nats of evidence, so if there’s a lot of published science that matches those assumptions (which is dubious), then they’re merely weak evidence, not necessarily wrong.
It’s not the job of the complexity penalty to “prove the null hypothesis is correct”. Proving what’s right and what’s wrong is a job for evidence. The penalty was merely a cheap substitute for an informed prior.