One more example of misapplied statistics. You see a 5-standard-deviation signal for faster-than-light neutrinos and reason like this:
“Well, prior to seeing these data I would rate the odds of neutrinos traveling FTL as, say, 1:1000, but this is full 5 standard deviations so the likelihood ratio is about 3.5 million for them going FTL after all, so I must revise my belief and now accept that those neutrinos travel FTL with ~3500:1 odds.”
...which, of course, only happens because you have oversimplified by only considering two hypotheses. Whereas in reality you should also have thrown in some other possibilities, like e.g. some undiscovered flaw in the measurements. Which I would, prior to seeing the data, assign a higher probability, say, odds of 1:100 (a mere 1% probability of faulty experiment, just to be generous to the experimenters). Now after seeing the data we also have to revise this probability, and it comes out as ~35000:1 odds for faulty experimental design.
In other words, the more “statistically significant” is the result in such an experiment, the more it is evidence for faulty measurement and against the experimenters’ claim (here, FTL neutrinos).
In other words, the more “statistically significant” is the result in such an experiment, the more it is evidence for faulty measurement and against the experimenters’ claim (here, FTL neutrinos).
It’s still evidence for the claims, just also evidence for the experiment being faulty.
Technically, when viewed on the log-odds scale, it is exactly the same amount of evidence for either hypothesis.
On the (0;1) scale of probabilities it is, however, stronger evidence for flawed experiment. E.g. if we had a null hypothesis H0 at a prior P of 0.25 and two hypotheses H1 and H2 at 0.25 and 0.5, respectively, and we see evidence that has a likelihood of 5:1 for each of these hypotheses over H0. Then we have posterior P(H0|D)/P(H1|D)=1/5 and P(H0|D)/P(H2|D)=1/10, and after normalization, P(H0)=1/16, P(H1)=5/16, P(H2)=10/16. So, in this situation, P(H2) has increased by 2⁄16 and P(H1) only by 1⁄16. Which is what I meant in my comment: the probability of the more likely hypothesis increases faster, so H2 becomes more more probable than H1 :) Although on the log-odds scale, of course, both H1 and H2 received the same amount of evidence.
One more example of misapplied statistics. You see a 5-standard-deviation signal for faster-than-light neutrinos and reason like this:
“Well, prior to seeing these data I would rate the odds of neutrinos traveling FTL as, say, 1:1000, but this is full 5 standard deviations so the likelihood ratio is about 3.5 million for them going FTL after all, so I must revise my belief and now accept that those neutrinos travel FTL with ~3500:1 odds.”
...which, of course, only happens because you have oversimplified by only considering two hypotheses. Whereas in reality you should also have thrown in some other possibilities, like e.g. some undiscovered flaw in the measurements. Which I would, prior to seeing the data, assign a higher probability, say, odds of 1:100 (a mere 1% probability of faulty experiment, just to be generous to the experimenters). Now after seeing the data we also have to revise this probability, and it comes out as ~35000:1 odds for faulty experimental design.
In other words, the more “statistically significant” is the result in such an experiment, the more it is evidence for faulty measurement and against the experimenters’ claim (here, FTL neutrinos).
It’s still evidence for the claims, just also evidence for the experiment being faulty.
Technically, when viewed on the log-odds scale, it is exactly the same amount of evidence for either hypothesis.
On the (0;1) scale of probabilities it is, however, stronger evidence for flawed experiment. E.g. if we had a null hypothesis H0 at a prior P of 0.25 and two hypotheses H1 and H2 at 0.25 and 0.5, respectively, and we see evidence that has a likelihood of 5:1 for each of these hypotheses over H0. Then we have posterior P(H0|D)/P(H1|D)=1/5 and P(H0|D)/P(H2|D)=1/10, and after normalization, P(H0)=1/16, P(H1)=5/16, P(H2)=10/16. So, in this situation, P(H2) has increased by 2⁄16 and P(H1) only by 1⁄16. Which is what I meant in my comment: the probability of the more likely hypothesis increases faster, so H2 becomes more more probable than H1 :) Although on the log-odds scale, of course, both H1 and H2 received the same amount of evidence.