Although I’m a little concerned that it, too, is attacking another strawman. At the beginning of chapter 37, it seems that the author just doesn’t understand what good researchers do. In the medical example given at the start of the chapter (458-462ish), many good researchers would use a one-sided hypothesis rather than a two-sided hypothesis (I would), which would better catch the weak relationship. One can also avoid false negatives by measuring the power of one’s test. McKay also claims that “this answer does not say how much more effective A is than B.” But that’s just false: one can get an idea of the size of the effect with either sharper techniques (like doing a linear regression, getting beta values and calculating r-squared) or just by modifying one’s null hypothesis (i.e. demanding that a datum improve on control by X amount before it counts in favor of the alternative hypothesis).
Given all that, I’m going to withhold judgment. McKay’s argument on the coin flip example is convincing on the surface. But given his history from the prior pages of understating the counterarguments, I’m not going to give it credence until I find a better statistician than I to give me the response, if any, from a “sampling theory” perspective.
Cyan, that source is slightly more convincing.
Although I’m a little concerned that it, too, is attacking another strawman. At the beginning of chapter 37, it seems that the author just doesn’t understand what good researchers do. In the medical example given at the start of the chapter (458-462ish), many good researchers would use a one-sided hypothesis rather than a two-sided hypothesis (I would), which would better catch the weak relationship. One can also avoid false negatives by measuring the power of one’s test. McKay also claims that “this answer does not say how much more effective A is than B.” But that’s just false: one can get an idea of the size of the effect with either sharper techniques (like doing a linear regression, getting beta values and calculating r-squared) or just by modifying one’s null hypothesis (i.e. demanding that a datum improve on control by X amount before it counts in favor of the alternative hypothesis).
Given all that, I’m going to withhold judgment. McKay’s argument on the coin flip example is convincing on the surface. But given his history from the prior pages of understating the counterarguments, I’m not going to give it credence until I find a better statistician than I to give me the response, if any, from a “sampling theory” perspective.