However, in other scenarios, Leek’s shortcut will give approximations that are clearly not right. Suppose that a study meets all of the criteria except that on one criterion it is fatally flawed. Suppose the sample size is far too small (e.g. there is only one participant), or rather than mortality and morbidity rates, the study measures something that doesn’t matter at all like physician satisfaction with the surveys. This can cripple the study entirely, and so the Bayes Factor should not just from 64 to 32 - it should fall to around 1. A fatally flawed study is no-longer useful evidence—it is no likelier to appear where the headline is true, in world 1, than in world 2 where it is false.”
I criticised it here: http://careyryan.com/fivethirtyeights-formula-for-decoding-health-news-makes-no-sense/ “Dr Leek’s shortcut gives a decent approximation of the Bayes Factor. The study, though not an RCT, is a large one showing a big positive effect on relevant metrics. We would be unlikely to encounter this evidence if the headline was false.
However, in other scenarios, Leek’s shortcut will give approximations that are clearly not right. Suppose that a study meets all of the criteria except that on one criterion it is fatally flawed. Suppose the sample size is far too small (e.g. there is only one participant), or rather than mortality and morbidity rates, the study measures something that doesn’t matter at all like physician satisfaction with the surveys. This can cripple the study entirely, and so the Bayes Factor should not just from 64 to 32 - it should fall to around 1. A fatally flawed study is no-longer useful evidence—it is no likelier to appear where the headline is true, in world 1, than in world 2 where it is false.”