We live at a time where up to 70% of scientific research can’t be replicated. Frequentism might not be to blame for all of that, but it does play it’s part. There are issues such an the Bem paper about porno-precognition where frequentist techniques did suggest that porno-precognition is real but analysing Bems data with Bayesian methods suggested it’s not.
It seems to me that there’s a bigger risk from Bayesian methods. They’re more sensitive to small effect sizes (doing a frequentist meta-analysis you’d count a study that got a p=0.1 result as evidence against, doing a bayesian one it might be evidence for). If the prior isn’t swamped then it’s important and we don’t have good best practices for choosing priors; if the prior is swamped then the bayesianism isn’t terribly relevant. And simply having more statistical tools available and giving researchers more choices makes it easier for bias to creep in.
Bayes’ theorem is true (duh) and I’d accept that there are situations where bayesian analysis is more effective than frequentist, but I think it would do more harm than good in formal science.
doing a frequentist meta-analysis you’d count a study that got a p=0.1 result as evidence against
Why would you do that? If I got a p=0.1 result doing a meta-analysis, I wouldn’t be surprised at all since things like random-effects means it takes a lot of data to turn in a positive result at the arbitrary threshold of 0.05. And as it happens, in some areas, an alpha of 0.1 is acceptable: for example, because of the poor power of tests for publication bias, you can find respected people like Ioannides using that particular threshold (I believe I last saw that in his paper on the binomial test for publication bias).
If people really acted that way, we’d see odd phenomenon where people saw successive meta-analysts on whether grapes cure cancer: 0.15 that grapes cure cancer (decreases belief grapes cure cancer), 0.10 (decreases), 0.07 (decreases), someone points out that random-effects is inappropriate because studies show very low heterogeneity and the better fixed-effects analysis suddenly reveals that the true p-value is now at 0.05 (everyone’s beliefs radically flip as they go from ‘grapes have been refuted and are quack alt medicine!’ to ‘grapes cure cancer! quick, let’s apply to the FDA under a fast track’). Instead, we see people acting more like Bayesians...
And simply having more statistical tools available and giving researchers more choices makes it easier for bias to creep in.
Is that a guess, or a fact based on meta-studies showing that Bayesian-using papers cook the books more than NHST users with p-hacking etc?
everyone’s beliefs radically flip as they go from ‘grapes have been refuted and are quack alt medicine!’ to ‘grapes cure cancer! quick, let’s apply to the FDA under a fast track’
Turns out I am overoptimistic and in some cases people have done just that: interpreted a failure to reject the null (due to insufficient power, despite being evidence for an effect) as disproving the alternative in a series of studies which all point the same way, only changing their minds when an individually big enough study comes out. Hauer says this is exactly what happened with a series of studies on traffic mortalities.
(As if driving didn’t terrify me enough, now I realize traffic laws and road safety designs are being engineered by vulgarized NHST practitioners who apparently don’t know how to patch the paradigm up with emphasis on power or meta-analysis.)
It seems to me that there’s a bigger risk from Bayesian methods. They’re more sensitive to small effect sizes (doing a frequentist meta-analysis you’d count a study that got a p=0.1 result as evidence against, doing a bayesian one it might be evidence for). If the prior isn’t swamped then it’s important and we don’t have good best practices for choosing priors; if the prior is swamped then the bayesianism isn’t terribly relevant. And simply having more statistical tools available and giving researchers more choices makes it easier for bias to creep in.
Bayes’ theorem is true (duh) and I’d accept that there are situations where bayesian analysis is more effective than frequentist, but I think it would do more harm than good in formal science.
Why would you do that? If I got a p=0.1 result doing a meta-analysis, I wouldn’t be surprised at all since things like random-effects means it takes a lot of data to turn in a positive result at the arbitrary threshold of 0.05. And as it happens, in some areas, an alpha of 0.1 is acceptable: for example, because of the poor power of tests for publication bias, you can find respected people like Ioannides using that particular threshold (I believe I last saw that in his paper on the binomial test for publication bias).
If people really acted that way, we’d see odd phenomenon where people saw successive meta-analysts on whether grapes cure cancer: 0.15 that grapes cure cancer (decreases belief grapes cure cancer), 0.10 (decreases), 0.07 (decreases), someone points out that random-effects is inappropriate because studies show very low heterogeneity and the better fixed-effects analysis suddenly reveals that the true p-value is now at 0.05 (everyone’s beliefs radically flip as they go from ‘grapes have been refuted and are quack alt medicine!’ to ‘grapes cure cancer! quick, let’s apply to the FDA under a fast track’). Instead, we see people acting more like Bayesians...
Is that a guess, or a fact based on meta-studies showing that Bayesian-using papers cook the books more than NHST users with p-hacking etc?
Turns out I am overoptimistic and in some cases people have done just that: interpreted a failure to reject the null (due to insufficient power, despite being evidence for an effect) as disproving the alternative in a series of studies which all point the same way, only changing their minds when an individually big enough study comes out. Hauer says this is exactly what happened with a series of studies on traffic mortalities.
(As if driving didn’t terrify me enough, now I realize traffic laws and road safety designs are being engineered by vulgarized NHST practitioners who apparently don’t know how to patch the paradigm up with emphasis on power or meta-analysis.)
No. The most basic version of meta-analysis is, roughly, that if you have two p=0.1 studies, the combined conclusion is p=0.01.