Skills: Learn both bayesian and frequentist statistics. E T Jaynes’s book, also Gelman’s Bayesian Data Analysis, and any solid frequentist textbook e.g. Goodman Teach Yourself Statistics 1972 edition. Also Judea Pearl Causality. Read the papers critiquing current methods (why most published research findings are false, the recent papers criticising the use of P values).
You will need calculus and linear algebra to get far but for reading the medical literature you can probably ignore measure theory.
Heuristics: Look at sponsorship, both for the study itself and for the researchers (speaking fees, sponsorship of other papers. This massively skews results.
Look for ideological or prior commitments by authors. This also massively skews results.
Look out for p hacking / garden of forking paths i.e. degrees of freedom that result in ‘significant’ results being claimed when this is not valid.
Understand the difference between statistical significance and practical significance. Understand how arbitrary the 5% threshold for statistical significance is. Understand that a result falling short of statistical significance may actually be evidence *for* an effect. No significant effect /= no effect, may mean probably is an effect.
Understand how little most medical people from GP to professors know about statistics and how often basic statistical errors occur in the literature (e.g. lack of statistical significant taken to be disproof as in the Vioxx debacle).
Read the methods section first. Don’t read the results part of the abstract or if you do, check that all the claims made are backed up by the body of the paper.
When reading meta-analyses look hard at the papers they are based on—you cannot make silk from sows ears. Be very wary of any study that has not been replicated by independent researchers.
Be aware of the extreme weaknesses of epidemiological and observational studies and be very sceptical of claims to have “controlled for” some variable. Such attempts are usually miserable failures, invalid and can make things actually worse. See Pearl’s book.
A replication will always cite the original study. Google scholar can show you all studies that cite a given page and that list is often a good place to look.
Usually conflicts of interest and funding are disclosed (these days) in the paper. Usually I go there first, before the second step which is reading the methods section.
There are also registers of funding for medical researchers.
Skills: Learn both bayesian and frequentist statistics. E T Jaynes’s book, also Gelman’s Bayesian Data Analysis, and any solid frequentist textbook e.g. Goodman Teach Yourself Statistics 1972 edition. Also Judea Pearl Causality. Read the papers critiquing current methods (why most published research findings are false, the recent papers criticising the use of P values).
You will need calculus and linear algebra to get far but for reading the medical literature you can probably ignore measure theory.
Heuristics: Look at sponsorship, both for the study itself and for the researchers (speaking fees, sponsorship of other papers. This massively skews results.
Look for ideological or prior commitments by authors. This also massively skews results.
Look out for p hacking / garden of forking paths i.e. degrees of freedom that result in ‘significant’ results being claimed when this is not valid.
Understand the difference between statistical significance and practical significance. Understand how arbitrary the 5% threshold for statistical significance is. Understand that a result falling short of statistical significance may actually be evidence *for* an effect. No significant effect /= no effect, may mean probably is an effect.
Understand how little most medical people from GP to professors know about statistics and how often basic statistical errors occur in the literature (e.g. lack of statistical significant taken to be disproof as in the Vioxx debacle).
Read the methods section first. Don’t read the results part of the abstract or if you do, check that all the claims made are backed up by the body of the paper.
When reading meta-analyses look hard at the papers they are based on—you cannot make silk from sows ears. Be very wary of any study that has not been replicated by independent researchers.
Be aware of the extreme weaknesses of epidemiological and observational studies and be very sceptical of claims to have “controlled for” some variable. Such attempts are usually miserable failures, invalid and can make things actually worse. See Pearl’s book.
Practically speaking, how might I go about checking if a study has been replicated independently?
A replication will always cite the original study. Google scholar can show you all studies that cite a given page and that list is often a good place to look.
I tend to search “<title of study> replication” in Google, as well as “<core claim of the study> replication”
how do you find the sponsorships of studies and researchers?
Usually conflicts of interest and funding are disclosed (these days) in the paper. Usually I go there first, before the second step which is reading the methods section.
There are also registers of funding for medical researchers.
Australia
https://ses.library.usyd.edu.au/handle/2123/20224
https://ses.library.usyd.edu.au/handle/2123/20223
US
https://openpaymentsdata.cms.gov/
But it is imperfect
https://www.nytimes.com/2018/12/08/health/medical-journals-conflicts-of-interest.html
and of course disclosure is not a complete answer. Disclosed funding greatly affects the reported results.