Some general comments about medical research. Source: I have studied the statistics books in detail, and have read several cubic meters of medical papers and learned most of the lessons the hard way.
When reading medical papers look for
1. Funding sources for the study or for the authors of the study (e.g. “speaking fees” and “consulting fees”). He who pays the piper calls the tune.
2. Statistical incompetence, which is rife in medical research. For example, you routinely see “lack of statistical significance” interpreted as “proof of no effect”.
3. Pre publication of the study design, end points and intended statistical analysis. There is a lot of scope to move the goalposts and engage in p-hacking and other nefarious activities.
4. Differences between the abstract and the text. Often you can read the abstract and wonder if it refers to the same paper .
5. In meta-analyses look for whether the selection criteria were adhered to or not or whether subjective criteria were used to exclude inconvenient studies.
6. Financial interests. For example it is notable that countries like India, that make generic drugs, appear to be more favourable to generic drugs. Meanwhile in the US, there seems to be a strong bias in favour of drugs in patent.
7. Read the methods section very carefully. Once you have read enough papers this will become instinctive.
8. Be ready for the vast majority of papers to be of low quality and worthless.
9. I routinely see studies rigged to deliver a predetermined outcome. For example, if you want to find a non-statistically significant effect which can be misrepresented as “no effect”, then run a small study, for a short period, and use suboptimal doses or take other measures to minimize differences between the groups compared.
This all sounds rather grim, an extreme case of the hype and uneven quality that probably afflicts all research areas now… Number 8 seems especially grim, even though it doesn’t involve outright corruption, since it means that any counter-institution trying to do quality control will be overwhelmed by the sheer quantity of papers… Nonetheless: What you describe is a way to check the quality of an individual paper. Is there any kind of resource that reliably turns up high-quality papers? Perhaps literature reviews or citation counts?
Is there any kind of resource that reliably turns up high-quality papers?
No you just have to filter. In any particular field you get to know the agendas and limitations of many of the researchers. X is a shill for company Y, A pushes the limits for p hacking, B has a fixed mindset about low fat diets. etc. Some researchers also tend to produce me-too and derivative papers, others are more innovative.
Also you do get quicker at spotting the fatal flaw.
In finance there are blogs that pick out recent good papers; these are a huge time saver (e.g. Alpha Architect which I have mentioned before).
Some general comments about medical research. Source: I have studied the statistics books in detail, and have read several cubic meters of medical papers and learned most of the lessons the hard way.
When reading medical papers look for
1. Funding sources for the study or for the authors of the study (e.g. “speaking fees” and “consulting fees”). He who pays the piper calls the tune.
2. Statistical incompetence, which is rife in medical research. For example, you routinely see “lack of statistical significance” interpreted as “proof of no effect”.
3. Pre publication of the study design, end points and intended statistical analysis. There is a lot of scope to move the goalposts and engage in p-hacking and other nefarious activities.
4. Differences between the abstract and the text. Often you can read the abstract and wonder if it refers to the same paper .
5. In meta-analyses look for whether the selection criteria were adhered to or not or whether subjective criteria were used to exclude inconvenient studies.
6. Financial interests. For example it is notable that countries like India, that make generic drugs, appear to be more favourable to generic drugs. Meanwhile in the US, there seems to be a strong bias in favour of drugs in patent.
7. Read the methods section very carefully. Once you have read enough papers this will become instinctive.
8. Be ready for the vast majority of papers to be of low quality and worthless.
9. I routinely see studies rigged to deliver a predetermined outcome. For example, if you want to find a non-statistically significant effect which can be misrepresented as “no effect”, then run a small study, for a short period, and use suboptimal doses or take other measures to minimize differences between the groups compared.
This all sounds rather grim, an extreme case of the hype and uneven quality that probably afflicts all research areas now… Number 8 seems especially grim, even though it doesn’t involve outright corruption, since it means that any counter-institution trying to do quality control will be overwhelmed by the sheer quantity of papers… Nonetheless: What you describe is a way to check the quality of an individual paper. Is there any kind of resource that reliably turns up high-quality papers? Perhaps literature reviews or citation counts?
No you just have to filter. In any particular field you get to know the agendas and limitations of many of the researchers. X is a shill for company Y, A pushes the limits for p hacking, B has a fixed mindset about low fat diets. etc. Some researchers also tend to produce me-too and derivative papers, others are more innovative.
Also you do get quicker at spotting the fatal flaw.
In finance there are blogs that pick out recent good papers; these are a huge time saver (e.g. Alpha Architect which I have mentioned before).