I’ve probably read about 1000 papers. Lessons learned the hard way...
1. Look at the sponsorship of the research and of the researchers (previous sponsorship, “consultancies” etc are also important for up to 10-15 years). This creates massive bias. E.g: A lot of medical bodies and researchers are owned by pharmaceutical companies
2. Look at ideological biases of the authors. E.g. a lot of social science research assumes as a given that genes have no effect on personality or intelligence. (Yes, really).
3. Understand statistics very deeply. There is no pain-free way to get this knowledge, but without it you cannot win here. E.g. a) The assumptions behind all the statistical models b) the limitations of alleged “corrections”. You need to understand both Bayesian and Frequentist statistics in depth, to the point that they are obvious and intuitive to you.
4. Understand how researchers rig results. e.g. undisclosed multiple comparisons, peeking at the data before deciding what analysis to do, failing to pre-publish the design and end points and to follow that pre-publication, “run-in periods” for drug trials, sponsor-controlled committees to review and change diagnoses… There are papers about this e.g. “why most published research findings are false”.
5. After sponsorship, read the methods section carefully. Look for problems. Have valid and appropriate statistics been used? Were the logical end points assessed? Maybe then look at the conclusions. Do the conclusions match the body of the paper? Has the data from the study been made available to all qualified researchers to check the analysis? Things can change a lot when that happens e.g. Tamiflu. Is the data is only available to commercial interests and their stooges this is a bad sign.
6. Has the study been replicated by independent researchers?
7. Is the study observational? If so, does is meet generally accepted criteria for valid observational studies? (large effect, dose-response gradient, well understood causal model, well understood confounders, confounders smaller than the published effect etc).
8. Do not think you can read abstracts only and learn much that is useful.
9. Read some of the vitriolic books about the problems in research e.g. “Deadly Medicines and Organised Crime How big pharma has corrupted healthcare” by PETER C GØTZSCHE. Not everything in this book is true but it will open your eyes about what can happen.
10. Face up to the fact that 80-90% of studies are useless or wrong. You will spend a lot of time reading things only to conclude that there is not much there.
(a minor thing—I used to have a separate MSWord file with a table for “techniques”. Some people prefer Excel and so on, but I find that Word helps me keep it laconic. The columns were: Species; Purpose; Fixation/Storage; Treatment; and Reference (with a hyperlink). Within Treatment I just highlighted specific terms. Very easy to see something out of the ordinary.)
I’ve probably read about 1000 papers. Lessons learned the hard way...
1. Look at the sponsorship of the research and of the researchers (previous sponsorship, “consultancies” etc are also important for up to 10-15 years). This creates massive bias. E.g: A lot of medical bodies and researchers are owned by pharmaceutical companies
2. Look at ideological biases of the authors. E.g. a lot of social science research assumes as a given that genes have no effect on personality or intelligence. (Yes, really).
3. Understand statistics very deeply. There is no pain-free way to get this knowledge, but without it you cannot win here. E.g. a) The assumptions behind all the statistical models b) the limitations of alleged “corrections”. You need to understand both Bayesian and Frequentist statistics in depth, to the point that they are obvious and intuitive to you.
4. Understand how researchers rig results. e.g. undisclosed multiple comparisons, peeking at the data before deciding what analysis to do, failing to pre-publish the design and end points and to follow that pre-publication, “run-in periods” for drug trials, sponsor-controlled committees to review and change diagnoses… There are papers about this e.g. “why most published research findings are false”.
5. After sponsorship, read the methods section carefully. Look for problems. Have valid and appropriate statistics been used? Were the logical end points assessed? Maybe then look at the conclusions. Do the conclusions match the body of the paper? Has the data from the study been made available to all qualified researchers to check the analysis? Things can change a lot when that happens e.g. Tamiflu. Is the data is only available to commercial interests and their stooges this is a bad sign.
6. Has the study been replicated by independent researchers?
7. Is the study observational? If so, does is meet generally accepted criteria for valid observational studies? (large effect, dose-response gradient, well understood causal model, well understood confounders, confounders smaller than the published effect etc).
8. Do not think you can read abstracts only and learn much that is useful.
9. Read some of the vitriolic books about the problems in research e.g. “Deadly Medicines and Organised Crime How big pharma has corrupted healthcare” by PETER C GØTZSCHE. Not everything in this book is true but it will open your eyes about what can happen.
10. Face up to the fact that 80-90% of studies are useless or wrong. You will spend a lot of time reading things only to conclude that there is not much there.
One of the most miserable things about the LW experience is realizing how little you actually know with confidence.
Very cool. How have these been split across different fields/domains?
Mostly medicine, nutrition, metabolism. Also finance and economics.
What kind of experiences were the hard lesson? How did the moments of learning look like?
Mostly belatedly realizing that studies I took as Gospel turned out to be wrong. This triggered an intense desire to know why and how.
This is a great answer and should be taught to everyone.
Is there an online way to better tag which studies are suspect and which ones aren’t—for the sake of everyone else who reads after?
Check out PubPeer.
I am using https://scite.ai/ with a plugin for browsers, but I would love a similar service with user-generated flags.
(a minor thing—I used to have a separate MSWord file with a table for “techniques”. Some people prefer Excel and so on, but I find that Word helps me keep it laconic. The columns were: Species; Purpose; Fixation/Storage; Treatment; and Reference (with a hyperlink). Within Treatment I just highlighted specific terms. Very easy to see something out of the ordinary.)