Rather than speculating in the abstract, the best solution is perhaps to try to come up with concrete standardization ideas, and discuss whether they would work.
I’m generally in favor of these. The recommendations are usually quite reasonable, and a tremendous amount of research neglects the basics or adds in pointless variation—certainly, people will plead that methodologist-nazis will cramp their style, but when I try to meta-analyze creatine and find that in the ~10 studies, they collectively employ something like 31 different endpoints (some simply made up by that particular set of researchers), most for no compelling reason, so that only 2 metrics were used in >3 studies, it’s hard for me to take this concern seriously—we clearly are nowhere near an equilibrium where researchers are being herded into using all the same inappropriate metrics.
he recommendations are usually quite reasonable, and a tremendous amount of research neglects the basics or adds in pointless variation—certainly, people will plead that methodologist-nazis will cramp their style, but when I try to meta-analyze creatine and find that in the ~10 studies, they collectively employ something like 31 different endpoints (some simply made up by that particular set of researchers), most for no compelling reason, so that only 2 metrics were used in >3 studies, it’s hard for me to take this concern seriously—we clearly are nowhere near an equilibrium where researchers are being herded into using all the same inappropriate metrics.
But when the fools begin their invasion, some communities think themselves too good to use their banhammer for—gasp!—censorship.
After all—anyone acculturated by academia knows that censorship is a very grave sin… in their walled gardens where it costs thousands and thousands of dollars to enter, and students fear their professors’ grading, and heaven forbid the janitors should speak up in the middle of a colloquium.
It is easy to be naive about the evils of censorship when you already live in a carefully kept garden. Just like it is easy to be naive about the universal virtue of unconditional nonviolent pacifism, when your country already has armed soldiers on the borders, and your city already has police. It costs you nothing to be righteous, so long as the police stay on their jobs.
Standardization of techniques and terminology could be seen as a kind of “censorship” (certainly it is seen as such by the people who talk of methodologist-nazis). Just as academics fail to see how many advantages the kind of censorship the academia’s system for excluding ignorants has, it is easy to fail to see the many advantages the standardization of terminology and techniques we already have brings with it. The present academic system, and certain terminology and techniques, have become so natural to us that we fail to see that they are the result of a process which in effect functioned as a kind of censorship (“don’t do so and so if you want to be taken seriously”). Hence we fail to see how many advantages such censorship has over anarchism.
Or look at existing efforts. There’s two main categories I know of, reporting checklists and quality-evaluation checklists (in addition to the guidelines/recommendations published by professional groups like the APA’s manual based apparently on JARS or AERA’s standards).
Some reporting checklists:
STROBE (checklists; justification) for cohort, case-control, and cross-sectional studies
TREND (checklist; justification) for non-randomized experiments
CONSORT (checklist; justification) for RCTs
QUOROM/PRISMA (checklist; justification) for reviews/meta-analyses
Some quality-evaluation scales:
Jadad scale
NOS (scale; manual)
CEBM
I’m generally in favor of these. The recommendations are usually quite reasonable, and a tremendous amount of research neglects the basics or adds in pointless variation—certainly, people will plead that methodologist-nazis will cramp their style, but when I try to meta-analyze creatine and find that in the ~10 studies, they collectively employ something like 31 different endpoints (some simply made up by that particular set of researchers), most for no compelling reason, so that only 2 metrics were used in >3 studies, it’s hard for me to take this concern seriously—we clearly are nowhere near an equilibrium where researchers are being herded into using all the same inappropriate metrics.
Thanks, very good post.
I agree. This reminds me of this post:
Standardization of techniques and terminology could be seen as a kind of “censorship” (certainly it is seen as such by the people who talk of methodologist-nazis). Just as academics fail to see how many advantages the kind of censorship the academia’s system for excluding ignorants has, it is easy to fail to see the many advantages the standardization of terminology and techniques we already have brings with it. The present academic system, and certain terminology and techniques, have become so natural to us that we fail to see that they are the result of a process which in effect functioned as a kind of censorship (“don’t do so and so if you want to be taken seriously”). Hence we fail to see how many advantages such censorship has over anarchism.
Excellent material as usual, thank you.