The “false positive psychology” paper above better meets that description: they’re both intentional satires to make a methodological point. Readers should not be confused about the fact that Bem really believes in his conclusions (he has a long history of publishing on this stuff, inconsistent with a Sokal affair), and people like Ben Goertzel and Damien Broderick have vigorously promoted Bem 2011 for the psi, not as a methodological warning.
As I said myself in the post! There is no disagreement on that, but there is an additional factor which I mentioned. The point was that if papers 1, 2, and 3 all display property A, while only 1 and 2 display property B, then 2 is prima facie more similar to 1 than 3.
I don’t understand the analogy here. “Parachute use...” takes a question with an obvious answer and complains about the lack of rigorously obtained results pertaining to that question in order to ridicule people who went over the top in demanding rigor.
The only way to connect that with Bem’s paper that comes to mind is to claim that Bem was trolling, he believes PSI to be obviously nonexistent and did the study to show that you can get all sorts of obviously wrong but scientific™, legitimate looking results if you experiment cleverly enough. Is that what you mean? Because here I was sure that he had been deadly serious about this stuff.
We, the readers, can take both the parachute paper and the Bem paper as highlighting flaws and limitations of standard methods (in psychology and evidence-based medicine) by using them to derive bogus conclusions (don’t use parachutes, and precognition is real). Likewise with the “False Positive Psychology” paper at the top.
But saying that Bem’s paper wasn’t about parapsychology suggests that he intended it as a warning against flawed methods just like the parachute people did. That looks like defending people who do bad science by saying “it was all a joke, really!”
Quote: “Conclusions As with many interventions intended to prevent ill health, the effectiveness of parachutes has not been subjected to rigorous evaluation by using randomised controlled trials. Advocates of evidence-based medicine have criticised the adoption of interventions evaluated by using only observational data. We think that everyone might benefit if the most radical protagonists of evidence based medicine organised and participated in a double blind, randomised, placebo controlled, crossover trial of the parachute.”
So how does one decide whether a paper’s claim is over the top? For much of the middle of the 20th century, one might apply such a label to plate tectonics. Later, one could see someone plausibly applying such a label to papers with evidence that ulcers were caused by bacteria rather than stress. Completely ignoring papers in this fashion seems like an unreliable heuristic.
Bem 2011 was as much a paper on parapsychology as “Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomised controlled trials” was a paper on aviation safety.
The “false positive psychology” paper above better meets that description: they’re both intentional satires to make a methodological point. Readers should not be confused about the fact that Bem really believes in his conclusions (he has a long history of publishing on this stuff, inconsistent with a Sokal affair), and people like Ben Goertzel and Damien Broderick have vigorously promoted Bem 2011 for the psi, not as a methodological warning.
By principle of death of the author, Bem 2011 is a methodological warning regardless of what Bem says outside the paper.
As I said myself in the post! There is no disagreement on that, but there is an additional factor which I mentioned. The point was that if papers 1, 2, and 3 all display property A, while only 1 and 2 display property B, then 2 is prima facie more similar to 1 than 3.
You don’t need to go so far as let the author be dead. Mere applicability theory will accomplish this.
I don’t understand the analogy here. “Parachute use...” takes a question with an obvious answer and complains about the lack of rigorously obtained results pertaining to that question in order to ridicule people who went over the top in demanding rigor.
The only way to connect that with Bem’s paper that comes to mind is to claim that Bem was trolling, he believes PSI to be obviously nonexistent and did the study to show that you can get all sorts of obviously wrong but scientific™, legitimate looking results if you experiment cleverly enough. Is that what you mean? Because here I was sure that he had been deadly serious about this stuff.
We, the readers, can take both the parachute paper and the Bem paper as highlighting flaws and limitations of standard methods (in psychology and evidence-based medicine) by using them to derive bogus conclusions (don’t use parachutes, and precognition is real). Likewise with the “False Positive Psychology” paper at the top.
But saying that Bem’s paper wasn’t about parapsychology suggests that he intended it as a warning against flawed methods just like the parachute people did. That looks like defending people who do bad science by saying “it was all a joke, really!”
Quote: “Conclusions As with many interventions intended to prevent ill health, the effectiveness of parachutes has not been subjected to rigorous evaluation by using randomised controlled trials. Advocates of evidence-based medicine have criticised the adoption of interventions evaluated by using only observational data. We think that everyone might benefit if the most radical protagonists of evidence based medicine organised and participated in a double blind, randomised, placebo controlled, crossover trial of the parachute.”
Can’t this be used as a fully general excuse for any bad paper, intentional or not?
I see lesswrong has gone full reddit, with people getting downvoted just because etc.
Anyway, this excuse works only for papers with over the top claims that follow accepted methodology very well. There are extremely few such papers.
So how does one decide whether a paper’s claim is over the top? For much of the middle of the 20th century, one might apply such a label to plate tectonics. Later, one could see someone plausibly applying such a label to papers with evidence that ulcers were caused by bacteria rather than stress. Completely ignoring papers in this fashion seems like an unreliable heuristic.