You need to provide links because I read a fair bit on the subject and don’t recall this.
That’s a fair request. I don’t really have the time to go digging for those details, though. If you feel so inspired, again I’d point to the work done at the Stanford Research Institute (or at least I think it was that) where they did a ridiculous number of trials of all kinds and did get several standard deviations away from the expected mean predicted based on the null hypothesis. I honestly don’t remember the numbers at all, so you could be right that there has never been anything like a six-s.d. deviation in parapsychological experiments. I seem to recall that they got somewhere around ten—but it has been something like six years since I read anything on this topic.
That said, I get the feeling there’s a bit of goalpost-moving going on in this discussion. In Eliezer’s original reference to parapsychology as the control group for science, his point was that there are some amazingly subjective effects that come into play with frequentist statistics that could account for even the good (by frequentist standards) positive-result studies from parapsychology. I agree, there’s a lot of problem with things like publication bias and the like, and that does offer an explanation for a decent chunk of parapsychology’s material. But to quote Eliezer:
Parapsychology, the control group for science, would seem to be a thriving field with “statistically significant” results aplenty. Oh, sure, the effect sizes are minor. Sure, the effect sizes get even smaller (though still “statistically significant”) as they collect more data. Sure, if you find that people can telekinetically influence the future, a similar experimental protocol is likely to produce equally good results for telekinetically influencing the past. Of which I am less tempted to say, “How amazing! The power of the mind is not bound by time or causality!” and more inclined to say, “Bad statistics are time-symmetrical.” But here’s the thing: Parapsychologists are constantly protesting that they are playing by all the standard scientific rules, and yet their results are being ignored—that they are unfairly being held to higher standards than everyone else. I’m willing to believe that. It just means that the standard statistical methods of science are so weak and flawed as to permit a field of study to sustain itself in the complete absence of any subject matter.
I haven’t looked at the CERN group’s methods in enough detail to know if they’re making the same kind of error. I’m just trying to point out that we can’t assign an abysmally low probability to their making a common kind of statistical error that finds a small-but-low-p-value effect without simultaneously assigning a lower probability to parapsychologists making this same mistake than Eliezer seems to.
And to be clear, I am not saying “Either the CERN group made statistical errors or telepathy exists.” Nor am I trying to defend parapsychology. I’m simply pointing out that we have to be even-handed in our dismissal of low-p-value thinking.
Sure, if you find that people can telekinetically influence the future, a similar experimental protocol is likely to produce equally good results for telekinetically influencing the past. Of which I am less tempted to say, “How amazing! The power of the mind is not bound by time or causality!” and more inclined to say, “Bad statistics are time-symmetrical.”
That doesn’t actually strike me as all that much extra improbability. A whole bunch of the mechanisms would allow both!
That’s a fair request. I don’t really have the time to go digging for those details, though. If you feel so inspired, again I’d point to the work done at the Stanford Research Institute (or at least I think it was that) where they did a ridiculous number of trials of all kinds and did get several standard deviations away from the expected mean predicted based on the null hypothesis. I honestly don’t remember the numbers at all, so you could be right that there has never been anything like a six-s.d. deviation in parapsychological experiments. I seem to recall that they got somewhere around ten—but it has been something like six years since I read anything on this topic.
That said, I get the feeling there’s a bit of goalpost-moving going on in this discussion. In Eliezer’s original reference to parapsychology as the control group for science, his point was that there are some amazingly subjective effects that come into play with frequentist statistics that could account for even the good (by frequentist standards) positive-result studies from parapsychology. I agree, there’s a lot of problem with things like publication bias and the like, and that does offer an explanation for a decent chunk of parapsychology’s material. But to quote Eliezer:
I haven’t looked at the CERN group’s methods in enough detail to know if they’re making the same kind of error. I’m just trying to point out that we can’t assign an abysmally low probability to their making a common kind of statistical error that finds a small-but-low-p-value effect without simultaneously assigning a lower probability to parapsychologists making this same mistake than Eliezer seems to.
And to be clear, I am not saying “Either the CERN group made statistical errors or telepathy exists.” Nor am I trying to defend parapsychology. I’m simply pointing out that we have to be even-handed in our dismissal of low-p-value thinking.
That doesn’t actually strike me as all that much extra improbability. A whole bunch of the mechanisms would allow both!