Yeah, but value of information calculations have two terms. Even if changing our minds on psi provides huge value (and I agree it would), we still have to consider the probability that investigating psi will change our minds (miniscule). Bring in the opportunity cost of such an investigation and it’s a definite waste of time given a skeptical prior.
Why is that probability a priori miniscule? Something like 50% of people in the US believe in some form of psi and there’s an entire academic discipline devoted to researching it, many of the scientists in which are reputable and work at prestigious universities (and many have a background in physics). No psi effects that I know of are theoretically ruled out given our uncertainty about QM (even the retrocausal stuff fits in with many interpretations, e.g. the transactional interpretation) and even QM is incomplete. There is a history of unseen forces turning out to have huge harness-able effects, like nuclear power or electricity. There is also a general perception that the more science we do the less agency we see in the world, which is some evidence against psi. Overall I think the prior should be low but not miniscule, and that since the value of information is so high because psi is a phenomenon which if it exists would falsify many ingrained inductive biases, it would be very wise to look into the only phenomenon that could at least in principle falsify those biases.
(I might be rationalizing my conclusion somewhat here, as I do in fact know that psi probably exists. Before carefully looking into the issue I had basically the same beliefs as you do. Note that I find the probable existence of psi really annoying and didn’t start looking into the issue because I wanted to believe. Practical epistemology would be much easier in a psi-less world,)
, Before carefully looking into the issue I had basically the same beliefs as you do. Note that I find the probable existence of psi really annoying and didn’t start looking into the issue because I wanted to believe.
Questions like that can’t be answered due to problems of social epistemology. If I give you two pieces of evidence then it’s assumed I’ve given you my two most convincing pieces of evidence and that I don’t have any other pieces of evidence, and thus that you should update even further away from the conclusion that I’ve reached. If I regularly talked to you and could keep up a thread of conversation, and if both of us didn’t anticipate needless antagonism involving the presumption that the other hadn’t already considered things that they’d in fact considered many times (like “psi appears to be quite capricious, how convenient for believers”), then I could talk to you about the evidence without fear of unjustifiably cementing either of our beliefs. As it is you can only have that kind of relationship among decently close friends. Do you know of other ways of discussing tricky topics, ways that don’t have those requirements?
involving the presumption that the other hadn’t already considered things that they’d in fact considered many times (like “psi appears to be quite capricious, how convenient for believers”)
I assume you know most of the standard considerations and evidence. I think your belief in psi is probably the result of a bias in favor of such hypotheses (partly directly, and partly affecting your enthusiasm for other beliefs that feed into psi) but that you might have found something interesting while spelunking (which need not be convincing to be interesting).
Do you know of other ways of discussing tricky topics, ways that don’t have those requirements?
It would be easy to specify a bunch of stylized facts, and then incrementally add to the list or contest various elements. For instance, here’s a start:
A large proportion of the general population and academics report belief in psychic powers, although not as large as for religion, and various other falsified beliefs
Belief in psychic powers and psychic experiences track general gullibility (susceptibility to various sorts of fraud, witchcraft, jinn, etc, etc), lack of education, etc
*Selective memory, unconscious processing, lack of statistical thinking, wishful thinking, etc, explain at least most psychic reports; we would expect high belief in psychic powers even in their absence
If we start off with a wide prior for psi effect sizes, the puny-but-just-barely-almost-detectable range is quite small; since this is predicted by “no psi,” “no psi detectable by current methods,” and “just barely detectable psi” but the latter has much lower prior, we get a big update against psi reports being real
Parapsychologists are often able to produce small positive effects in noisy statistics-based experiments
Big effects are often revealed to be the result of cheating, or the work of labs that have cheated
Parapsychology suffers from anti-epistemology (even more than the already atrocious regular psychology), e.g. tweaking experiments until effects are found, retrospective data-dredging, etc; measure of publication bias, “decline effect” and so forth are quite bad
Not much mainstream effort goes into trying to replicate parapsychology results these days: there are few rewards for doing so, and the parapsychologists use idiosyncratic methods (the Daryl Bem article was unusual in being easier to replicate); those who bother are often strongly motivated skeptics frustrated with the widespread fraud and bad practices
We should expect bad science in parapsychology, since it’s so common elsewhere, there could be a psi effect far punier than even the parapsychologists claim the para
These effects are not reliably repeatable by others, particularly
New forces have been discovered in the past, as well as biological exploitation of their capabilities (magnetic navigation, electric eels, etc)
On the scales of biological organisms (as opposed to quantum gravity issues related to black holes, the Big Bang, etc) the Standard model does pretty well
There are proposed quantum mechanics interpretations with nonlocal effects; these interpretations still involve quantum mechanics working normally and making its predictions correctly, which leaves little room for psi powers to act as a God of the Gaps
The Simulation Hypothesis and allied claims allow for psi powers, or for Keanu Reeves to start flying around in the air, or for Thor and Zeus to appear on the 6 o’clock news; this depends on further claims about the frequency or measure of simulations and the relative abundance of worlds with psi powers that look bogus (as opposed to visible comic book style powers or the absence of psi); given other ways in which our world does not look like a fantasy story, the vastness of fictional genres, and non-fiction purposes for creating simulations, the Matrix-psi story is pretty low-prior
The evolutionary and anatomical picture for psi powers adds additional burden
A large proportion of the general population and academics report belief in psychic powers, although not as large as for religion, and various other falsified beliefs
I tentatively disagree that religion count as a falsified belief. This sounds nit-picky but I think indicates a substantial meta-level difference in our prior, i.e. the extent to which we think the average person is or isn’t crazy. E.g.:
Selective memory, unconscious processing, lack of statistical thinking, wishful thinking, etc, explain at least most psychic reports; we would expect high belief in psychic powers even in their absence
I don’t know whether to agree or disagree with this; I don’t really have much evidence either way. What do you have in mind? It’s clearly crucial, as I don’t see the parapsychology literature as providing very much evidence for psi, and you seem(?) to see it as providing some amount of evidence against psi. I looked through much of the heuristics and biases literature and was profoundly disappointed with what I found. Since then I’ve been skeptical of claims that the average person is highly susceptible to false beliefs. Why do you think people are so prone to “selective memory, unconscious processing, lack of statistical thinking, wishful thinking, etc”, beyond the psychology literature? Or am I wrong to be so unimpressed with the psychology literature?
New forces have been discovered in the past, as well as biological exploitation of their capabilities (magnetic navigation, electric eels, etc)
Recently, quantum stuff in biological systems generally. Only important to note because various proposed psi mechanisms involve quantum computation going on in the brain, which until recently seemed rather implausible but is now a serious possibility.
If we start off with a wide prior for psi effect sizes, the puny-but-just-barely-almost-detectable range is quite small; since this is predicted by “no psi,” “no psi detectable by current methods,” and “just barely detectable psi” but the latter has much lower prior, we get a big update against psi reports being real
Given that the methods used are practically designed so as to minimize any psi phenomena, e.g. taking away any real incentives for humans to use psi (with a few exceptions like a few of Bem’s experiments), why should we have a wide prior for psi effect sizes? E.g. Ben Goertzel’s analogy to experimentation on the alleged existence of “falling in love” comes to mind. I can’t find it but it was on his “Multiverse According to Ben” blog. Might you explain more why you think a wide prior seems good?
The Simulation Hypothesis and allied claims allow for psi powers, or for Keanu Reeves to start flying around in the air, or for Thor and Zeus to appear on the 6 o’clock news; this depends on further claims about the frequency or measure of simulations and the relative abundance of worlds with psi powers that look bogus (as opposed to visible comic book style powers or the absence of psi); given other ways in which our world does not look like a fantasy story, the vastness of fictional genres, and non-fiction purposes for creating simulations, the Matrix-psi story is pretty low-prior
Agreed. (Only noting this ’cuz I think you might think I think that simulation-style explanations are worth thinking about.)
The evolutionary and anatomical picture for psi powers adds additional burden
I don’t follow, might you flesh this out more?
Some other points:
There is a well-known experimenter effect where some experiments seem to get highly different results depending on which scientist is conducting them; a priori this would seem to be evidence of non-psi experimental manipulation, whether conscious or not.
It is well-known that psi seems to be react in agentic ways to attempts to experiment on it. This is really weird and seems to indicate flaws in the research rather than that psi is actually an agentic process that is trying to stay on the verge of detectability.
I tentatively disagree that religion count as a falsified belief. This sounds nit-picky but I think indicates a substantial meta-level difference in our prior, i.e. the extent to which we think the average person is or isn’t crazy.
The religions are contradictory. The behavior of the founders (moreso the more recent the religion) maps onto random guys creating cults today in great detail. People express belief in their particular religion being true, the particular founder or books being right, in young earth creationism, in Islamic medicine, in miraculous healing, etc. They do so at least as fervently as they do belief in psychic powers. You can try to make up a naturalistic version of religion that, while improbable on its face, doesn’t get much of a further penalty from the evidence (simulator deism, whatever) but that doesn’t rescue the accuracy of the masses.
On direct causes, consider sleep paralysis and alien abductions. Drug use can create experiences interpreted as religious or psychic or whatever, and various biological conditions can do likewise. False memories can be cultivated by therapists or group efforts (there was an interesting book by a Harvard researcher studying people who claimed to have been abducted). The formation of urban legends by the telephone effect and optimization for good stories is well-documented. Exposed frauds are pretty common, and frauds have to be more common than exposed ones, probably far more given the scarcity of debunking resources. These things vary from culture to culture, e.g. fan death giving an indication of how much people can shape their experiences and beliefs to fit memes floating around.
I looked through much of the heuristics and biases literature and was profoundly disappointed with what I found. Since then I’ve been skeptical of claims that the average person is highly susceptible to false beliefs.
Psychology research tends to be bad. Most of the effects claimed don’t exist (far more for repeatedly replicated experiments done by independent groups), and the effects that do exist are mostly greatly exaggerated in their effect sizes and generalizability. That is also true of heuristics and biases. And yet surveys (the basis of these “lots of people believe in ESP” claims) reveal that people have lots of demonstrably false beliefs, often strong beliefs, especially on topics where they get little feedback. Bryan Caplan has many such: basic facts about the economy, demographics, toxicology, risks of death. People believe having sex with virgins cures HIV, despite the absence of any actual documented examples. They form superstitions about homeopathic placebos and acupuncture.
People do not get feedback about their capricious/subtle psi beliefs (healing that heals amputees does get enough negative feedback that it’s a rare belief), and can get utility out of believing they have psi or that the world is magical, indeed will often go out of their way to protect the belief.
Recently, quantum stuff in biological systems generally. Only important to note because various proposed psi mechanisms involve quantum computation going on in the brain, which until recently seemed rather implausible but is now a serious possibility.
I contest. Biological systems are hot and noisy, and quantum effects are at the wrong scales in nervous systems. Second, quantum computation!= telepathy, ESP, telekinesis, etc. Usually these arguments are of the form “QM and ESP are mysterious, and could be related” or “this interpretation of QM assumes nonlocal effects, or even nonlocal effects driven by magic consciousness, so there’s a proof of concept for nonlocal ESP”. These are nonsense. I would want to see an example of a good argument for quantum computation-psi connections.
Given that the methods used are practically designed so as to minimize any psi phenomena, e.g. taking away any real incentives for humans to use psi (with a few exceptions like a few of Bem’s experiments), why should we have a wide prior for psi effect sizes? E.g. Ben Goertzel’s analogy to experimentation on the alleged existence of “falling in love” comes to mind. I can’t find it but it was on his “Multiverse According to Ben” blog. Might you explain more why you think a wide prior seems good?
Claims of psi powers tend to expand to fill the gaps of observation and alternative explanation. E.g. flying witches in Africa, jinn turning people into animals, etc. Claims that psi is small seem to be a “God of the Gaps” side-effect of modern recording technologies and other developments, rewriting the prior ex post. Repelling bullets, visible healing, visible telekinesis, and similar dramatic powers have often been claimed as effects “in the wild” but they always retreat from sight. We have casinos, the stock market, and similar opportunities for precognition, telekinesis could show up in sporting events, etc, etc.
E.g. Ben Goertzel’s analogy to experimentation on the alleged existence of “falling in love” comes to mind.
Actually, all sorts of nifty research on falling in love has been performed with useful predictions about oxytocin levels, fidelity, various sorts of future self-report, divorce, frequency of sex, etc, etc. Experiments in the lab can do quite a lot to distinguish couples in love or not, and to increase or decrease it. That’s more of a rhetorical trick, insofar as we use “falling in love” in a lot of signalling games and to some extent we’re unwilling to identify it with any natural kinds that actually exist.
The evolutionary and anatomical picture for psi powers adds additional burden
I don’t follow, might you flesh this out more?
Where is the animal psi? What organs do it? How do the inputs affect human responses and behavior? How did the capacity evolve in the first place, and if it did evolve, why didn’t it keep doing so?
It is well-known that psi seems to be react in agentic ways to attempts to experiment on it. This is really weird and seems to indicate flaws in the research rather than that psi is actually an agentic process that is trying to stay on the verge of detectability.
Well, it seems that way to pro-psi researchers, but it’s not so different from research into unrelated bogus phenomena where researchers are motivated to keep trying different angles of attack, e.g. cross-country international development models in economics.
I would want to see an example of a good argument for quantum computation-psi connections.
Sorry, I didn’t mean to imply that that part is plausible as I haven’t looked into it at all. But the “too hot and noisy” objection seems less certain now that there’s research into quantum “computation” allowing for photosynthesis as well as something else quantum involved in birds’ navigation. Links here. As you suggest, perhaps it should be presumed irrelevant until some plausible mechanism connecting it to psi is proposed.
Actually, all sorts of nifty research on falling in love has been performed with useful predictions about oxytocin levels, fidelity, various sorts of future self-report, divorce, frequency of sex, etc, etc.
Yeah, um, basic sanity fail on my part. Not a good sign.
Re common false beliefs, you obviously know significantly more about the subject than I do. Are there any books you’d recommend?
Claims that psi is small seem to be a “God of the Gaps” side-effect of modern recording technologies and other developments, rewriting the prior ex post.
Point conceded. Until I look at the literature more closely I’ll agree with you that the lack of large effects seems is a decent chunk of evidence against psi.
It would seem that the only psi-friendly explanation of the weak results from parapsychology then would be something like actively evasive psi. You say similar hypotheses have shown up to explain the lack of certain expected phenomena in macroeconomics? Or, might you explain the connection?
By the way thank you for talking to me about this stuff, very few people have the prerequisite knowledge/skills/patience to do so fruitfully.
I’m starting to think that maybe all evidence I have for psi is incommunicable. Bleh!
Are there any books you’d recommend?
Bryan Caplan’s references section in his Myth of the Rational Voter book has cites for many good sources.
It would seem that the only psi-friendly explanation of the weak results from parapsychology then would be something like actively evasive psi. You say similar hypotheses have shown up to explain the lack of certain expected phenomena in macroeconomics? Or, might you explain the connection?
I was referring to data-mining for correlations to produce predictive rules using historical datasets of country growth rates, rules which then fail badly when applied to new datasets. Beliefs in stronger causal conclusions persist, and folk with such beliefs talk about how the processes of growth are changing over time, or the great difficulty of pulling conclusions from the noisy data. The closest think to “actively evasive psi” would probably be academic claims to have found predictive rules for stock and other liquid markets: “my data-mined rule was for real, but now that I’ve published the markets are taking it into account, which is why it no longer works.”
I’m starting to think that maybe all evidence I have for psi is incommunicable. Bleh!
The evidential part of intuitions or personal psi experiences are communicable (save for the possibility of conscious lies, but it’s pretty clear that those are not needed), a bigger dataset of intuitions is better than a single case, etc. Robin’s common priors paper is relevant here.
But the “too hot and noisy” objection seems less certain now that there’s research into quantum “computation” allowing for photosynthesis as well as something else quantum involved in birds’ navigation.
It’s a bit of a tangent, but I thought the recent stuff about some special “quantum” coherence enabling photosynthesis was mostly hype, as it could happen just the same classically.
Most of the effects claimed don’t exist (far more for repeatedly replicated experiments done by independent groups), and the effects that do exist are mostly greatly exaggerated in their effect sizes and generalizability. That is also true of heuristics and biases.
Do you mean this in the trivial sense, or are you making a stronger claim against the heuristics and biases literature?
This sense. I don’t think the heuristics and biases literature is among the worst psychological subfields in terms of quality. I don’t think it is the best.
I think your belief in psi is probably the result of a bias in favor of such hypotheses (partly directly, and partly affecting your enthusiasm for other beliefs that feed into psi)
Might you characterize the bias?
(When I introspect on my rationale for taking these things seriously, it seems it’s because I think it’s best to assume the worst. I.e. psi and UFOs would mean the universe is significantly weirder than I’m comfortable with and I’m not at all sure how that would fit in with plans for reducing x-risk or more generally doing the right thing.
Secondary and tertiary reasons (not the result of introspection but of guessing) would be wanting the universe to be really weird and interesting, and wanting to be maximally contrarian. I’m not sure the secondary reason is true but the third likely is.)
I can’t really explain the apparent lack except by postulating that casual readers haven’t looked very closely at the parapsychology literature. Many have noted that parapsychology experiments are significantly more rigorous than other experiments in psychology because the standards are so high. Note that parapsychologists go out of their way to set up environments where you’d least expect psi, ’cuz those are the only environments that don’t have a huge amount of confounding factors. Even under such austere conditions there are many rigorous experiments confirming psi. Here is a source of some clearly rigorous studies.
Note that there is a huge amount of motivated skepticism in the parapsychology literature; many papers attack other papers merely because it is incredibly easy to gain status by attacking low status ideas. Many complaints about statistical deficiencies are in fact retarded. E.g. in many cases there will be complaints about the file drawer effect even in scenarios where such complaints are patently ridiculous.
In the end it seems to me that one can either tentatively accept the results of parapsychology or conclude that statistics simply does not work. I think the latter hypothesis is rather plausible, but anyone who goes down that route should know that the heuristics and biases and social psychology literature is chock full of results that are significantly less well pinned down than those of parapsychology. The social sciences rarely tell us anything useful with any confidence.
Also note that there are legitimate problems with the results from parapsychology and one should look out for attempts to conceal these problems. The results clearly indicate that psi is capricious. If you wish to skip ahead to the biggest problems with the results, you can find a summary and some proposed explanations here. Because capriciousness and experimenter effects look so shifty some parapsychologists have tried to hide them or minimize their importance, further damaging the reputation of parapsychology. Pay careful attention to make sure that the experimenter isn’t trying to sweep something under the rug.
It perhaps might be best to focus on a single experimenter. Then you can be more confident that there isn’t substantial fraud going on or huge file drawer effects. Looking at the PEAR results exclusively might be a good idea so as to indicate what you can expect from the rest of the field.
Note that I do not take the evidence from parapsychology as strongly suggesting for or against psi. I haven’t yet looked at it very carefully and am skeptical of the benefit of continued searching considering I’m not willing to do a detailed analysis of the statistics used in every paper. If I did not have evidence outside of the parapsychology literature then I would not be as confident of the existence of psi. The parapsychology literature provides merely one kind of evidence, namely statistical evidence, and alone isn’t enough to suggest clear conclusions.
I assign very low probability to psi for several reasons. The first is evolution. If psi is an evolved feature of human brains, then it should be universal in humans and provide fitness advantages, but I’ve seen no evidence of these fitness advantages.
The second is that it is convenient. Humans like fantasizing about having superpowers. That is by far a simpler explanation for widespread belief and academic study.
The third is the economic argument, as explained by Randall Munroe here.
The fourth you’ve already mentioned; psi is an almost textbook example of the mind-projection fallacy.
None of these are knock-down arguments, but together they convince me that psi is not worth looking into unless something changes (a big result published in a mainstream journal that replicates, e.g.).
I couldn’t immediately find it when I searched a few minutes ago. I remember reading that it had failed to replicate like twice out of like seven and succeeded the other five times. (I remember having read your article and thus was surprised to see it had in fact replicated. This was before I took psi at all seriously.) I presume that someone wouldn’t straight-up lie about how many times it’d been replicated. I think I was reading a reputable source but am not super-confident of that. Help, anyone? If not I’ll try searching again soon.
I updated my post on this with numbers and links for several more of the attempted replications.
Looking back at this exchange, I want to note that you were able to invert the public empirical data on this question, along with the several other errors elsewhere in this thread, despite (or because of) the overconfidence of your initial claims. Similar things have happened when you have stuck your neck out on decision theory for attentive experts to chop off. You should generalize and a) put more effort into disconfirming your ideas; b) reduce your confidence in seemingly crazy contrarian views backed by vague impressions (lacking good metadata, for instance) from wide-ranging reading.
Something interesting happened here; for a few months whenever I doubted my memory it always ended up that my memory was correct and my doubt was needless, so eventually I decided to stop doubting my memory as much and trust it more. As soon as I started doing that is when I inverted the replication results, which implies that the doubt itself was keeping my memory honest. I’m not sure if that should have been the obvious model beforehand.
Also our discussion caused me to update somewhat away from thinking that parapsychology is positive rather than neutral or negative evidence for psi; that wasn’t a belief I held strongly in the first place. I think it’s unfortunate that the focus was on Bem rather than on, say, PEAR, and would like to discuss the PEAR studies specifically at some point, those being the studies that I am most familiar with. Anyway thanks for putting so much effort into looking into the question; I think it’d be cool if you made a post specifically about lessons learned from psi and how they apply to other fields, especially the heuristics and biases parts of social psychology. The last paragraph of your most recent post was I think the most important.
lessons learned from psi and how they apply to other fields
One lesson is that it’s possible to waste the valuable time and money of many people by not checking claims before throwing them out. Bogus papers and other nonsense can create negative externalities quite a bit larger than their cost of production. Applying local checks (confirmatory experiments, active search for disconfirmation) is worth doing before fouling the common pool.
If you want to talk about PEAR you should present your arguments and references, and make a prediction about how much of the important stuff (as judged later) you have found. I don’t want to play whack-a-mole.
If you want to talk about PEAR you should present your arguments and references, and make a prediction about how much of the important stuff (as judged later) you have found. I don’t want to play whack-a-mole.
I get the impression that the opportunity cost of your time is high and that I could never be confident enough that my presentation of the arguments was at a sufficiently high level that it’d be worth taking the risk of imposing even a minor moral obligation on you to respond, so that’ll probably never happen.
Your continued posting on this is more trouble to me than efficiently responding. A few quick points:
The PEAR people are now selling supposed psi-controlled-meditation-lamps for $189 each as well as DVDs and other junk; the PEAR research was donation supported, and positive results meant more donations
In Damien Broderick’s pro-psi book (in agreement with the PEAR docs) he notes that “PK” effects show up in the literature even if they’re already set in advance (e.g. digits of pi). Broderick’s account is that the psi reaches across time and space; bad statistics are time-symmetrical, PK takes a big probability penalty (even aside from time, the people don’t get to see or be present at the setting of the numbers.
PK experiments “worked” with macroscopic objects like dice, and were said to so work by psi-proponents like Radin (they also have positive meta-analyses, declining with better controls, dealing with cheating and misrecording, etc), but can’t be delivered for tasks like moving very light (stationary) objects, affecting ultra-sensitive scales, etc; bad statistics work regardless of scale, but for psi that’s a wacky combo
Failure of replication
Failure of registration, lack of blinding, specified confirmatory studies with large fixed samples (this could have been done by using a random number generator in the hands of a third party, with electronic communication, leaving an unambigous trail)
Effect sizes so small (a few per ten thousand according to PEAR) that the combination of biased errors in data entry (found in audits of other studies), tiny amounts of fraud (there were many staff over time), some publication bias, etc, could easily generate the results
Broken up into many individualized experiments: combined with optional stopping and other effects, enabling concentration of “hits” in the published component (larger studies, smaller effects)
Multiple long-term and rotating employees with the opportunity for some fraud, it’s not a question of one person
Your continued posting on this is more trouble to me than efficiently responding.
Aight, then I won’t post about parapsychology.
Thanks for the quick points, I disagree on a few points but I think it’s essentially certain that you’re taking into account the significance of failure of registration and replication in ways that I don’t have enough knowledge to have done, which almost certainly overrides any superior knowledge I might have on the points where I disagree.
Also I really would like an example of a case where I stuck my head out about decision theory and it was chopped off; I think there’s a serious risk that you’re overgeneralizing, especially as I never had much confidence in (my appraisal of) the worth of the parapsychology literature in the first place.
ETA: My interest in parapsychology was explicitly the result of rationalization; I started out by thinking that psi was real, then looked at the literature to see which parts seemed like legitimate support of that known fact. Unsurprisingly the rationalized findings weren’t as good as they seemed. This style of model-building has very little to do with the style of model-building I use when actually thinking, e.g. thinking about decision theory or moral philosophy generally.
Similar things have happened when you have stuck your neck out on decision theory for attentive experts to chop off.
Huh? Link/example?
Looking back at this exchange, I want to note that you were able to invert the public empirical data on this question, along with the several other errors elsewhere in this thread, despite (or because of) the overconfidence of your initial claims.
Inverting the replication results is the only error I see; I admitted to another error that I don’t actually think was an error as such (I think I was trying too hastily to appear reasonable and conciliatory) and intend to go back and explain why I disagree. I made a bad analogy but that doesn’t defeat the point I was trying to argue with the analogy.
Note that being flustered and making abnormal errors while arguing about parapsychology is exactly what my model of evasive psi predicts.
Try Google Scholar, and papers citing Bem. In my article I mentioned and linked one apparent replication that turned out not to be one. I described the Wiseman paper’s saga. Here’s something claiming to be another failure to replicate, conducted at a UFOlogy event, and citing another failed replication (although maybe not a closely-matched one).
If you were hanging out at psychic blogs they might have been stretching the definition of replication to mean anything supporting time-traveling ESP (regardless of whether it uses the same procedures, gets the same results, was written after Bem, or was pre-registered), or they might have access to work that has not yet been put up on the internet in working paper or publication form.
ETA: The wikipedia article on Daryl Bem says that at least one of the Wiseman-registered studies has now replicated something from Bem, but doesn’t identify the study or provide a source for the claim.
Yeah, if a majority of sanely-conducted replications of that study succeed, I’ll reassess and take psi much more seriously. But so far the Bem saga is almost exactly what I would expect to see, given psi being fake.
Thanks, I’d appreciate that. I’d also love to hear about other kinds of evidence that are approximately as convincing as replicated big journal studies. I don’t expect that evidence to exist or else I’d have updated already, but if it does that information is valuable.
See Wagenmaker’s et al “Why psychologists must change the way they analyze data: the case of psy” for why the Bem results are simply artifacts of bad method and statistics. Pdf here.
The methods that Bem uses in his experimentation itself has been viewed as controversial as well. According to understood statistical methodology, Bem incorrectly provides one-sided p values when he should have used a two-sided p values.[17] This could possibly account for the marginally significant results that he produced in his experiment. A rebuttal to the Wagenmakers et al. critique by Bem and two statisticians was subsequently published in the Journal of Personality and Social Psychology.[18]
When I have seen back-and-forths like this it’s always been the pro-psi parapsychologists who understood the statistics better. Do you or does anyone know if that isn’t true in this case?
One common error made by skeptics is to say that the low prior on psi means that after a Bayesian correction any individual experiment or paper isn’t enough to drive belief in psi, so it is “not scientific evidence.” That’s overstatement: if psi were real one would combine the odds ratio of multiple experiments (insofar as they were honest and independent) and overcome that, so the individual pieces would have to be published to accumulate that evidence. It’s partly driven by scientific etiquette: the reason one can’t aggregate studies like that is because there are systematic errors, bias, and fraud. Given the sheer extent of those observed in the record, it’s very hard for a set of experiments to provide decisive evidence without some extraordinary evidence supporting their quality and honesty. Jaynes has an impressively thorough discussion of these issues in his probability textbook. The linked paper critiquing Bem didn’t make that error.
Looking at the exchange you mention here are links to the continuation:
Part of the argument was over whether to expect effect sizes to be miniscule if psi exists (Bem argues that existing research has already disconfirmed big psi effects, so that the penalty for that should be incorporated into our beliefs prior to his experiments, rather than the odds ratio stemming from the experiments. The rest was over whether Bem engaged in data-mining. Bem denies it, but has also written guides to students advocating intensive data-mining, and there are various suspicious elements in the paper that suggest it.
Both sides here seem to understand the statistics under discussion well enough, the back-and-forth is about psi and Bem’s methods or honesty, i.e. flaws in the experimental design, data mining, deception, or luck/file drawer/publication bias effects. Failures to replicate will indicate one or more of those (replications will have to be tested for systematic flaws in the replication package, of course).
Can you link to a replication? I know of several failed replications that aren’t well known thanks to publication bias, but I’m unfamiliar with any successful ones.
Yeah, but value of information calculations have two terms. Even if changing our minds on psi provides huge value (and I agree it would), we still have to consider the probability that investigating psi will change our minds (miniscule). Bring in the opportunity cost of such an investigation and it’s a definite waste of time given a skeptical prior.
Why is that probability a priori miniscule? Something like 50% of people in the US believe in some form of psi and there’s an entire academic discipline devoted to researching it, many of the scientists in which are reputable and work at prestigious universities (and many have a background in physics). No psi effects that I know of are theoretically ruled out given our uncertainty about QM (even the retrocausal stuff fits in with many interpretations, e.g. the transactional interpretation) and even QM is incomplete. There is a history of unseen forces turning out to have huge harness-able effects, like nuclear power or electricity. There is also a general perception that the more science we do the less agency we see in the world, which is some evidence against psi. Overall I think the prior should be low but not miniscule, and that since the value of information is so high because psi is a phenomenon which if it exists would falsify many ingrained inductive biases, it would be very wise to look into the only phenomenon that could at least in principle falsify those biases.
(I might be rationalizing my conclusion somewhat here, as I do in fact know that psi probably exists. Before carefully looking into the issue I had basically the same beliefs as you do. Note that I find the probable existence of psi really annoying and didn’t start looking into the issue because I wanted to believe. Practical epistemology would be much easier in a psi-less world,)
What is the evidence that convinced you?
Questions like that can’t be answered due to problems of social epistemology. If I give you two pieces of evidence then it’s assumed I’ve given you my two most convincing pieces of evidence and that I don’t have any other pieces of evidence, and thus that you should update even further away from the conclusion that I’ve reached. If I regularly talked to you and could keep up a thread of conversation, and if both of us didn’t anticipate needless antagonism involving the presumption that the other hadn’t already considered things that they’d in fact considered many times (like “psi appears to be quite capricious, how convenient for believers”), then I could talk to you about the evidence without fear of unjustifiably cementing either of our beliefs. As it is you can only have that kind of relationship among decently close friends. Do you know of other ways of discussing tricky topics, ways that don’t have those requirements?
I assume you know most of the standard considerations and evidence. I think your belief in psi is probably the result of a bias in favor of such hypotheses (partly directly, and partly affecting your enthusiasm for other beliefs that feed into psi) but that you might have found something interesting while spelunking (which need not be convincing to be interesting).
It would be easy to specify a bunch of stylized facts, and then incrementally add to the list or contest various elements. For instance, here’s a start:
A large proportion of the general population and academics report belief in psychic powers, although not as large as for religion, and various other falsified beliefs
Belief in psychic powers and psychic experiences track general gullibility (susceptibility to various sorts of fraud, witchcraft, jinn, etc, etc), lack of education, etc *Selective memory, unconscious processing, lack of statistical thinking, wishful thinking, etc, explain at least most psychic reports; we would expect high belief in psychic powers even in their absence
If we start off with a wide prior for psi effect sizes, the puny-but-just-barely-almost-detectable range is quite small; since this is predicted by “no psi,” “no psi detectable by current methods,” and “just barely detectable psi” but the latter has much lower prior, we get a big update against psi reports being real
Parapsychologists are often able to produce small positive effects in noisy statistics-based experiments
Big effects are often revealed to be the result of cheating, or the work of labs that have cheated
Parapsychology suffers from anti-epistemology (even more than the already atrocious regular psychology), e.g. tweaking experiments until effects are found, retrospective data-dredging, etc; measure of publication bias, “decline effect” and so forth are quite bad
Not much mainstream effort goes into trying to replicate parapsychology results these days: there are few rewards for doing so, and the parapsychologists use idiosyncratic methods (the Daryl Bem article was unusual in being easier to replicate); those who bother are often strongly motivated skeptics frustrated with the widespread fraud and bad practices
We should expect bad science in parapsychology, since it’s so common elsewhere, there could be a psi effect far punier than even the parapsychologists claim the para
These effects are not reliably repeatable by others, particularly
New forces have been discovered in the past, as well as biological exploitation of their capabilities (magnetic navigation, electric eels, etc)
On the scales of biological organisms (as opposed to quantum gravity issues related to black holes, the Big Bang, etc) the Standard model does pretty well
There are proposed quantum mechanics interpretations with nonlocal effects; these interpretations still involve quantum mechanics working normally and making its predictions correctly, which leaves little room for psi powers to act as a God of the Gaps
The Simulation Hypothesis and allied claims allow for psi powers, or for Keanu Reeves to start flying around in the air, or for Thor and Zeus to appear on the 6 o’clock news; this depends on further claims about the frequency or measure of simulations and the relative abundance of worlds with psi powers that look bogus (as opposed to visible comic book style powers or the absence of psi); given other ways in which our world does not look like a fantasy story, the vastness of fictional genres, and non-fiction purposes for creating simulations, the Matrix-psi story is pretty low-prior
The evolutionary and anatomical picture for psi powers adds additional burden
That’s a really good list!
I tentatively disagree that religion count as a falsified belief. This sounds nit-picky but I think indicates a substantial meta-level difference in our prior, i.e. the extent to which we think the average person is or isn’t crazy. E.g.:
I don’t know whether to agree or disagree with this; I don’t really have much evidence either way. What do you have in mind? It’s clearly crucial, as I don’t see the parapsychology literature as providing very much evidence for psi, and you seem(?) to see it as providing some amount of evidence against psi. I looked through much of the heuristics and biases literature and was profoundly disappointed with what I found. Since then I’ve been skeptical of claims that the average person is highly susceptible to false beliefs. Why do you think people are so prone to “selective memory, unconscious processing, lack of statistical thinking, wishful thinking, etc”, beyond the psychology literature? Or am I wrong to be so unimpressed with the psychology literature?
Recently, quantum stuff in biological systems generally. Only important to note because various proposed psi mechanisms involve quantum computation going on in the brain, which until recently seemed rather implausible but is now a serious possibility.
Given that the methods used are practically designed so as to minimize any psi phenomena, e.g. taking away any real incentives for humans to use psi (with a few exceptions like a few of Bem’s experiments), why should we have a wide prior for psi effect sizes? E.g. Ben Goertzel’s analogy to experimentation on the alleged existence of “falling in love” comes to mind. I can’t find it but it was on his “Multiverse According to Ben” blog. Might you explain more why you think a wide prior seems good?
Agreed. (Only noting this ’cuz I think you might think I think that simulation-style explanations are worth thinking about.)
I don’t follow, might you flesh this out more?
Some other points:
There is a well-known experimenter effect where some experiments seem to get highly different results depending on which scientist is conducting them; a priori this would seem to be evidence of non-psi experimental manipulation, whether conscious or not.
It is well-known that psi seems to be react in agentic ways to attempts to experiment on it. This is really weird and seems to indicate flaws in the research rather than that psi is actually an agentic process that is trying to stay on the verge of detectability.
The religions are contradictory. The behavior of the founders (moreso the more recent the religion) maps onto random guys creating cults today in great detail. People express belief in their particular religion being true, the particular founder or books being right, in young earth creationism, in Islamic medicine, in miraculous healing, etc. They do so at least as fervently as they do belief in psychic powers. You can try to make up a naturalistic version of religion that, while improbable on its face, doesn’t get much of a further penalty from the evidence (simulator deism, whatever) but that doesn’t rescue the accuracy of the masses.
On direct causes, consider sleep paralysis and alien abductions. Drug use can create experiences interpreted as religious or psychic or whatever, and various biological conditions can do likewise. False memories can be cultivated by therapists or group efforts (there was an interesting book by a Harvard researcher studying people who claimed to have been abducted). The formation of urban legends by the telephone effect and optimization for good stories is well-documented. Exposed frauds are pretty common, and frauds have to be more common than exposed ones, probably far more given the scarcity of debunking resources. These things vary from culture to culture, e.g. fan death giving an indication of how much people can shape their experiences and beliefs to fit memes floating around.
Psychology research tends to be bad. Most of the effects claimed don’t exist (far more for repeatedly replicated experiments done by independent groups), and the effects that do exist are mostly greatly exaggerated in their effect sizes and generalizability. That is also true of heuristics and biases. And yet surveys (the basis of these “lots of people believe in ESP” claims) reveal that people have lots of demonstrably false beliefs, often strong beliefs, especially on topics where they get little feedback. Bryan Caplan has many such: basic facts about the economy, demographics, toxicology, risks of death. People believe having sex with virgins cures HIV, despite the absence of any actual documented examples. They form superstitions about homeopathic placebos and acupuncture.
People do not get feedback about their capricious/subtle psi beliefs (healing that heals amputees does get enough negative feedback that it’s a rare belief), and can get utility out of believing they have psi or that the world is magical, indeed will often go out of their way to protect the belief.
I contest. Biological systems are hot and noisy, and quantum effects are at the wrong scales in nervous systems. Second, quantum computation!= telepathy, ESP, telekinesis, etc. Usually these arguments are of the form “QM and ESP are mysterious, and could be related” or “this interpretation of QM assumes nonlocal effects, or even nonlocal effects driven by magic consciousness, so there’s a proof of concept for nonlocal ESP”. These are nonsense. I would want to see an example of a good argument for quantum computation-psi connections.
Claims of psi powers tend to expand to fill the gaps of observation and alternative explanation. E.g. flying witches in Africa, jinn turning people into animals, etc. Claims that psi is small seem to be a “God of the Gaps” side-effect of modern recording technologies and other developments, rewriting the prior ex post. Repelling bullets, visible healing, visible telekinesis, and similar dramatic powers have often been claimed as effects “in the wild” but they always retreat from sight. We have casinos, the stock market, and similar opportunities for precognition, telekinesis could show up in sporting events, etc, etc.
Actually, all sorts of nifty research on falling in love has been performed with useful predictions about oxytocin levels, fidelity, various sorts of future self-report, divorce, frequency of sex, etc, etc. Experiments in the lab can do quite a lot to distinguish couples in love or not, and to increase or decrease it. That’s more of a rhetorical trick, insofar as we use “falling in love” in a lot of signalling games and to some extent we’re unwilling to identify it with any natural kinds that actually exist.
Where is the animal psi? What organs do it? How do the inputs affect human responses and behavior? How did the capacity evolve in the first place, and if it did evolve, why didn’t it keep doing so?
Well, it seems that way to pro-psi researchers, but it’s not so different from research into unrelated bogus phenomena where researchers are motivated to keep trying different angles of attack, e.g. cross-country international development models in economics.
Sorry, I didn’t mean to imply that that part is plausible as I haven’t looked into it at all. But the “too hot and noisy” objection seems less certain now that there’s research into quantum “computation” allowing for photosynthesis as well as something else quantum involved in birds’ navigation. Links here. As you suggest, perhaps it should be presumed irrelevant until some plausible mechanism connecting it to psi is proposed.
Yeah, um, basic sanity fail on my part. Not a good sign.
Re common false beliefs, you obviously know significantly more about the subject than I do. Are there any books you’d recommend?
Point conceded. Until I look at the literature more closely I’ll agree with you that the lack of large effects seems is a decent chunk of evidence against psi.
It would seem that the only psi-friendly explanation of the weak results from parapsychology then would be something like actively evasive psi. You say similar hypotheses have shown up to explain the lack of certain expected phenomena in macroeconomics? Or, might you explain the connection?
By the way thank you for talking to me about this stuff, very few people have the prerequisite knowledge/skills/patience to do so fruitfully.
I’m starting to think that maybe all evidence I have for psi is incommunicable. Bleh!
I was referring to data-mining for correlations to produce predictive rules using historical datasets of country growth rates, rules which then fail badly when applied to new datasets. Beliefs in stronger causal conclusions persist, and folk with such beliefs talk about how the processes of growth are changing over time, or the great difficulty of pulling conclusions from the noisy data. The closest think to “actively evasive psi” would probably be academic claims to have found predictive rules for stock and other liquid markets: “my data-mined rule was for real, but now that I’ve published the markets are taking it into account, which is why it no longer works.”
The evidential part of intuitions or personal psi experiences are communicable (save for the possibility of conscious lies, but it’s pretty clear that those are not needed), a bigger dataset of intuitions is better than a single case, etc. Robin’s common priors paper is relevant here.
It’s a bit of a tangent, but I thought the recent stuff about some special “quantum” coherence enabling photosynthesis was mostly hype, as it could happen just the same classically.
Do you mean this in the trivial sense, or are you making a stronger claim against the heuristics and biases literature?
This sense. I don’t think the heuristics and biases literature is among the worst psychological subfields in terms of quality. I don’t think it is the best.
Might you characterize the bias?
(When I introspect on my rationale for taking these things seriously, it seems it’s because I think it’s best to assume the worst. I.e. psi and UFOs would mean the universe is significantly weirder than I’m comfortable with and I’m not at all sure how that would fit in with plans for reducing x-risk or more generally doing the right thing.
Secondary and tertiary reasons (not the result of introspection but of guessing) would be wanting the universe to be really weird and interesting, and wanting to be maximally contrarian. I’m not sure the secondary reason is true but the third likely is.)
Best not to get into it, at least on a public forum.
I didn’t understand a few of these. I’ll think about them more to see if I can extract their meaning and if not I’ll ask for clarification later.
This is the Goertzel piece.
Can you at least tell us how your hypothesis explains the apparent lack of statistically rigorous experiments confirming psi?
I can’t really explain the apparent lack except by postulating that casual readers haven’t looked very closely at the parapsychology literature. Many have noted that parapsychology experiments are significantly more rigorous than other experiments in psychology because the standards are so high. Note that parapsychologists go out of their way to set up environments where you’d least expect psi, ’cuz those are the only environments that don’t have a huge amount of confounding factors. Even under such austere conditions there are many rigorous experiments confirming psi. Here is a source of some clearly rigorous studies.
Note that there is a huge amount of motivated skepticism in the parapsychology literature; many papers attack other papers merely because it is incredibly easy to gain status by attacking low status ideas. Many complaints about statistical deficiencies are in fact retarded. E.g. in many cases there will be complaints about the file drawer effect even in scenarios where such complaints are patently ridiculous.
In the end it seems to me that one can either tentatively accept the results of parapsychology or conclude that statistics simply does not work. I think the latter hypothesis is rather plausible, but anyone who goes down that route should know that the heuristics and biases and social psychology literature is chock full of results that are significantly less well pinned down than those of parapsychology. The social sciences rarely tell us anything useful with any confidence.
Also note that there are legitimate problems with the results from parapsychology and one should look out for attempts to conceal these problems. The results clearly indicate that psi is capricious. If you wish to skip ahead to the biggest problems with the results, you can find a summary and some proposed explanations here. Because capriciousness and experimenter effects look so shifty some parapsychologists have tried to hide them or minimize their importance, further damaging the reputation of parapsychology. Pay careful attention to make sure that the experimenter isn’t trying to sweep something under the rug.
It perhaps might be best to focus on a single experimenter. Then you can be more confident that there isn’t substantial fraud going on or huge file drawer effects. Looking at the PEAR results exclusively might be a good idea so as to indicate what you can expect from the rest of the field.
Note that I do not take the evidence from parapsychology as strongly suggesting for or against psi. I haven’t yet looked at it very carefully and am skeptical of the benefit of continued searching considering I’m not willing to do a detailed analysis of the statistics used in every paper. If I did not have evidence outside of the parapsychology literature then I would not be as confident of the existence of psi. The parapsychology literature provides merely one kind of evidence, namely statistical evidence, and alone isn’t enough to suggest clear conclusions.
I assign very low probability to psi for several reasons. The first is evolution. If psi is an evolved feature of human brains, then it should be universal in humans and provide fitness advantages, but I’ve seen no evidence of these fitness advantages.
The second is that it is convenient. Humans like fantasizing about having superpowers. That is by far a simpler explanation for widespread belief and academic study.
The third is the economic argument, as explained by Randall Munroe here.
The fourth you’ve already mentioned; psi is an almost textbook example of the mind-projection fallacy.
None of these are knock-down arguments, but together they convince me that psi is not worth looking into unless something changes (a big result published in a mainstream journal that replicates, e.g.).
A big result was very recently published in a mainstream journal and has been replicated. Now what?
ETA: See Carl’s correction. I can’t immediately find the alleged replications and there have been a few failed replications.
Well, it has failed to replicate a few times. Link to the replication please?
I couldn’t immediately find it when I searched a few minutes ago. I remember reading that it had failed to replicate like twice out of like seven and succeeded the other five times. (I remember having read your article and thus was surprised to see it had in fact replicated. This was before I took psi at all seriously.) I presume that someone wouldn’t straight-up lie about how many times it’d been replicated. I think I was reading a reputable source but am not super-confident of that. Help, anyone? If not I’ll try searching again soon.
I updated my post on this with numbers and links for several more of the attempted replications.
Looking back at this exchange, I want to note that you were able to invert the public empirical data on this question, along with the several other errors elsewhere in this thread, despite (or because of) the overconfidence of your initial claims. Similar things have happened when you have stuck your neck out on decision theory for attentive experts to chop off. You should generalize and a) put more effort into disconfirming your ideas; b) reduce your confidence in seemingly crazy contrarian views backed by vague impressions (lacking good metadata, for instance) from wide-ranging reading.
Something interesting happened here; for a few months whenever I doubted my memory it always ended up that my memory was correct and my doubt was needless, so eventually I decided to stop doubting my memory as much and trust it more. As soon as I started doing that is when I inverted the replication results, which implies that the doubt itself was keeping my memory honest. I’m not sure if that should have been the obvious model beforehand.
Also our discussion caused me to update somewhat away from thinking that parapsychology is positive rather than neutral or negative evidence for psi; that wasn’t a belief I held strongly in the first place. I think it’s unfortunate that the focus was on Bem rather than on, say, PEAR, and would like to discuss the PEAR studies specifically at some point, those being the studies that I am most familiar with. Anyway thanks for putting so much effort into looking into the question; I think it’d be cool if you made a post specifically about lessons learned from psi and how they apply to other fields, especially the heuristics and biases parts of social psychology. The last paragraph of your most recent post was I think the most important.
One lesson is that it’s possible to waste the valuable time and money of many people by not checking claims before throwing them out. Bogus papers and other nonsense can create negative externalities quite a bit larger than their cost of production. Applying local checks (confirmatory experiments, active search for disconfirmation) is worth doing before fouling the common pool.
If you want to talk about PEAR you should present your arguments and references, and make a prediction about how much of the important stuff (as judged later) you have found. I don’t want to play whack-a-mole.
I get the impression that the opportunity cost of your time is high and that I could never be confident enough that my presentation of the arguments was at a sufficiently high level that it’d be worth taking the risk of imposing even a minor moral obligation on you to respond, so that’ll probably never happen.
Your continued posting on this is more trouble to me than efficiently responding. A few quick points:
The PEAR people are now selling supposed psi-controlled-meditation-lamps for $189 each as well as DVDs and other junk; the PEAR research was donation supported, and positive results meant more donations
In Damien Broderick’s pro-psi book (in agreement with the PEAR docs) he notes that “PK” effects show up in the literature even if they’re already set in advance (e.g. digits of pi). Broderick’s account is that the psi reaches across time and space; bad statistics are time-symmetrical, PK takes a big probability penalty (even aside from time, the people don’t get to see or be present at the setting of the numbers.
PK experiments “worked” with macroscopic objects like dice, and were said to so work by psi-proponents like Radin (they also have positive meta-analyses, declining with better controls, dealing with cheating and misrecording, etc), but can’t be delivered for tasks like moving very light (stationary) objects, affecting ultra-sensitive scales, etc; bad statistics work regardless of scale, but for psi that’s a wacky combo
Failure of replication
Failure of registration, lack of blinding, specified confirmatory studies with large fixed samples (this could have been done by using a random number generator in the hands of a third party, with electronic communication, leaving an unambigous trail)
Effect sizes so small (a few per ten thousand according to PEAR) that the combination of biased errors in data entry (found in audits of other studies), tiny amounts of fraud (there were many staff over time), some publication bias, etc, could easily generate the results
Broken up into many individualized experiments: combined with optional stopping and other effects, enabling concentration of “hits” in the published component (larger studies, smaller effects)
Multiple long-term and rotating employees with the opportunity for some fraud, it’s not a question of one person
Aight, then I won’t post about parapsychology.
Thanks for the quick points, I disagree on a few points but I think it’s essentially certain that you’re taking into account the significance of failure of registration and replication in ways that I don’t have enough knowledge to have done, which almost certainly overrides any superior knowledge I might have on the points where I disagree.
Also I really would like an example of a case where I stuck my head out about decision theory and it was chopped off; I think there’s a serious risk that you’re overgeneralizing, especially as I never had much confidence in (my appraisal of) the worth of the parapsychology literature in the first place.
ETA: My interest in parapsychology was explicitly the result of rationalization; I started out by thinking that psi was real, then looked at the literature to see which parts seemed like legitimate support of that known fact. Unsurprisingly the rationalized findings weren’t as good as they seemed. This style of model-building has very little to do with the style of model-building I use when actually thinking, e.g. thinking about decision theory or moral philosophy generally.
Huh? Link/example?
Inverting the replication results is the only error I see; I admitted to another error that I don’t actually think was an error as such (I think I was trying too hastily to appear reasonable and conciliatory) and intend to go back and explain why I disagree. I made a bad analogy but that doesn’t defeat the point I was trying to argue with the analogy.
Note that being flustered and making abnormal errors while arguing about parapsychology is exactly what my model of evasive psi predicts.
This video has Samuel Moulton discussing an unpublished, unregistered failure to replicate Bem’s work.
Try Google Scholar, and papers citing Bem. In my article I mentioned and linked one apparent replication that turned out not to be one. I described the Wiseman paper’s saga. Here’s something claiming to be another failure to replicate, conducted at a UFOlogy event, and citing another failed replication (although maybe not a closely-matched one).
If you were hanging out at psychic blogs they might have been stretching the definition of replication to mean anything supporting time-traveling ESP (regardless of whether it uses the same procedures, gets the same results, was written after Bem, or was pre-registered), or they might have access to work that has not yet been put up on the internet in working paper or publication form.
ETA: The wikipedia article on Daryl Bem says that at least one of the Wiseman-registered studies has now replicated something from Bem, but doesn’t identify the study or provide a source for the claim.
Yeah, if a majority of sanely-conducted replications of that study succeed, I’ll reassess and take psi much more seriously. But so far the Bem saga is almost exactly what I would expect to see, given psi being fake.
Makes sense. If I find any replicated studies in big journals I’ll let you know; I only remembered the Bem one because it was pretty recent.
Thanks, I’d appreciate that. I’d also love to hear about other kinds of evidence that are approximately as convincing as replicated big journal studies. I don’t expect that evidence to exist or else I’d have updated already, but if it does that information is valuable.
See Wagenmaker’s et al “Why psychologists must change the way they analyze data: the case of psy” for why the Bem results are simply artifacts of bad method and statistics. Pdf here.
Wikipedia
When I have seen back-and-forths like this it’s always been the pro-psi parapsychologists who understood the statistics better. Do you or does anyone know if that isn’t true in this case?
One common error made by skeptics is to say that the low prior on psi means that after a Bayesian correction any individual experiment or paper isn’t enough to drive belief in psi, so it is “not scientific evidence.” That’s overstatement: if psi were real one would combine the odds ratio of multiple experiments (insofar as they were honest and independent) and overcome that, so the individual pieces would have to be published to accumulate that evidence. It’s partly driven by scientific etiquette: the reason one can’t aggregate studies like that is because there are systematic errors, bias, and fraud. Given the sheer extent of those observed in the record, it’s very hard for a set of experiments to provide decisive evidence without some extraordinary evidence supporting their quality and honesty. Jaynes has an impressively thorough discussion of these issues in his probability textbook. The linked paper critiquing Bem didn’t make that error.
Looking at the exchange you mention here are links to the continuation:
http://dbem.ws/ResponsetoWagenmakers.pdf http://www.ruudwetzels.com/articles/ClarificationsForBemUttsJohnson.pdf
Part of the argument was over whether to expect effect sizes to be miniscule if psi exists (Bem argues that existing research has already disconfirmed big psi effects, so that the penalty for that should be incorporated into our beliefs prior to his experiments, rather than the odds ratio stemming from the experiments. The rest was over whether Bem engaged in data-mining. Bem denies it, but has also written guides to students advocating intensive data-mining, and there are various suspicious elements in the paper that suggest it.
Both sides here seem to understand the statistics under discussion well enough, the back-and-forth is about psi and Bem’s methods or honesty, i.e. flaws in the experimental design, data mining, deception, or luck/file drawer/publication bias effects. Failures to replicate will indicate one or more of those (replications will have to be tested for systematic flaws in the replication package, of course).
(Will continue the general discussion soon. Just airing out my brain a bit.)
Previously I claimed:
I’m thinking of, say, four or five times when I looked into it. I was wondering, does your experience agree with mine, or disagree?
Also, follow the links to the two blog posts mentioned at the bottom of this.
Can you link to a replication? I know of several failed replications that aren’t well known thanks to publication bias, but I’m unfamiliar with any successful ones.