the basic heuristic is to trust what a highly regarded member of this community with Scott Alexander says about it
The even more basic heuristic is to look at the history of psi claims and observe that the overwhelming majority of them have failed to stand up to scrutiny. It might even be true that not a single psi claim has ever stood up to scrutiny (though this depends a bit on what you mean by “scrutiny”.) It is certainly true that no psi phenomenon has ever been reliably reproduced, and that the Randi prize went unclaimed despite there having been over 1000 applicants. One person’s opinion, no matter how well respected, doesn’t make much of a Bayesian dent in that mountain of negative evidence. People can be fooled, and people make mistakes. Even smart people.
It is certainly true that no psi phenomenon has ever been reliably reproduced, and that the Randi prize went unclaimed despite there having been over 1000 applicants.
The linked paper is about a meta-study that described how a specifc psi phenomena was reliably reproduced.
There’s an older meta-review of Ganzfeld experiments that came to a similar result. Ganzfeld experiments have been multiple time reproduced.
The standard of the Randi prize (betting chances of 1:1,000,000) is substantially more tough than the standards of evidence-based medicine (two trials that beat 1:20).
You don’t go around and say that there’s no significant evidence that FDA approved drugs work because they aren’t proven to work for Randi’s standards.
a meta-study that described how a specifc psi phenomena was reliably reproduced.
That’s not what reliable reproduction means. What it means is that you would be willing to place a real-money bet on the outcome of a future experiment.
You don’t go around and say that there’s no significant evidence that FDA approved drugs work
Actually, there’s good reason to believe that the drug testing process has some serious flaws. But even under ideal circumstances, the odds of getting a false positive are 1 in 400. There have been about 1500 drugs approved by the FDA so almost certainly at least 3 or 4 of them actually don’t work. Those are good enough odds for me, particularly when compared to the alternatives.
The standard of the Randi prize (betting chances of 1:1,000,000) is substantially more tough than the standards of evidence-based medicine (two trials that beat 1:20).
Yes, of course. If Randi had used the FDA standard, then with over 1000 applicants, you would expect two or three of them to win at 1:400 purely by chance. I’ll take the bet at 1:400 confidence if you’re willing to pay 1/40th of the prize as an entrance fee, and are willing to do the experiment more than once. In fact, I’ll go you one better: You name any statistical test you think you can beat and I’ll take a bet at 10:1 odds. Heck, make it 2:1. (If it’s good enough for a Vegas casino, it’s good enough for me.) Truthfully, that’s a bet I would be absolutely thrilled to lose.
But my prediction is that you will not accept this offer.
Could you refer me to any authoritive scientific body that defines the test of what scientific results are defined as reliably produced over real-money bets?
I think you can make an argument that the scientific establishment should work that way, but if you consider that to be the sole standard you have to throw out most scientific findings because nobody ever bet on them.
You name any statistical test you think you can beat and I’ll take a bet at 10:1 odds. Heck it 2:1
I think Dean Radin or Bem might be open to do a run at the Ganzfeld experiment provided the amount of money involved justifies the effort or there reputation to be gained by a prominent skeptic holding the other side of the bet.
On the other hand I’m personally not in the business of doing parapsychology experiments.
You are asking me to advance an argument from authority? Seriously?
nobody ever bet on them
That’s not true. You place implicit bets on the reliable reproducibility of scientific results every time you use any piece of modern technology. If you’ve ever flown in an airplane you have actually staked your life on the proposition that scientific results are reliably reproducible.
And, BTW, the reason very few people ever place “real” bets on the reliable reproducibility of scientific results is that there’s a shortage of suckers stupid enough to bet against it.
I’m personally not in the business of doing parapsychology experiments.
Aaaaand… here come the excuses. What a surprise.
I’m not asking you to conduct the experiment, I’m just asking you how much you’d be willing to bet.
You place implicit bets on the reliable reproducibility of scientific results every time you use any piece of modern technology. If you’ve ever flown in an airplane you have actually staked your life on the proposition that scientific results are reliably reproducible.
There a plenty of Yoga people who produce reliable Yoga results by using a theory that assumes the existance of chakra’s. That doesn’t prove the theory of chakra’s to be right. Successful technology is not the same thing as successful science.
Futhermore there’s a lot of science which is not effectively used in technology, do you consider that to be illegimate?
I’m not asking you to conduct the experiment, I’m just asking you how much you’d be willing to bet.
The core idea with betting is to be specific about the terms of the bet. If you wan to bet about whether Bem or Dean Radin produce publishable statistically significant results for paranormal phenomena in the future, then I would be pretty confident that they do.
It’s just that I don’t expect you to take such a bet, because you likely would want some sort of tamperproof enviroment.
That’s where things get complicated and I’m not interested in setting up such an enivroment. People like Bem or Radin on the other hand seem to me open to be engaged in such a way. People who are publically on the record for stating that their reason not to participate in Randi experiment was that their effect sizes where lower than what’s required for Randi.
For my myself the topic isn’t worth the effort of building up experiments and investing hundreds of hours for that.
Apart from that I personally don’t take bets with strangers and if I would than rather about topics where there definite evidence in one direction and not topics that are grey.
Successful technology is not the same thing as successful science.
That’s why I specified modern technology. Computers. Airplanes. Microwave ovens.
Futhermore there’s a lot of science which is not effectively used in technology, do you consider that to be illegimate?
That’s a non-sequitur. Your claim was:
you have to throw out most scientific findings because nobody ever bet on them
In fact, every time you turn on your cell phone you are placing an implicit bet on vast swaths of scientific knowledge. That there might be additional areas of scientific knowledge that you aren’t placing bets on doesn’t change that.
The core idea with betting is to be specific about the terms of the bet.
Here are the terms: choose your favorite psi experiment where the odds of a positive result based on pure chance are objectively computable. Let’s say those odds are N:1. We bet at M*N:1 odds for some multiplier M>=2. If you make M larger than 2 that indicates greater confidence and you get more bragging rights should you win. We iterate until you either concede, or you have won $10,000 from me.
their reason not to participate in Randi experiment was that their effect sizes where lower than what’s required for Randi.
There are well known techniques for extracting signals from noisy channels to any desired level of statistical confidence. Physicist regularly get six-sigma results (odds of resulting by change of less than one in 3 million) from signal-to-noise ratios in the negative tens of dB. This is part of what it means to be “reliably reproducible.”
you likely would want some sort of tamperproof enviroment.
Well, duh. Any intellectually honest seeker of the truth should want some assurances that they are not being deceived.
The even more basic heuristic is to look at the history of psi claims and observe that the overwhelming majority of them have failed to stand up to scrutiny. It might even be true that not a single psi claim has ever stood up to scrutiny (though this depends a bit on what you mean by “scrutiny”.) It is certainly true that no psi phenomenon has ever been reliably reproduced, and that the Randi prize went unclaimed despite there having been over 1000 applicants. One person’s opinion, no matter how well respected, doesn’t make much of a Bayesian dent in that mountain of negative evidence. People can be fooled, and people make mistakes. Even smart people.
I already conceded this point, so you are now attacking a straw man. If you’re not going to cut me any slack over terminology the you should expect me to cut you any when you don’t pay attention. (Though I submit that this conversation will be more fruitful if we both cut each other a little more slack.)
The linked paper is about a meta-study that described how a specifc psi phenomena was reliably reproduced.
There’s an older meta-review of Ganzfeld experiments that came to a similar result. Ganzfeld experiments have been multiple time reproduced.
The standard of the Randi prize (betting chances of 1:1,000,000) is substantially more tough than the standards of evidence-based medicine (two trials that beat 1:20).
You don’t go around and say that there’s no significant evidence that FDA approved drugs work because they aren’t proven to work for Randi’s standards.
That’s not what reliable reproduction means. What it means is that you would be willing to place a real-money bet on the outcome of a future experiment.
Actually, there’s good reason to believe that the drug testing process has some serious flaws. But even under ideal circumstances, the odds of getting a false positive are 1 in 400. There have been about 1500 drugs approved by the FDA so almost certainly at least 3 or 4 of them actually don’t work. Those are good enough odds for me, particularly when compared to the alternatives.
Yes, of course. If Randi had used the FDA standard, then with over 1000 applicants, you would expect two or three of them to win at 1:400 purely by chance. I’ll take the bet at 1:400 confidence if you’re willing to pay 1/40th of the prize as an entrance fee, and are willing to do the experiment more than once. In fact, I’ll go you one better: You name any statistical test you think you can beat and I’ll take a bet at 10:1 odds. Heck, make it 2:1. (If it’s good enough for a Vegas casino, it’s good enough for me.) Truthfully, that’s a bet I would be absolutely thrilled to lose.
But my prediction is that you will not accept this offer.
Could you refer me to any authoritive scientific body that defines the test of what scientific results are defined as reliably produced over real-money bets?
I think you can make an argument that the scientific establishment should work that way, but if you consider that to be the sole standard you have to throw out most scientific findings because nobody ever bet on them.
I think Dean Radin or Bem might be open to do a run at the Ganzfeld experiment provided the amount of money involved justifies the effort or there reputation to be gained by a prominent skeptic holding the other side of the bet.
On the other hand I’m personally not in the business of doing parapsychology experiments.
You are asking me to advance an argument from authority? Seriously?
That’s not true. You place implicit bets on the reliable reproducibility of scientific results every time you use any piece of modern technology. If you’ve ever flown in an airplane you have actually staked your life on the proposition that scientific results are reliably reproducible.
And, BTW, the reason very few people ever place “real” bets on the reliable reproducibility of scientific results is that there’s a shortage of suckers stupid enough to bet against it.
Aaaaand… here come the excuses. What a surprise.
I’m not asking you to conduct the experiment, I’m just asking you how much you’d be willing to bet.
For my part, I’ll put up $10,000.
There a plenty of Yoga people who produce reliable Yoga results by using a theory that assumes the existance of chakra’s. That doesn’t prove the theory of chakra’s to be right. Successful technology is not the same thing as successful science.
Futhermore there’s a lot of science which is not effectively used in technology, do you consider that to be illegimate?
The core idea with betting is to be specific about the terms of the bet. If you wan to bet about whether Bem or Dean Radin produce publishable statistically significant results for paranormal phenomena in the future, then I would be pretty confident that they do.
It’s just that I don’t expect you to take such a bet, because you likely would want some sort of tamperproof enviroment. That’s where things get complicated and I’m not interested in setting up such an enivroment. People like Bem or Radin on the other hand seem to me open to be engaged in such a way. People who are publically on the record for stating that their reason not to participate in Randi experiment was that their effect sizes where lower than what’s required for Randi.
For my myself the topic isn’t worth the effort of building up experiments and investing hundreds of hours for that.
Apart from that I personally don’t take bets with strangers and if I would than rather about topics where there definite evidence in one direction and not topics that are grey.
That’s why I specified modern technology. Computers. Airplanes. Microwave ovens.
That’s a non-sequitur. Your claim was:
In fact, every time you turn on your cell phone you are placing an implicit bet on vast swaths of scientific knowledge. That there might be additional areas of scientific knowledge that you aren’t placing bets on doesn’t change that.
Here are the terms: choose your favorite psi experiment where the odds of a positive result based on pure chance are objectively computable. Let’s say those odds are N:1. We bet at M*N:1 odds for some multiplier M>=2. If you make M larger than 2 that indicates greater confidence and you get more bragging rights should you win. We iterate until you either concede, or you have won $10,000 from me.
There are well known techniques for extracting signals from noisy channels to any desired level of statistical confidence. Physicist regularly get six-sigma results (odds of resulting by change of less than one in 3 million) from signal-to-noise ratios in the negative tens of dB. This is part of what it means to be “reliably reproducible.”
Well, duh. Any intellectually honest seeker of the truth should want some assurances that they are not being deceived.