Following the OP’s sniper example, suppose you take the same pill the next time you have a flare-up, and it helps again. Would you still stick to your prewritten bottom line of “it ought to be something else”?
To better frame my earlier point, suppose that you flare up six times during a season, and the six times it gets better after having applied “X” method which was statistically shown to be no better than placebo, and after rational analysis you find out that each time you applied method X you were also circumstantially applying method Y at the same time (that is, you’re also applying a placebo, since unless in your entire life you have never been relieved of some ailment by taking a pill, your body/brain will remember the principle and synchronize, just like any other form of positive reinforcement training).
In other words, both P(X-works|cured) and P(Y-works|cured) are raised, but only by half, since statistically they’ve been shown to have the same effect, and thus your priors are that they are equally as likely of being the cure-cause, and while both could be a cure-cause, the cure-cause could also be to have applied both of them. Since those two latter possibilities end up evening out, you divide the posteriors in the two, from my understanding. I might be totally off though, since I haven’t been learning about Bayes’ theorem all that long and I’m very much still a novice in bayesian rationality and probability.
To make a long story short, yes, I would stick to it, because within the presented context there are potentially thousands of X, Y and Z possible cure-causes, so while the likeliness that one of them is a cure is going up really fast each time I cure myself under the same circumstances, only careful framing of said circumstances will allow anyone to really rationally establish which factors become more likely to be truly causal and which are circumstantial (or correlated in another, non-directly-causal fashion).
Since homeopathy almost invariably involves hundreds of other factors, many of which are unknown and some of which we might be completely unaware, it becomes extremely difficult to reliably test for its effectiveness in some circumstances. This is why we assign greater trust in the large-scale double-blind studies, because our own analysis is of lower comparative confidence. At least within the context of this particular sniper example.
The homoeopathy claims interaction of water molecules with each other leads to water memory… it can even claim that all the physics is the same (hypothesis size per Solomonoff induction is same) yet the physics works out to the stuff happening. And we haven’t really ran sufficiently detailed simulation to rule it out, just arguments of uncertain validity. There’s no way to estimate it’s probability.
We do something different than estimating probability of homoeopathy being true. It’s actually very beautiful method, very elegant solution. We say, well, let’s strategically take the risk of one-in-a-million that we discard a true curing method homoeopathy with such and such clinical effect. Then we run the clinical trials, and find evidence that there’s less than one-in-a-million chance that those trials happened as they were, if the homoeopathy was true. We still do not know the probability of homoeopathy, but we strategically discard it, and we know the probability of wrongfully having discarded it, we can even find upper bound on how much that strategy lost in terms of expected utility, by discarding. That’s how science works. The cut off strategy can be set from utility maximization considerations (with great care to avoid pascal’s wager).
So, suppose that Sabadil cured your allergies 10 times out of 10, you will not take again unless forced to, because “There’s no way to estimate it’s (sic) probability.”? Maybe you need to reread chapter 1 of HPMOR, and brush up on how to actually change your mind.
If it did cure allergies 10 times out of 10, and that ALL other possible cure-causes had been eliminated as causal beforehand (including the placebo effect which is inherent to most acts of taking a homoeopathic pill, even when the patient doesn’t believe it’ll work, simply out of subconscious memory of being cured by taking a pill), then yes, the posterior belief in its effectiveness would shoot up.
However, “the body curing itself by wanting to and being willing to even try things we know probably won’t work based on what-ifs alone” is itself a major factor, one that has also been documented.
Par contre, if it did work 10 times out of 10, then I almost definitely would take it again, since it has now been shown to be, at worst, statistically correlated with whatever actually does cure me of my symptoms, whether that’s the homoeopathic treatment or not. While doing that, I would keep attempting to rationally identify the proper causal links between events.
The point is that there is a decision method that allows me to decide without anyone having to make a prior.
Say, the cost of trial is a, the cost (utility loss) of missing valid cure to strategy failure is b, you do the N trials , N such that a N < (the probability of trials given assumption of validity of cure) b , then you proclaim cure not working. Then you can do more trials if the cost of trial falls. You don’t know the probability and you still decide in an utility-maximizing manner (on choice of strategy), because you have the estimate on the utility loss that the strategy will incur in general.
edit: clearer. Also I am not claiming it is the best possible method, it isn’t, but it’s a practical solution that works. You can know the probability that you will end up going uncured if the cure actually works.
let’s strategically take the risk of one-in-a-million that we discard a true curing method homoeopathy with such and such clinical effect
Where does your choice of “such and such clinical effect” come from? Keeping your one-in-a-million chance of being wrong fixed, the scale of the clinical trials required depends on the effect size of homeopathy. If homeopathy is a guaranteed cure, it’s enough to dose one incurably sick person. If it helps half of the patients, you might need to dose on the order of twenty. And so on for smaller effect sizes. The homeopathy claim is not just a single hypothesis but a compound hypothesis consisting of all these hypotheses. Choosing which of these hypotheses to entertain is a probabilistic judgment; it can’t be escaped by just picking one of the hypotheses, since that’s just concentrating the prior mass at one point.
(Pardon the goofy notation. Don’t want to deal with the LaTeX engine.)
The compound hypothesis is well-defined. Suppose that the baseline cure probability for a placebo is θ ∈ [0,1). Then hypotheses take the form H ⊂ [0,1], which have the interpretation that the cure rate for homeopathy is in H. The standing null hypothesis in this case is Hθ = { θ }. The alternative hypothesis that homeopathy works is H>θ = (θ,1] = { x : x > θ }. For any θ′ ∈ H>θ, we can construct a “one-in-a-million chance of being wrong” test for the simple hypothesis Hθ′ that homeopathy is effective with effect size exactly θ′. It is convenient that such tests work just as well for the hypothesis H≥θ′. However, we can’t construct a test for H>θ.
Bringing in falsifiability only confuses the issue. No clinical data exist that will strictly falsify any of the hypotheses considered above. On the other hand, rejecting Hθ′ seems like it should provide weak support for rejecting H>θ. My take on this is that since such a research program seems to work in practice, falsifiability doesn’t fully describe how science works in this case (see Popper vs. Kuhn, Lakatos, Feyerabend, etc.).
Clinical data still exists that would allow a strategy to stop doing more tests at specific cut off point as the payoff from the hypothesis being right is dependent to the size of the effect and there will be clinical data at some point where the integral of payoff over lost clinical effects is small enough. It just gets fairly annoying to calculate. . Taking the strategy will be similar to gambling decision.
I do agree that there is a place for occam’s razor here but there exist no formalism that actually lets you quantify this weak support. There’s the Solomonoff induction, which is un-computable and awesome for work like putting an upper bound on how good induction can (or rather, can’t) ever be.
Following the OP’s sniper example, suppose you take the same pill the next time you have a flare-up, and it helps again. Would you still stick to your prewritten bottom line of “it ought to be something else”?
To better frame my earlier point, suppose that you flare up six times during a season, and the six times it gets better after having applied “X” method which was statistically shown to be no better than placebo, and after rational analysis you find out that each time you applied method X you were also circumstantially applying method Y at the same time (that is, you’re also applying a placebo, since unless in your entire life you have never been relieved of some ailment by taking a pill, your body/brain will remember the principle and synchronize, just like any other form of positive reinforcement training).
In other words, both P(X-works|cured) and P(Y-works|cured) are raised, but only by half, since statistically they’ve been shown to have the same effect, and thus your priors are that they are equally as likely of being the cure-cause, and while both could be a cure-cause, the cure-cause could also be to have applied both of them. Since those two latter possibilities end up evening out, you divide the posteriors in the two, from my understanding. I might be totally off though, since I haven’t been learning about Bayes’ theorem all that long and I’m very much still a novice in bayesian rationality and probability.
To make a long story short, yes, I would stick to it, because within the presented context there are potentially thousands of X, Y and Z possible cure-causes, so while the likeliness that one of them is a cure is going up really fast each time I cure myself under the same circumstances, only careful framing of said circumstances will allow anyone to really rationally establish which factors become more likely to be truly causal and which are circumstantial (or correlated in another, non-directly-causal fashion).
Since homeopathy almost invariably involves hundreds of other factors, many of which are unknown and some of which we might be completely unaware, it becomes extremely difficult to reliably test for its effectiveness in some circumstances. This is why we assign greater trust in the large-scale double-blind studies, because our own analysis is of lower comparative confidence. At least within the context of this particular sniper example.
The homoeopathy claims interaction of water molecules with each other leads to water memory… it can even claim that all the physics is the same (hypothesis size per Solomonoff induction is same) yet the physics works out to the stuff happening. And we haven’t really ran sufficiently detailed simulation to rule it out, just arguments of uncertain validity. There’s no way to estimate it’s probability.
We do something different than estimating probability of homoeopathy being true. It’s actually very beautiful method, very elegant solution. We say, well, let’s strategically take the risk of one-in-a-million that we discard a true curing method homoeopathy with such and such clinical effect. Then we run the clinical trials, and find evidence that there’s less than one-in-a-million chance that those trials happened as they were, if the homoeopathy was true. We still do not know the probability of homoeopathy, but we strategically discard it, and we know the probability of wrongfully having discarded it, we can even find upper bound on how much that strategy lost in terms of expected utility, by discarding. That’s how science works. The cut off strategy can be set from utility maximization considerations (with great care to avoid pascal’s wager).
So, suppose that Sabadil cured your allergies 10 times out of 10, you will not take again unless forced to, because “There’s no way to estimate it’s (sic) probability.”? Maybe you need to reread chapter 1 of HPMOR, and brush up on how to actually change your mind.
If it did cure allergies 10 times out of 10, and that ALL other possible cure-causes had been eliminated as causal beforehand (including the placebo effect which is inherent to most acts of taking a homoeopathic pill, even when the patient doesn’t believe it’ll work, simply out of subconscious memory of being cured by taking a pill), then yes, the posterior belief in its effectiveness would shoot up.
However, “the body curing itself by wanting to and being willing to even try things we know probably won’t work based on what-ifs alone” is itself a major factor, one that has also been documented.
Par contre, if it did work 10 times out of 10, then I almost definitely would take it again, since it has now been shown to be, at worst, statistically correlated with whatever actually does cure me of my symptoms, whether that’s the homoeopathic treatment or not. While doing that, I would keep attempting to rationally identify the proper causal links between events.
The point is that there is a decision method that allows me to decide without anyone having to make a prior.
Say, the cost of trial is a, the cost (utility loss) of missing valid cure to strategy failure is b, you do the N trials , N such that a N < (the probability of trials given assumption of validity of cure) b , then you proclaim cure not working. Then you can do more trials if the cost of trial falls. You don’t know the probability and you still decide in an utility-maximizing manner (on choice of strategy), because you have the estimate on the utility loss that the strategy will incur in general.
edit: clearer. Also I am not claiming it is the best possible method, it isn’t, but it’s a practical solution that works. You can know the probability that you will end up going uncured if the cure actually works.
Where does your choice of “such and such clinical effect” come from? Keeping your one-in-a-million chance of being wrong fixed, the scale of the clinical trials required depends on the effect size of homeopathy. If homeopathy is a guaranteed cure, it’s enough to dose one incurably sick person. If it helps half of the patients, you might need to dose on the order of twenty. And so on for smaller effect sizes. The homeopathy claim is not just a single hypothesis but a compound hypothesis consisting of all these hypotheses. Choosing which of these hypotheses to entertain is a probabilistic judgment; it can’t be escaped by just picking one of the hypotheses, since that’s just concentrating the prior mass at one point.
It’s part of the hypothesis, without it the idea is not a defined hypothesis. See falsifiability.
(Pardon the goofy notation. Don’t want to deal with the LaTeX engine.)
The compound hypothesis is well-defined. Suppose that the baseline cure probability for a placebo is θ ∈ [0,1). Then hypotheses take the form H ⊂ [0,1], which have the interpretation that the cure rate for homeopathy is in H. The standing null hypothesis in this case is Hθ = { θ }. The alternative hypothesis that homeopathy works is H>θ = (θ,1] = { x : x > θ }. For any θ′ ∈ H>θ, we can construct a “one-in-a-million chance of being wrong” test for the simple hypothesis Hθ′ that homeopathy is effective with effect size exactly θ′. It is convenient that such tests work just as well for the hypothesis H≥θ′. However, we can’t construct a test for H>θ.
Bringing in falsifiability only confuses the issue. No clinical data exist that will strictly falsify any of the hypotheses considered above. On the other hand, rejecting Hθ′ seems like it should provide weak support for rejecting H>θ. My take on this is that since such a research program seems to work in practice, falsifiability doesn’t fully describe how science works in this case (see Popper vs. Kuhn, Lakatos, Feyerabend, etc.).
Clinical data still exists that would allow a strategy to stop doing more tests at specific cut off point as the payoff from the hypothesis being right is dependent to the size of the effect and there will be clinical data at some point where the integral of payoff over lost clinical effects is small enough. It just gets fairly annoying to calculate. . Taking the strategy will be similar to gambling decision.
I do agree that there is a place for occam’s razor here but there exist no formalism that actually lets you quantify this weak support. There’s the Solomonoff induction, which is un-computable and awesome for work like putting an upper bound on how good induction can (or rather, can’t) ever be.