But but what if you get inconsistent result? Let’s say you try the ritual 5 times and the placebo 5 times and …
Any test with such a small sample size is barely worth the bother of conducting.
Well, it’s standard in medicine to have large RCTs because of various reasons(*), but I’d hardly say “barely worth the bother of conducting”. Every bit of randomized data gives you evidence about cause and effect that, while sometimes weak, does let you update your posterior (a little or a lot) without worrying about the myriad issues of confounding that plague any observational data. Randomization is very useful even in small doses. [though getting consent of the participants is usually hard, even when the preliminary evidence is still very shaky.]
(*) the reasons include the clear ulterior motives of drug companies, the need to consent individuals to randomization combined with delicate arguments around the ethics of “equipoise”, the difficulties of “meta-analysis”, a long history of frequentist statistics, the standards of journals vs. the possibilities of free and open science (based hypothetically on privacy-secure but comprehensively integrated health records), safety issues, etc… But another large reason is that doctors really really like “certainty” and would rather let “best practice” to tell them what to do rather than collect evidence, condition, and decide what’s best for the patient themselves. [some of this seems to be training, but also that they must defend themselves against malpractice. In the end, maybe this isn’t so bad. Thinking is hard and probably all in all it’s better not to trust them to do it most of the time, so I’m not rallying for change in clinical practice here, except to have as much randomization as possible.]
It’s true that you could get evidence from such an experiment which would allow you to update your posterior (although if you’re using significance testing like most experiments, you’re very unlikely to achieve statistical significance, and your experiment almost certainly won’t get published.) But even if you’re doing it purely for your own evidence, the amount of evidence you’d collect is likely to be so small that it hardly justifies the effort of conducting the experiment.
Well, it’s standard in medicine to have large RCTs because of various reasons(*), but I’d hardly say “barely worth the bother of conducting”. Every bit of randomized data gives you evidence about cause and effect that, while sometimes weak, does let you update your posterior (a little or a lot) without worrying about the myriad issues of confounding that plague any observational data. Randomization is very useful even in small doses. [though getting consent of the participants is usually hard, even when the preliminary evidence is still very shaky.]
(*) the reasons include the clear ulterior motives of drug companies, the need to consent individuals to randomization combined with delicate arguments around the ethics of “equipoise”, the difficulties of “meta-analysis”, a long history of frequentist statistics, the standards of journals vs. the possibilities of free and open science (based hypothetically on privacy-secure but comprehensively integrated health records), safety issues, etc… But another large reason is that doctors really really like “certainty” and would rather let “best practice” to tell them what to do rather than collect evidence, condition, and decide what’s best for the patient themselves. [some of this seems to be training, but also that they must defend themselves against malpractice. In the end, maybe this isn’t so bad. Thinking is hard and probably all in all it’s better not to trust them to do it most of the time, so I’m not rallying for change in clinical practice here, except to have as much randomization as possible.]
It’s true that you could get evidence from such an experiment which would allow you to update your posterior (although if you’re using significance testing like most experiments, you’re very unlikely to achieve statistical significance, and your experiment almost certainly won’t get published.) But even if you’re doing it purely for your own evidence, the amount of evidence you’d collect is likely to be so small that it hardly justifies the effort of conducting the experiment.