As far as I can tell, there is a strong trend, at least in the medical sciences towards more rigorous meta-analyses (eg, the Cochrane Reviews) which to a large extent avoid this problem. I’m not sure to what extent meta-analyses of this type are the norm in other sciences.
I think homeopathy is as silly as you do, but there are reasons to look at only the positive results, and the first step to doing it right is acknowledging what you’re doing, so I don’t fault them for the title.
Consider cases where there are a lot of unknowns on how to do it right. Most studies will find no effect, which is very good evidence that the average researcher can’t make it work and you shouldn’t just ask a random researcher to can treat you. However, if you look at the distribution, you might find that there’s a false positive distribution and a second—an interesting distribution of people that it does consistently work for. Then you can go on to study this separate distribution and see what they’re doing right.
Proper quacks even announce that they’re cherry-picking in the title… An Overview of Positive Homeopathy Research (PDF).
As far as I can tell, there is a strong trend, at least in the medical sciences towards more rigorous meta-analyses (eg, the Cochrane Reviews) which to a large extent avoid this problem. I’m not sure to what extent meta-analyses of this type are the norm in other sciences.
I think homeopathy is as silly as you do, but there are reasons to look at only the positive results, and the first step to doing it right is acknowledging what you’re doing, so I don’t fault them for the title.
Consider cases where there are a lot of unknowns on how to do it right. Most studies will find no effect, which is very good evidence that the average researcher can’t make it work and you shouldn’t just ask a random researcher to can treat you. However, if you look at the distribution, you might find that there’s a false positive distribution and a second—an interesting distribution of people that it does consistently work for. Then you can go on to study this separate distribution and see what they’re doing right.