Publication in general doesn’t have to be rare and special, only the publications of negative results has to be uncommon.
I realize that, and I’ve already pointed out why the difference in rates is not going to be that large & that your cite does not explain the excess significance in their sample.
There’s other issues, e.g. how many of those tests were re-testing simple, effective FDA-approved drugs and such?
Doesn’t matter that much. Power, usually quite low, sets the upper limit to how many of the results should have been positive even if we assume every single one was testing a known-efficacious drug (which hypothesis raises its own problems: how is that consistent with your claims about the language bias towards publishing cool new results?)
Also, for the Soviet union, there would be a certain political advantage in finding no efficacy of drugs that are expensive to manufacture or import.
So? I don’t care why the Russian literature is biased, just that it is.
What’s of the chemical safety studies? There’s a very strong bias to fail to disprove the null hypothesis.
Yes, but toxicology studies being done by industry is not aimed at academic publication, and the ones aimed at academic publication have the usual incentives to find something and so are part of the overall problem.
Yet your paper somehow found a ridiculously high positive rate for acupuncture. The way I think it would work, well, first thing first it’s very difficult to blind acupuncture studies and inadequately blinded experiments should find positive result from the placebo effect,
Huh? The paper finds that acupuncture study rates vary by region. USA/Sweden/Germany 53/59%/63%, China/Japan/Taiwan 100% etc
secondarily, because that’s the case, nobody really cares about that effect, and thirdly, de-facto the system did not result in construction of acupuncture centres.
How much have you looked? There’s plenty of acupuncture centres in the USA despite a relatively low acupuncture success rate.
I haven’t really noticed nootropics being a big thing
Does a fish notice water? But fine, maybe you don’t, feel free to supply your own example of Russian pseudoscience and traditional medicine. I doubt Russian science is a shining jewel of perfection with no faults given its 91% acupuncture success rate (admittedly on a small base).
but somehow they didn’t end up replacing antibiotics with homebrew phage therapy
When antibiotics were discovered in 1941 and marketed widely in the U.S. and Europe, Western scientists mostly lost interest in further use and study of phage therapy for some time.[12] Isolated from Western advances in antibiotic production in the 1940s, Russian scientists continued to develop already successful phage therapy to treat the wounds of soldiers in field hospitals. During World War II, the Soviet Union used bacteriophages to treat many soldiers infected with various bacterial diseases e.g. dysentery and gangrene. Russian researchers continued to develop and to refine their treatments and to publish their research and results. However, due to the scientific barriers of the Cold War, this knowledge was not translated and did not proliferate across the world.
Anyway,
To summarize, I see this allegation of some grave fault but I fail to see the consequences of this fault.
How do you see the unseen? Unless someone has done a large definitive RCT, how does one ever prove that a result was bogus? Nobody is ever going to take the time and resources to refute those shitty animal experiments with a much better experiment. Most scientific findings never gets that sort of black-and-white refutation, it just gets quietly forgotten and buried, and even the specialists don’t know about it. Most bad science doesn’t look like Lysenko. Or look at evidence-based medicine in the West: rubbish medicine doesn’t look like a crazy doc slicing open patients with a scalpel, it just looks like regular old medicine which ‘somehow’ turns up no benefit when rigorously tested and is quietly dropped from the medical textbooks.
To diagnose bad science, you need to look at overall metrics and indirect measures—like excess significance. Like 91% of acupuncture studies working.
Doesn’t matter that much. Power, usually quite low...
If you want to persist in your mythical ideas regarding western civilization by postulating what ever you need and making shit up, there’s nothing I or anyone else can do about it.
So? I don’t care why the Russian literature is biased, just that it is.
Your study is making a more specific claim than mere bias in research, it’s claiming bias in one particular direction.
Not sure that’s a good example, as Wikipedia seems to disagree about homebrew phage therapy not being applied:
The point is that the SU was, mostly, using antibiotics (once production was set up, i.e. from some time after ww2).
There’s plenty of acupuncture centres in the USA despite a relatively low acupuncture success rate.
Well, and there wasn’t a plenty in the soviet union despite supposedly higher success rate.
Huh? The paper finds that acupuncture study rates vary by region. USA/Sweden/Germany 53/59%/63%, China/Japan/Taiwan 100% etc
If you don’t know correct rate you can’t tell which specific rate is erroneous. It’s not realistically possible to construct a blind study of acupuncture, so, unlike, say, homoeopathy, it is a very shitty measure of research errors.
To diagnose bad science, you need to look at overall metrics and indirect measures—like excess significance. Like 91% of acupuncture studies working.
I really doubt that 91% of Russian language acupuncture studies published in Soviet Union found a positive effect (I dunno about 1991-1998 Russia, it was fucked up beyond belief at that time), and I don’t know how many studies should have found a positive effect (followed by a note that more adequate blinding must be invented to study it properly).
And we know that what ever was the case there was no Soviet abandonment of normal medicine in favour of acupuncture—the system somehow worked out ok in the end.
If you want to persist in your mythical ideas regarding western civilization by postulating what ever you need and making shit up, there’s nothing I or anyone else can do about it.
That’s not a reply to what I wrote.
Your study is making a more specific claim than mere bias in research, it’s claiming bias in one particular direction.
Yes, that’s what a bias is. A systematic tendency in one direction. As opposed to random error.
The point is that the SU was, mostly, using antibiotics (once production was set up, i.e. from some time after ww2).
And before that, they were using phages despite apparently pretty shaky evidence it was anything but a placebo. That said, pointing out the systematic bias of Russian science (among many other countries, and I’m fascinated, incidentally, how the only country you’re defending like this is… your own. No love for Korea?) does not commit me to the premise that phages do or not work—you’re the one who brought them up as an example of how excellent Russian science is, not me.
Well, and there wasn’t a plenty in the soviet union despite supposedly higher success rate.
How many are there now? Shouldn’t you have looked that up?
If you don’t know correct rate you can’t tell which specific rate is erroneous.
Difference in rates is prima facie evidence of bias, due to the disagreement. If someone says A and someone else says not-A, you don’t need to know what A actually is to observe the contradiction and know at least one party is wrong.
It’s not realistically possible to construct a blind study of acupuncture
Yes it is.
I really doubt that 91% of Russian language acupuncture studies published in Soviet Union found a positive effect (I dunno about 1991-1998 Russia, it was fucked up beyond belief at that time), and I don’t know how many studies should have found a positive effect (followed by a note that more adequate blinding must be invented to study it properly).
And naturally, you have not looked for anything on the topic, you just doubt it.
And we know that what ever was the case there was no Soviet abandonment of normal medicine in favour of acupuncture—the system somehow worked out ok in the end.
Strawman. No country engages in ‘abandonment of normal medicine’ - if you go to China, do you only find acupuncturists? Of course not. The problem is that you find acupuncturists sucking up resources in dispensing expensive placbeos and you find that the scientific community is not strong enough to resist the cultural & institutional pressures and find that acupuncture doesn’t work, resulting in real working medicine being intermeshed with pseudomedicine.
Fortunately, normal medicine (after tremendous investments in R&D and evidence-based medicine) currently works fairly well and I think it would take a long time for it to decay into something as overall bad as pre-modern Western medicine was; I also think some core concepts like germ theory are sufficiently simple & powerful that they can’t be lost, but that would be cold comfort in the hypothetical cargo cult scenario (‘good news: doctors still know what infections are and how to fight epidemics; bad news: everything else they do is so much witch-doctor mumbojumbo based on unproven new therapies, misinterpretations of old therapies which used to work, and traditional treatments like acupuncture’).
Difference in rates is prima facie evidence of bias, due to the disagreement. If someone says A and someone else says not-A, you don’t need to know what A actually is to observe the contradiction and know at least one party is wrong.
Unless A contains indexicals that point to different things in the two cases.
(Maybe Asian acupunturists are better than European ones, or maybe East Asians respond better to acupuncture than Caucasians for some reason, or...)
( I’m not saying that this is likely, just that it’s possible.)
among many other countries, and I’m fascinated, incidentally, how the only country you’re defending like this is… your own.
That’s the one I know most about, obviously. I have no clue about what’s going on in China, Korea, or Japan.
does not commit me to the premise that phages do or not work
Look, it doesn’t matter if phages work or don’t work! The treatment, in favour of which there would be strong bias, got replaced with another treatment, which would have been biased against. Something that wouldn’t have happened if science systematically failed to work in such an extreme and ridiculous manner. I keep forgetting that I really really need to spell out any conclusions when arguing with you. It’s like you’re arguing that a car is missing the wheels but I just drove here on it.
That’s the one I know most about, obviously. I have no clue about what’s going on in China, Korea, or Japan.
So why do you think your defense would not apply equally well (or poorly) to them? What’s the phage of China?
Look, it doesn’t matter if phages work or don’t work! The treatment, in favour of which there would be strong bias, got replaced with another treatment, which would have been biased against. Something that wouldn’t have happened if science systematically failed to work in such an extreme and ridiculous manner.
Oh wow. What a convincing argument. ‘Look, some Russians once did this! Now they do that! No, it doesn’t matter if they were right or wrong before or after!’ Cool. So does that mean I get to point to every single change of medical treatment in the USA as evidence it’s just peachy there? ‘Look, some Americans once did lobotomy! Now they don’t! It doesn’t matter if lobotomies work or don’t work!’
I keep forgetting that I really really need to spell out any conclusions when arguing with you. It’s like you’re arguing that a car is missing the wheels but I just drove here on it.
You didn’t drive shit anywhere.
Besides, the 90%+ proportion of positive results is also the case in the west
That’s on a different dataset, covering more recent time periods, which, as the abstract says, still shows serious problems in East Asia (compromised by relatively small sample: trying to show trends in ‘AS’ using 204 studies over 17 years isn’t terribly precise compared to the 2627 they have for the USA) with the latest data being 85% vs 100%. And 100% significance is a ceiling, so who knows how bad the East Asian research has actually gotten during the same time period Western numbers continue to deteriorate...
If you want to persist in your mythical ideas regarding western civilization by postulating what ever you need and making shit up, there’s nothing I or anyone else can do about it.
You are trying to pull a “everyone and me is against you” stunt against Gwern? Do you have any idea how dumbfoundingly absurd this would sound to most of those of the class “anyone else” who happens to see this exchange?
I realize that, and I’ve already pointed out why the difference in rates is not going to be that large & that your cite does not explain the excess significance in their sample.
Doesn’t matter that much. Power, usually quite low, sets the upper limit to how many of the results should have been positive even if we assume every single one was testing a known-efficacious drug (which hypothesis raises its own problems: how is that consistent with your claims about the language bias towards publishing cool new results?)
So? I don’t care why the Russian literature is biased, just that it is.
Yes, but toxicology studies being done by industry is not aimed at academic publication, and the ones aimed at academic publication have the usual incentives to find something and so are part of the overall problem.
Huh? The paper finds that acupuncture study rates vary by region. USA/Sweden/Germany 53/59%/63%, China/Japan/Taiwan 100% etc
How much have you looked? There’s plenty of acupuncture centres in the USA despite a relatively low acupuncture success rate.
Does a fish notice water? But fine, maybe you don’t, feel free to supply your own example of Russian pseudoscience and traditional medicine. I doubt Russian science is a shining jewel of perfection with no faults given its 91% acupuncture success rate (admittedly on a small base).
Not sure that’s a good example, as Wikipedia seems to disagree about homebrew phage therapy not being applied: https://en.wikipedia.org/wiki/Phage_therapy#History
Anyway,
How do you see the unseen? Unless someone has done a large definitive RCT, how does one ever prove that a result was bogus? Nobody is ever going to take the time and resources to refute those shitty animal experiments with a much better experiment. Most scientific findings never gets that sort of black-and-white refutation, it just gets quietly forgotten and buried, and even the specialists don’t know about it. Most bad science doesn’t look like Lysenko. Or look at evidence-based medicine in the West: rubbish medicine doesn’t look like a crazy doc slicing open patients with a scalpel, it just looks like regular old medicine which ‘somehow’ turns up no benefit when rigorously tested and is quietly dropped from the medical textbooks.
To diagnose bad science, you need to look at overall metrics and indirect measures—like excess significance. Like 91% of acupuncture studies working.
Well, humans do notice air some of the time. (SCNR.)
If you want to persist in your mythical ideas regarding western civilization by postulating what ever you need and making shit up, there’s nothing I or anyone else can do about it.
Your study is making a more specific claim than mere bias in research, it’s claiming bias in one particular direction.
The point is that the SU was, mostly, using antibiotics (once production was set up, i.e. from some time after ww2).
Well, and there wasn’t a plenty in the soviet union despite supposedly higher success rate.
If you don’t know correct rate you can’t tell which specific rate is erroneous. It’s not realistically possible to construct a blind study of acupuncture, so, unlike, say, homoeopathy, it is a very shitty measure of research errors.
I really doubt that 91% of Russian language acupuncture studies published in Soviet Union found a positive effect (I dunno about 1991-1998 Russia, it was fucked up beyond belief at that time), and I don’t know how many studies should have found a positive effect (followed by a note that more adequate blinding must be invented to study it properly).
And we know that what ever was the case there was no Soviet abandonment of normal medicine in favour of acupuncture—the system somehow worked out ok in the end.
That’s not a reply to what I wrote.
Yes, that’s what a bias is. A systematic tendency in one direction. As opposed to random error.
And before that, they were using phages despite apparently pretty shaky evidence it was anything but a placebo. That said, pointing out the systematic bias of Russian science (among many other countries, and I’m fascinated, incidentally, how the only country you’re defending like this is… your own. No love for Korea?) does not commit me to the premise that phages do or not work—you’re the one who brought them up as an example of how excellent Russian science is, not me.
How many are there now? Shouldn’t you have looked that up?
Difference in rates is prima facie evidence of bias, due to the disagreement. If someone says A and someone else says not-A, you don’t need to know what A actually is to observe the contradiction and know at least one party is wrong.
Yes it is.
And naturally, you have not looked for anything on the topic, you just doubt it.
Strawman. No country engages in ‘abandonment of normal medicine’ - if you go to China, do you only find acupuncturists? Of course not. The problem is that you find acupuncturists sucking up resources in dispensing expensive placbeos and you find that the scientific community is not strong enough to resist the cultural & institutional pressures and find that acupuncture doesn’t work, resulting in real working medicine being intermeshed with pseudomedicine.
Fortunately, normal medicine (after tremendous investments in R&D and evidence-based medicine) currently works fairly well and I think it would take a long time for it to decay into something as overall bad as pre-modern Western medicine was; I also think some core concepts like germ theory are sufficiently simple & powerful that they can’t be lost, but that would be cold comfort in the hypothetical cargo cult scenario (‘good news: doctors still know what infections are and how to fight epidemics; bad news: everything else they do is so much witch-doctor mumbojumbo based on unproven new therapies, misinterpretations of old therapies which used to work, and traditional treatments like acupuncture’).
Unless A contains indexicals that point to different things in the two cases.
(Maybe Asian acupunturists are better than European ones, or maybe East Asians respond better to acupuncture than Caucasians for some reason, or...)
( I’m not saying that this is likely, just that it’s possible.)
I was referring to your other comment.
That’s the one I know most about, obviously. I have no clue about what’s going on in China, Korea, or Japan.
Look, it doesn’t matter if phages work or don’t work! The treatment, in favour of which there would be strong bias, got replaced with another treatment, which would have been biased against. Something that wouldn’t have happened if science systematically failed to work in such an extreme and ridiculous manner. I keep forgetting that I really really need to spell out any conclusions when arguing with you. It’s like you’re arguing that a car is missing the wheels but I just drove here on it.
Besides, the 90%+ proportion of positive results is also the case in the west
(also, in the past we had stuff like lobotomy in the west)
So why do you think your defense would not apply equally well (or poorly) to them? What’s the phage of China?
Oh wow. What a convincing argument. ‘Look, some Russians once did this! Now they do that! No, it doesn’t matter if they were right or wrong before or after!’ Cool. So does that mean I get to point to every single change of medical treatment in the USA as evidence it’s just peachy there? ‘Look, some Americans once did lobotomy! Now they don’t! It doesn’t matter if lobotomies work or don’t work!’
You didn’t drive shit anywhere.
That’s on a different dataset, covering more recent time periods, which, as the abstract says, still shows serious problems in East Asia (compromised by relatively small sample: trying to show trends in ‘AS’ using 204 studies over 17 years isn’t terribly precise compared to the 2627 they have for the USA) with the latest data being 85% vs 100%. And 100% significance is a ceiling, so who knows how bad the East Asian research has actually gotten during the same time period Western numbers continue to deteriorate...
You are trying to pull a “everyone and me is against you” stunt against Gwern? Do you have any idea how dumbfoundingly absurd this would sound to most of those of the class “anyone else” who happens to see this exchange?