When drug trials don’t measure whether their efforts at blinding are actually successful that behavior is very similar to that of the rat psychologists that Feymann talks in his famous post about Cargo-Cult Science.
Running trials in a flawed way is bad, but not as bad as running no trials at all. Nobody has ever suggested that the current system is perfect and can’t be improved in any way, so who are you arguing with?
I don’t know whether that hypothesis is true but Evidence-based Medicine advocates don’t seem to try to find out whether it is and do science to find out, they rather do what Feymann described.
Presumably, Evidence-based Medicine advocates have decided to spend their limited resources on developing treatments using the current system, rather than on improving the system itself. Their position is not inconsistent, irrational or objectively worse than yours, as you may be suggesting.
I don’t think that mono-cultures are in general better than diversity. Laws that standardize a body of knowledge make innovation a lot harder. Medical education is very expensive because there are no system pressures to focus on teaching a doctor exactly the skills he needs to help his patients.
Mono-cultures have their benefits. Without a standardized body of knowledge most innovation is just reinventing the wheel. Was medical education very cheap and short before Evidence-based Medicine?
More generally, I don’t understand your position. On one hand you’re arguing that current tests aren’t rigorous enough, and the the doctors don’t understand statistics well enough. And on another hand, that we don’t need to run tests or teach statistics at all?
Running trials in a flawed way is bad, but not as bad as running no trials at all.
How do you know? In a scenario where all the effect of antidepressive interventions are “placebo” you will get worse outcomes with a system that tests diffenert pills against each other because the pill with the most obvious side-effects is going to win (because it unblinds itself).
Presumably, Evidence-based Medicine advocates have decided to spend their limited resources on developing treatments using the current system, rather than on improving the system itself. Their position is not inconsistent, irrational or objectively worse than yours, as you may be suggesting.
You can argue that Cargo Cults aren’t objectively worse that non-Cargo Cults because they just choose to use their limited resources in another way then checking whether their methods work but that doesn’t change the fact that it’s a Cargo Cult.
More generally, I don’t understand your position. On one hand you’re arguing that current tests aren’t rigorous enough, and the the doctors don’t understand statistics well enough. And on another hand, that we don’t need to run tests or teach statistics at all?
This post is focused on criticism and not on advocating alternatives. My post on Prediction-based Medicine lays out one approach of an alternative system that also uses statistics but very different statistics.
Kaj Sotala recently wrote how he used the NLP book Transform yourself by Steve Andreas to cure his depression. I see such a report as a reason to read the book and learn the underlying skills even through it’s not Evidence-based Medicine.
On the other hand there are people who reject that kind of decision making and want to outlaw it.
To show that Evidence-based Medicine is a worse system than what we had before, you’d have to show that health outcomes are declining (in a way that can’t be explained by external changes). I’m not aware that this is true, and you haven’t argued for it. It’s possible that the current system does fail miserably for depression treatments, but is still worth keeping for other benefits.
Kaj Sotala recently wrote how he used the NLP book Transform yourself by Steve Andreas to cure his depression.
Again, on one hand you’re criticizing trials for not having perfect placebos, and on the other hand you’re pointing to a single self-selected self-reported claim with no control at all, as a positive example. To be fair, I haven’t read your other post, maybe you have devised a statistically sound mechanism that can make use of such claims. However, that’s not what your position look like from here.
I’m not aware that this is true, and you haven’t argued for it.
I haven’t argued that it’s true. I have argued that we don’t know it’s true and you claim to know that you know it’s true. Do you take that claim back?
If I got you to the point where you agree that we don’t whether or not those trials help, my post had a decent impact.
Again, on one hand you’re criticizing trials for not having perfect placebos, and on the other hand you’re pointing to a single self-selected self-reported claim with no control at all, as a positive example.
No. I’m critizing them for pretending that they have perfect placebo’s when they don’t. Gathering the data about whether patients think they got a placebo or don’t think they got a placebo isn’t expensive. The only reason not to do it, is that it’s awkward and exposes the messiness of the underlying reality.
One issue with Evidence-based medicine is that people start trusting what other people wrote down instead of trusting their own empiric experience and ability to reason.
Eliezer wrote in Inadequate Equilibria about he cured the SAD of his wife by a combination of his own empirics and reasoning. He argued that our health care system is disfunctional enough that the present system has no good way to provide a effective market. Eliezer proposes his own system of how health care should work that also feature pay-by-performance. I would be happy with Eliezers system getting implemented.
What exactly?
There are people out there who advocate for various legal instruments to forbid the practice of medicine in ways that doesn’t follow Evidence-based medicine standards. But I don’t want to go too much into the details of existing health regulation here.
I have argued that we don’t know it’s true and you claim to know that you know it’s true.
Maybe I was unclear, I was referring to general health outcomes such as life expectancy, cancer survival rates, etc. Those are being measured and seem to be moving in positive directions. If medicine had really become a Cargo Cult, I would have expected them to be all stagnant or falling.
I’m critizing them for pretending that they have perfect placebo’s when they don’t.
Are you also the sort of person who criticizes economists for assuming that humans are rational, when they aren’t? It sounds similar. Both criticisms are fair, but their significance is dubious.
Inadequate
Yes, the system is inadequate, always has been, EBM has nothing to do with it. To clarify, I’m under the impression that you claim the system before EBM to be superior to EBM. Maybe that’s not true?
There are people out there who advocate for various legal instruments to forbid the practice of medicine in ways that doesn’t follow Evidence-based medicine standards.
Are you talking about regulatory capture? Again, that’s not a new problem, EBM has nothing to do with it.
Maybe I was unclear, I was referring to general health outcomes such as life expectancy, cancer survival rates, etc.
There’s a easy way to improve cancer survival rates. Diagnose more people who don’t have any problems with cancer. You might harm them by taking out some of their organs but they won’t die from that and you have increased cancer survival rates as a result.
In the last 50 years we spent a lot of effort in putting more filters into our power plants and getting cleaner air. That might be a way to explain the 5% fall in cancer deaths. People smoke less thanks to insight gathered from correlational studies and not well controlled trials. EBM as commonly practiced would suggest trusting those correlational studies less.
Are you also the sort of person who criticizes economists for assuming that humans are rational, when they aren’t? It sounds similar. Both criticisms are fair, but their significance is dubious.
Economists are interested in ways in which humans aren’t rational. There’s a lot of research money invested into behavorial economics. I can’t see the medical community giving Kirsch their Nobel price in the same way the economics community gave Kahnemann their Nobel price.
As far as I know there are no simply steps like recording which of the patients actually thinks they got the real drug that’s available to economists but that they don’t take.
Practically another problem with treating placebo’s this way is that doctors in our system don’t attempt to maximize positive placebo effects and minimize nocebo effects.
To clarify, I’m under the impression that you claim the system before EBM to be superior to EBM.
I’m not making a claim that is that strong. A lot of the ways I think EBM reduces innovation are about providing justifications for regulatory capture.
From my perspective it’s also more important to ask how we could create a system that’s better than what we have now then whether the system of the past was better. A system like the one I proposed under the title Prediction-based Medicine wouldn’t have been possible before the internet.
Sarah Constantin writes on her blog: “Cancer deaths have only fallen by 5% since 1950”
The post seems reasonable. It points out some stagnation and some isolated wins (also, the 3x reduction in heart disease?). It could be used to claim that cancer research in inadequate, but it absolutely does not defend your “EBM is a Cargo Cult” rhetoric.
There’s a lot of research money invested into behavorial economics.
That’s relatively recent. There is also a number of obvious reasons why medicine would lag behind economics in rethinking their dubious assumptions. Still I predict that in the coming decades some of those issues will be addressed.
how we could create a system that’s better than what we have now
I fully support your advocacy for better placebos. However, your PBM has issues, as all simple solutions to hard problems do.
I don’t think that the rat psychologists didn’t create any valid knowledge and I don’t think Feymann thought so either. He called them Cargo Cultists because they don’t care about investigating the assumption on which their research rests. I think the same is true with EBM.
Feymann says to not engage in Cargo Cultism the missing ingrediant is:
It’s a kind of scientific integrity, a principle of scientific thought that corresponds to a kind of utter honesty—a kind of leaning over backwards. For example, if you’re doing an experiment, you should report everything that you think might make it invalid—not only what you think is right about it: other causes that could possibly explain your results; and things you thought of that you’ve eliminated by some other experiment, and how they worked—to make sure the other fellow can tell they have been eliminated.
When researchers fails to report information about how well their attempts at blinding actually blind people they aren’t living up to that standard.
However, your PBM has issues, as all simple solutions to hard problems do.
It’s not perfect but it doesn’t have to be perfect to be better than the status quo. I’m also not calling for monoculture and that everything has to be done via PBM.
they don’t care about investigating the assumption on which their research rests
That sounds like a problem with the researchers and not with the system. I understand that you want to solve this problem with better incentives, but I don’t actually see how PBM helps with that.
It’s not perfect but it doesn’t have to be perfect to be better than the status quo.
The problem with novel ideas is that we’re often not clever enough to predict all the ways they will go wrong. Therefore, if a novel idea looks only slightly better than status quo, it’s probably going to be worse than the status quo by the time we’ve implemented it. And that’s before we consider switching costs.
I’m also not calling for monoculture and that everything has to be done via PBM.
That’s weird, you criticized cancer research a lot, but it seems that PBM wasn’t intended for that? It’s okay to have partial solutions. But while reading your comments about cancer, I assumed that you did have better ideas.
That’s partly what I meant by “weird tone” in the first comment. From my point of view the system is almost the best thing humans could reasonably make, with any flaws relatively minor and with some solutions presumably on the way. And from your point of view, presumably, it’s fatally flawed and near useless. But you aren’t providing much evidence that would make me change my view at all.
That sounds like a problem with the researchers and not with the system.
I don’t think that the placebo problem is caused by individual researchers being stupid but because of the incentives that the system sets.
Researchers don’t care for investigating the foundations because they can’t get grants for that purpose. On the other hand, they get grants for doing research that might lead to new drugs that make billions in profit.
Therefore, if a novel idea looks only slightly better than status quo, it’s probably going to be worse than the status quo by the time we’ve implemented it.
The solution I outline in my post is to not start by competing with hospitals but by going for treatments that are currently provided by hypnotists and bodyworkers who don’t practice much EBM anyway. It’s a class of people where individual skill between practioners matter a great deal and studies are therefore less likely to generalize than studies that are about giving out pills.
If you take chiropractics, which happens to be a class of body-workers where there are a lot of them it took till 2008 till we had a Cochrane meta-study according to which chiropractics provide a working treatment for lower back pain.
Moving from a system that takes decades to come to that conclusion and not being able to distinguish between skill differences between different chiropractics to a system that can do that in a year is more than just “slightly better”.
But you aren’t providing much evidence that would make me change my view at all.
A lot of the value from changing the system depends on how inadequate you believe the present system to be. It’s inherently difficult to provide evidence about the amount of low hanging fruit that’s out there because by it’s very nature providing you cases like EY’s SAD treatment means that there is only anecdotal evidence for those treatments.
This post from Sarah Constantin might give some indication that there are a lot of how hanging fruit out there that aren’t picked by our current system.
In The legend of healthcare Michael Vassar uses the inability of our system to get doctors to use mirrors to treat phantom limb pain as evidence that we don’t really have a healthcare system. Prediction-based Medicine would make it easy for one provider of mirros treatment to offer his treatment to all the people with phantom limb pain who seek treatment.
Researchers don’t care for investigating the foundations because they can’t get grants for that purpose. On the other hand, they get grants for doing research that might lead to new drugs that make billions in profit.
I don’t see why EBM is to blame or PBM would help.
Cochrane meta-study according to which chiropractics provide a working treatment for lower back pain
Reading the abstract, it doesn’t look all that positive.
more than just “slightly better”.
If you’ll let me be witty, I’ll suggest that your claims are lacking a kind of scientific integrity, a principle of scientific thought that corresponds to a kind of utter honesty—a kind of leaning over backwards. You’re not stating the assumptions that lead you to this conclusion, not explaining possible ways the assumptions could be wrong, and more generally, you don’t seem to try to find possible negative consequences or limitations of PBM. To be honest though, I don’t really want to discuss these possible problems. This thread already has enough going on.
It’s inherently difficult to provide evidence about the amount of low hanging fruit that’s out there
For example, some of these fruit involve wasted money (especially in US). I can agree that they are low hanging, because there are other medical systems providing similar outcomes in similar circumstances, for less money, and I can vaguely imagine that policies could be adopted in US to reduce costs. By the way, cost problems have very little to do with EBM. The talk you linked to also seems less about how EBM is inadequate and more about how doctors are failing to make good use of EBM.
More generally, I don’t need a list of problems that kind of look easy to solve. I need you to show me why exactly the problems aren’t already solved, how much benefit there would be if we did solve them, and that there exists a concrete and simple plan to solve them without assuming that we live in full communism.
Running trials in a flawed way is bad, but not as bad as running no trials at all. Nobody has ever suggested that the current system is perfect and can’t be improved in any way, so who are you arguing with?
Presumably, Evidence-based Medicine advocates have decided to spend their limited resources on developing treatments using the current system, rather than on improving the system itself. Their position is not inconsistent, irrational or objectively worse than yours, as you may be suggesting.
Mono-cultures have their benefits. Without a standardized body of knowledge most innovation is just reinventing the wheel. Was medical education very cheap and short before Evidence-based Medicine?
More generally, I don’t understand your position. On one hand you’re arguing that current tests aren’t rigorous enough, and the the doctors don’t understand statistics well enough. And on another hand, that we don’t need to run tests or teach statistics at all?
How do you know? In a scenario where all the effect of antidepressive interventions are “placebo” you will get worse outcomes with a system that tests diffenert pills against each other because the pill with the most obvious side-effects is going to win (because it unblinds itself).
You can argue that Cargo Cults aren’t objectively worse that non-Cargo Cults because they just choose to use their limited resources in another way then checking whether their methods work but that doesn’t change the fact that it’s a Cargo Cult.
This post is focused on criticism and not on advocating alternatives. My post on Prediction-based Medicine lays out one approach of an alternative system that also uses statistics but very different statistics.
Kaj Sotala recently wrote how he used the NLP book Transform yourself by Steve Andreas to cure his depression. I see such a report as a reason to read the book and learn the underlying skills even through it’s not Evidence-based Medicine.
On the other hand there are people who reject that kind of decision making and want to outlaw it.
To show that Evidence-based Medicine is a worse system than what we had before, you’d have to show that health outcomes are declining (in a way that can’t be explained by external changes). I’m not aware that this is true, and you haven’t argued for it. It’s possible that the current system does fail miserably for depression treatments, but is still worth keeping for other benefits.
Again, on one hand you’re criticizing trials for not having perfect placebos, and on the other hand you’re pointing to a single self-selected self-reported claim with no control at all, as a positive example. To be fair, I haven’t read your other post, maybe you have devised a statistically sound mechanism that can make use of such claims. However, that’s not what your position look like from here.
What exactly?
I haven’t argued that it’s true. I have argued that we don’t know it’s true and you claim to know that you know it’s true. Do you take that claim back?
If I got you to the point where you agree that we don’t whether or not those trials help, my post had a decent impact.
No. I’m critizing them for pretending that they have perfect placebo’s when they don’t. Gathering the data about whether patients think they got a placebo or don’t think they got a placebo isn’t expensive. The only reason not to do it, is that it’s awkward and exposes the messiness of the underlying reality.
One issue with Evidence-based medicine is that people start trusting what other people wrote down instead of trusting their own empiric experience and ability to reason.
Eliezer wrote in Inadequate Equilibria about he cured the SAD of his wife by a combination of his own empirics and reasoning. He argued that our health care system is disfunctional enough that the present system has no good way to provide a effective market. Eliezer proposes his own system of how health care should work that also feature pay-by-performance. I would be happy with Eliezers system getting implemented.
There are people out there who advocate for various legal instruments to forbid the practice of medicine in ways that doesn’t follow Evidence-based medicine standards. But I don’t want to go too much into the details of existing health regulation here.
Maybe I was unclear, I was referring to general health outcomes such as life expectancy, cancer survival rates, etc. Those are being measured and seem to be moving in positive directions. If medicine had really become a Cargo Cult, I would have expected them to be all stagnant or falling.
Are you also the sort of person who criticizes economists for assuming that humans are rational, when they aren’t? It sounds similar. Both criticisms are fair, but their significance is dubious.
Yes, the system is inadequate, always has been, EBM has nothing to do with it. To clarify, I’m under the impression that you claim the system before EBM to be superior to EBM. Maybe that’s not true?
Are you talking about regulatory capture? Again, that’s not a new problem, EBM has nothing to do with it.
There’s a easy way to improve cancer survival rates. Diagnose more people who don’t have any problems with cancer. You might harm them by taking out some of their organs but they won’t die from that and you have increased cancer survival rates as a result.
Sarah Constantin writes on her blog: “Cancer deaths have only fallen by 5% since 1950”
In the last 50 years we spent a lot of effort in putting more filters into our power plants and getting cleaner air. That might be a way to explain the 5% fall in cancer deaths. People smoke less thanks to insight gathered from correlational studies and not well controlled trials. EBM as commonly practiced would suggest trusting those correlational studies less.
Economists are interested in ways in which humans aren’t rational. There’s a lot of research money invested into behavorial economics. I can’t see the medical community giving Kirsch their Nobel price in the same way the economics community gave Kahnemann their Nobel price.
As far as I know there are no simply steps like recording which of the patients actually thinks they got the real drug that’s available to economists but that they don’t take.
Practically another problem with treating placebo’s this way is that doctors in our system don’t attempt to maximize positive placebo effects and minimize nocebo effects.
I’m not making a claim that is that strong. A lot of the ways I think EBM reduces innovation are about providing justifications for regulatory capture.
From my perspective it’s also more important to ask how we could create a system that’s better than what we have now then whether the system of the past was better. A system like the one I proposed under the title Prediction-based Medicine wouldn’t have been possible before the internet.
The post seems reasonable. It points out some stagnation and some isolated wins (also, the 3x reduction in heart disease?). It could be used to claim that cancer research in inadequate, but it absolutely does not defend your “EBM is a Cargo Cult” rhetoric.
That’s relatively recent. There is also a number of obvious reasons why medicine would lag behind economics in rethinking their dubious assumptions. Still I predict that in the coming decades some of those issues will be addressed.
I fully support your advocacy for better placebos. However, your PBM has issues, as all simple solutions to hard problems do.
I don’t think that the rat psychologists didn’t create any valid knowledge and I don’t think Feymann thought so either. He called them Cargo Cultists because they don’t care about investigating the assumption on which their research rests. I think the same is true with EBM.
Feymann says to not engage in Cargo Cultism the missing ingrediant is:
When researchers fails to report information about how well their attempts at blinding actually blind people they aren’t living up to that standard.
It’s not perfect but it doesn’t have to be perfect to be better than the status quo. I’m also not calling for monoculture and that everything has to be done via PBM.
That sounds like a problem with the researchers and not with the system. I understand that you want to solve this problem with better incentives, but I don’t actually see how PBM helps with that.
The problem with novel ideas is that we’re often not clever enough to predict all the ways they will go wrong. Therefore, if a novel idea looks only slightly better than status quo, it’s probably going to be worse than the status quo by the time we’ve implemented it. And that’s before we consider switching costs.
That’s weird, you criticized cancer research a lot, but it seems that PBM wasn’t intended for that? It’s okay to have partial solutions. But while reading your comments about cancer, I assumed that you did have better ideas.
That’s partly what I meant by “weird tone” in the first comment. From my point of view the system is almost the best thing humans could reasonably make, with any flaws relatively minor and with some solutions presumably on the way. And from your point of view, presumably, it’s fatally flawed and near useless. But you aren’t providing much evidence that would make me change my view at all.
I don’t think that the placebo problem is caused by individual researchers being stupid but because of the incentives that the system sets.
Researchers don’t care for investigating the foundations because they can’t get grants for that purpose. On the other hand, they get grants for doing research that might lead to new drugs that make billions in profit.
The solution I outline in my post is to not start by competing with hospitals but by going for treatments that are currently provided by hypnotists and bodyworkers who don’t practice much EBM anyway. It’s a class of people where individual skill between practioners matter a great deal and studies are therefore less likely to generalize than studies that are about giving out pills.
If you take chiropractics, which happens to be a class of body-workers where there are a lot of them it took till 2008 till we had a Cochrane meta-study according to which chiropractics provide a working treatment for lower back pain.
Moving from a system that takes decades to come to that conclusion and not being able to distinguish between skill differences between different chiropractics to a system that can do that in a year is more than just “slightly better”.
A lot of the value from changing the system depends on how inadequate you believe the present system to be. It’s inherently difficult to provide evidence about the amount of low hanging fruit that’s out there because by it’s very nature providing you cases like EY’s SAD treatment means that there is only anecdotal evidence for those treatments.
This post from Sarah Constantin might give some indication that there are a lot of how hanging fruit out there that aren’t picked by our current system.
In The legend of healthcare Michael Vassar uses the inability of our system to get doctors to use mirrors to treat phantom limb pain as evidence that we don’t really have a healthcare system. Prediction-based Medicine would make it easy for one provider of mirros treatment to offer his treatment to all the people with phantom limb pain who seek treatment.
I don’t see why EBM is to blame or PBM would help.
Reading the abstract, it doesn’t look all that positive.
If you’ll let me be witty, I’ll suggest that your claims are lacking a kind of scientific integrity, a principle of scientific thought that corresponds to a kind of utter honesty—a kind of leaning over backwards. You’re not stating the assumptions that lead you to this conclusion, not explaining possible ways the assumptions could be wrong, and more generally, you don’t seem to try to find possible negative consequences or limitations of PBM. To be honest though, I don’t really want to discuss these possible problems. This thread already has enough going on.
For example, some of these fruit involve wasted money (especially in US). I can agree that they are low hanging, because there are other medical systems providing similar outcomes in similar circumstances, for less money, and I can vaguely imagine that policies could be adopted in US to reduce costs. By the way, cost problems have very little to do with EBM. The talk you linked to also seems less about how EBM is inadequate and more about how doctors are failing to make good use of EBM.
More generally, I don’t need a list of problems that kind of look easy to solve. I need you to show me why exactly the problems aren’t already solved, how much benefit there would be if we did solve them, and that there exists a concrete and simple plan to solve them without assuming that we live in full communism.