I’ve definitely seen this in the academic literature. And it’s extra annoying if the study used a small sample; the p-values are going to be large simply because the study didn’t collect much evidence.
OTOH, chemotherapy isn’t a very good example because there are other factors at work:
Chemotherapy has serious side effects. There are good reasons to be cautious in using extra.
There are also not-as-good reasons to avoid using extra chemotherapy. Medical care is highly regulated and liability-prone (to varying extents in various areas). In the US, insurers are notoriously reluctant to pay for any treatment they consider unnecessary. Departing from standard practice is likely to be expensive.
I think the fact that chemotherapy isn’t a very good example demonstrates a broader problem with this post: that maybe in general your beliefs will be more accurate if you stick with the null hypothesis until you have significant evidence otherwise. Doing so often protects you from confirmation bias, bias towards doing something, and the more general failure to imagine alternative possibilities. Sure, there are some cases where, on the inside view, you should update before the studies come in, but there are also plenty of cases where your inside view is just wrong.
I’ve definitely seen this in the academic literature. And it’s extra annoying if the study used a small sample; the p-values are going to be large simply because the study didn’t collect much evidence.
OTOH, chemotherapy isn’t a very good example because there are other factors at work:
Chemotherapy has serious side effects. There are good reasons to be cautious in using extra.
There are also not-as-good reasons to avoid using extra chemotherapy. Medical care is highly regulated and liability-prone (to varying extents in various areas). In the US, insurers are notoriously reluctant to pay for any treatment they consider unnecessary. Departing from standard practice is likely to be expensive.
I think the fact that chemotherapy isn’t a very good example demonstrates a broader problem with this post: that maybe in general your beliefs will be more accurate if you stick with the null hypothesis until you have significant evidence otherwise. Doing so often protects you from confirmation bias, bias towards doing something, and the more general failure to imagine alternative possibilities. Sure, there are some cases where, on the inside view, you should update before the studies come in, but there are also plenty of cases where your inside view is just wrong.