In general, it’s good to check your intuitions against evidence where possible (so, seek out experiments and treat experimentally validated hypotheses as much stronger than intuitions).
The valley being described here is the idea that you should just discard your intuitions in favor of the null hypothesis, not just when experiments have failed to reject the null hypothesis (though even here, they could just be underpowered!), but when experiments haven’t been done at all!
It’s a generalized form of an isolated demand for rigor, where whatever gets defined as a null hypothesis gets a free pass, but anything else has to prove itself to a high standard. And that leads to really poor performance in domains where evidence is hard to come by (quickly enough), relative to trusting intuitive priors and weak evidence when that’s all that’s available.
Correct, favoring hypothesis H or NOT H simply because you label one “null hypothesis” are both bad. Equally bad when you don’t have evidence either way.
In this case, intuition favors “more chemo should kill more cancer cells”, and intuition counts as some evidence. The doctor ignores intuition (which is the only evidence we have here) and favors the opposite hypothesis because it’s labeled “null hypothesis”.
I was suggesting that there might be ways of assigning the label of “null hypothesis”.
X is good, more X is good. (intuition favors “more chemo should kill more cancer cells”)
X has a cost, we go as far as the standards say, and stop there. (Chemo kills cells. This works on your cells, and cancer cells. Maybe chemo isn’t like shooting someone—they aren’t that likely to die as a result—but just as you wouldn’t shoot someone to improve their health unless it was absolutely necessary, and no more, chemo should be treated the same way.) “Do no harm.” (This may implicitly distinguish between action and inaction.)
In general, it’s good to check your intuitions against evidence where possible (so, seek out experiments and treat experimentally validated hypotheses as much stronger than intuitions).
The valley being described here is the idea that you should just discard your intuitions in favor of the null hypothesis, not just when experiments have failed to reject the null hypothesis (though even here, they could just be underpowered!), but when experiments haven’t been done at all!
It’s a generalized form of an isolated demand for rigor, where whatever gets defined as a null hypothesis gets a free pass, but anything else has to prove itself to a high standard. And that leads to really poor performance in domains where evidence is hard to come by (quickly enough), relative to trusting intuitive priors and weak evidence when that’s all that’s available.
Having the reverse as the null hypothesis is also bad. Which is worse?
Correct, favoring hypothesis H or NOT H simply because you label one “null hypothesis” are both bad. Equally bad when you don’t have evidence either way.
In this case, intuition favors “more chemo should kill more cancer cells”, and intuition counts as some evidence. The doctor ignores intuition (which is the only evidence we have here) and favors the opposite hypothesis because it’s labeled “null hypothesis”.
I was suggesting that there might be ways of assigning the label of “null hypothesis”.
X is good, more X is good. (intuition favors “more chemo should kill more cancer cells”)
X has a cost, we go as far as the standards say, and stop there. (Chemo kills cells. This works on your cells, and cancer cells. Maybe chemo isn’t like shooting someone—they aren’t that likely to die as a result—but just as you wouldn’t shoot someone to improve their health unless it was absolutely necessary, and no more, chemo should be treated the same way.) “Do no harm.” (This may implicitly distinguish between action and inaction.)