But it isn’t theory-of-induction-free. It just pretends to be. There’s a theory of induction right in there, where you correctly identify it, in step 4->5. It’s no better, no more likely to be true, and no more robust, merely on account of being squashed up small and hidden.
You haven’t truly minimized the “amount of induction” in the argument; only the “amount of induction that’s easily visible”, which I don’t think is a parameter that deserves minimizing. You’d need just the same amount of induction if, say, instead of doing classical NHST you did Bayesian inference or maximum likelihood (= Bayesian inference where you pretend not to have priors) or something. You could squash it up just as small, too; you’d just need to make steps 1-4 more quantitative.
Consider a case—they’re not hard to find—where you have a test statistic that’s low-probability on any of your model hypotheses. Then the same logic as you’ve used says that you’re “forced” to conclude that all your hypotheses are false—even if it happens that one of them is right. (In practice your model is never exactly right, but never mind that.) To me, this shows that the whole enterprise is fundamentally non-deductive, and that trying to make it look as much as possible like a pure deduction is actively harmful.
Maybe a better way of phrasing what I’m trying to point out is that induction is isolated to a single step. Instead of working directly with probabilities which require a theory of what probabilities are, NHST waves it’s hands a bit and treats the inductive step as deductive, but transparently so (once you lay out the deduction anyway).
Your point about a test statistic that’s low-probability on all possible model hypotheses is a good one—and it suggests that the details of hypothesis testing should change even if the general logic is kept. I doubt that the details of actually used hypothesis testing are ideal for “induction-free induction” (which I’m realizing is a bad name for what I’m trying to convey), but what I’m really talking about is the general logic. I’d be surprised if some of the details didn’t have to change.
I don’t think I disagree with anything in your comment though. I don’t think I have a strong argument for using hypothesis testing, but it may be that the general logic can be salvaged for a reasonable method of doing induction without fully fleshing out an inductive theory (this is why I said one step requires hand waving).
The steel man is hypothesis testing as a theory-of-induction free way of doing induction.
But it isn’t theory-of-induction-free. It just pretends to be. There’s a theory of induction right in there, where you correctly identify it, in step 4->5. It’s no better, no more likely to be true, and no more robust, merely on account of being squashed up small and hidden.
You haven’t truly minimized the “amount of induction” in the argument; only the “amount of induction that’s easily visible”, which I don’t think is a parameter that deserves minimizing. You’d need just the same amount of induction if, say, instead of doing classical NHST you did Bayesian inference or maximum likelihood (= Bayesian inference where you pretend not to have priors) or something. You could squash it up just as small, too; you’d just need to make steps 1-4 more quantitative.
Consider a case—they’re not hard to find—where you have a test statistic that’s low-probability on any of your model hypotheses. Then the same logic as you’ve used says that you’re “forced” to conclude that all your hypotheses are false—even if it happens that one of them is right. (In practice your model is never exactly right, but never mind that.) To me, this shows that the whole enterprise is fundamentally non-deductive, and that trying to make it look as much as possible like a pure deduction is actively harmful.
Maybe a better way of phrasing what I’m trying to point out is that induction is isolated to a single step. Instead of working directly with probabilities which require a theory of what probabilities are, NHST waves it’s hands a bit and treats the inductive step as deductive, but transparently so (once you lay out the deduction anyway).
Your point about a test statistic that’s low-probability on all possible model hypotheses is a good one—and it suggests that the details of hypothesis testing should change even if the general logic is kept. I doubt that the details of actually used hypothesis testing are ideal for “induction-free induction” (which I’m realizing is a bad name for what I’m trying to convey), but what I’m really talking about is the general logic. I’d be surprised if some of the details didn’t have to change.
I don’t think I disagree with anything in your comment though. I don’t think I have a strong argument for using hypothesis testing, but it may be that the general logic can be salvaged for a reasonable method of doing induction without fully fleshing out an inductive theory (this is why I said one step requires hand waving).