I have to say, the reason the example is convincing is because of its artificiality. I don’t know many old-school frequentists (though I suppose I’m a frequentist myself, at least so far as I’m still really nervous about the whole priors business—but not quite so hard as all that), but I doubt that, presented with a stark case like the one above, they’d say the results would come out differently. For one thing, how would the math change?
But the case would never come up—that’s the thing. It’s empty counterfactual analysis. Nobody who is following a stopping rule as ridiculous as the one offered would be able to otherwise conduct the research properly. I mean, seriously. I think Benquo nailed it: the second researcher’s stopping rule ought to rather severely change our subjective probability in his/her having used a random sample, or for that matter not committed any number of other research sins, perhaps unconsciously. And that in turn should make us less confident about the results.
I have to say, the reason the example is convincing is because of its artificiality. I don’t know many old-school frequentists (though I suppose I’m a frequentist myself, at least so far as I’m still really nervous about the whole priors business—but not quite so hard as all that), but I doubt that, presented with a stark case like the one above, they’d say the results would come out differently. For one thing, how would the math change?
But the case would never come up—that’s the thing. It’s empty counterfactual analysis. Nobody who is following a stopping rule as ridiculous as the one offered would be able to otherwise conduct the research properly. I mean, seriously. I think Benquo nailed it: the second researcher’s stopping rule ought to rather severely change our subjective probability in his/her having used a random sample, or for that matter not committed any number of other research sins, perhaps unconsciously. And that in turn should make us less confident about the results.