Tentatively, I think we must treat the two differently, in some respect somewhere, or we are vulnerable to manipulation. Where does the flaw lie in the following?
If the second researcher had instead said, “I am going to run 1000 experiments with 100 people each, and publish only those whose cure rate exceeds 60%”, there is a huge selection bias in the data we see and our update should be tiny if we can’t get access to the discarded data.
If the researcher decided instead “I am going to run 1000 experiments in parallel, adding one person at a time to each, and publish only those whose cure rate exceeds 60%,” then we would still see the same sort of bias in the data—after he has added 100 people to each experiment we are in the same situation as above.
If 1000 researchers, ideologically aligned in their commitment to this result, run experiments where they post their results when the cure rate exceeds 60%, this is equivalent to the second scenario.
Therefore, if enough people behaved like researcher 2, we would see biases in published data (which may conceivably balance out, with enough difference in ideology, but this does not seem to be guaranteed and I am not convinced it is even likely).
Therefore, researcher 2 should be considered to be acting badly, and we should reject (or at least weaken) experiments done in that fashion.
The above assumes that we do not see data from in-progress experiments. If we do, then I believe we can safely consider each data point with the same weight.
That he discarded data is additional information to update on, that was not present in the above example. It doesn’t matter what the researcher intended, if he got that data on the first try.
My point is that if we allow researchers to follow methodology like this, they will in practice be discarding data, and we will be unaware of it—even when each individual researcher is otherwise completely honest in how they deal with the data.
Tentatively, I think we must treat the two differently, in some respect somewhere, or we are vulnerable to manipulation. Where does the flaw lie in the following?
If the second researcher had instead said, “I am going to run 1000 experiments with 100 people each, and publish only those whose cure rate exceeds 60%”, there is a huge selection bias in the data we see and our update should be tiny if we can’t get access to the discarded data.
If the researcher decided instead “I am going to run 1000 experiments in parallel, adding one person at a time to each, and publish only those whose cure rate exceeds 60%,” then we would still see the same sort of bias in the data—after he has added 100 people to each experiment we are in the same situation as above.
If 1000 researchers, ideologically aligned in their commitment to this result, run experiments where they post their results when the cure rate exceeds 60%, this is equivalent to the second scenario.
Therefore, if enough people behaved like researcher 2, we would see biases in published data (which may conceivably balance out, with enough difference in ideology, but this does not seem to be guaranteed and I am not convinced it is even likely).
Therefore, researcher 2 should be considered to be acting badly, and we should reject (or at least weaken) experiments done in that fashion.
The above assumes that we do not see data from in-progress experiments. If we do, then I believe we can safely consider each data point with the same weight.
That he discarded data is additional information to update on, that was not present in the above example. It doesn’t matter what the researcher intended, if he got that data on the first try.
My point is that if we allow researchers to follow methodology like this, they will in practice be discarding data, and we will be unaware of it—even when each individual researcher is otherwise completely honest in how they deal with the data.