A mentor of mine once told me that replication is useful, but not the most useful thing you could be doing because it’s often better to do a followup experiment that rests on the premises established by the initial experiment. If the first experiment was wrong, the second experiment will end up wrong too. Science should not go even slower than it already does—just update and move on, don’t obsess.
If you’re concerned about the velocity of scientific progress, you should also be concerned about wrong turns. A Type 1 Error (establishing a wrong result by incorrectly rejecting a null hypothesis) is, IMHO, far more damaging to science than failure to establish a correct result—possibly due to an insufficient experimental setup.
Yeah, there’s definitely an “exploration / rigor” trade-off here (or maybe “speed / accuracy”) and I’m not sure it’s clear which side we are erring on right now. I’m not terribly surprised that LW favors rigor, just due to the general personality profile of the users here, and that my favoring of exploration at the cost of being wrong a few times is in the minority.
I definitely think a rational agent would be more exploratory than science currently is, but on the other hand we’ve got systematic biases to contend with and rigor might offset that.
If you’re concerned about the velocity of scientific progress, you should also be concerned about wrong turns. A Type 1 Error (establishing a wrong result by incorrectly rejecting a null hypothesis) is, IMHO, far more damaging to science than failure to establish a correct result—possibly due to an insufficient experimental setup.
Yeah, there’s definitely an “exploration / rigor” trade-off here (or maybe “speed / accuracy”) and I’m not sure it’s clear which side we are erring on right now. I’m not terribly surprised that LW favors rigor, just due to the general personality profile of the users here, and that my favoring of exploration at the cost of being wrong a few times is in the minority.
I definitely think a rational agent would be more exploratory than science currently is, but on the other hand we’ve got systematic biases to contend with and rigor might offset that.