Eliezer, I accept your point about the underlying laws of probability. However, your example is extremely flawed.
Of course what the researcher operates by should affect our interpretation of the evidence; it is, in itself, another piece of evidence!
Specifically in this case, publishing your research only when you reach a certain conclusion implies that any similar researches that did not reach this threshold did not get published, and are thus not available to our evidence pool. This is filtered evidence.
So without knowing how many similar researches were conducted, the conclusion from the one research that did get published can’t be seen as very strong. Do I need to draw the Bayesian analysis that shows why?
Eliezer, I accept your point about the underlying laws of probability. However, your example is extremely flawed.
Of course what the researcher operates by should affect our interpretation of the evidence; it is, in itself, another piece of evidence! Specifically in this case, publishing your research only when you reach a certain conclusion implies that any similar researches that did not reach this threshold did not get published, and are thus not available to our evidence pool. This is filtered evidence.
So without knowing how many similar researches were conducted, the conclusion from the one research that did get published can’t be seen as very strong. Do I need to draw the Bayesian analysis that shows why?