You can claim that it should have the same likelihood either way, but you have to put the discrepancy somewhere.
Knowing the choice of stopping rule is evidence about the experimenter’s state of knowledge about the efficacy. You can say that it should be treated as a separate piece of evidence, or that knowing about the stopping rule should change your prior, but if you don’t bring it in somewhere, you’re ignoring critical information.
No, it practical terms it’s negligible. There’s a reason that double-blind trials are the gold standard—it’s because doctors are as prone to cognitive biases as anyone else.
Let me put it this way: recently a pair of doctors looked at the available evidence and concluded (foolishly!) that putting fecal bacteria in the brains of brain cancer patients was such a promising experimental treatment that they did an end-run around the ethics review process—and after leaving that job under a cloud, one of them was still considered a “star free agent”. Well, perhaps so—but I think this little episode illustrates very well that a doctor’s unsupported opinion about the efficacy of his or her novel experimental treatment isn’t worth the shit s/he wants to place inside your skull.
Of course experimental design is very important in general. But VAuroch and I agree that when two designs give rise to the same likelihood function, the information that comes in from the data are equivalent. We disagree about the weight to give to the information that comes in from what the choice of experimental design tells us about the experimenter’s prior state of knowledge.
Double-blind trials aren’t the gold standard, they’re the best available standard. They still don’t replicate far too often, because they don’t remove bias (and I’m not just referring to publication bias). Which is why, when considering how to interpret a study, you look at the history of what scientific positions the experimenter has supported in the past, and then update away from that to compensate for bias which you have good reason to think will show up in their data.
In the example, past results suggest that, even if the trial was double-blind, someone who is committed to achieving a good result for the treatment will get more favorable data than some other experimenter with no involvement.
And that’s on top of the trivial fact that someone with an interest in getting a successful trial is more likely to use a directionally-slanted stopping rule if they have doubts about the efficacy than if they are confident it will work, which is not explicitly relevant in Eliezer’s example.
You can claim that it should have the same likelihood either way, but you have to put the discrepancy somewhere. Knowing the choice of stopping rule is evidence about the experimenter’s state of knowledge about the efficacy. You can say that it should be treated as a separate piece of evidence, or that knowing about the stopping rule should change your prior, but if you don’t bring it in somewhere, you’re ignoring critical information.
No, it practical terms it’s negligible. There’s a reason that double-blind trials are the gold standard—it’s because doctors are as prone to cognitive biases as anyone else.
Let me put it this way: recently a pair of doctors looked at the available evidence and concluded (foolishly!) that putting fecal bacteria in the brains of brain cancer patients was such a promising experimental treatment that they did an end-run around the ethics review process—and after leaving that job under a cloud, one of them was still considered a “star free agent”. Well, perhaps so—but I think this little episode illustrates very well that a doctor’s unsupported opinion about the efficacy of his or her novel experimental treatment isn’t worth the shit s/he wants to place inside your skull.
Hold on- aren’t you saying the choice of experimental rule is VERY important (i.e. double blind vs. not double blind,etc)?
If so you are agreeing with VAuroch. You have to include the details of the experiment somewhere. The data does not speak for itself.
Of course experimental design is very important in general. But VAuroch and I agree that when two designs give rise to the same likelihood function, the information that comes in from the data are equivalent. We disagree about the weight to give to the information that comes in from what the choice of experimental design tells us about the experimenter’s prior state of knowledge.
Double-blind trials aren’t the gold standard, they’re the best available standard. They still don’t replicate far too often, because they don’t remove bias (and I’m not just referring to publication bias). Which is why, when considering how to interpret a study, you look at the history of what scientific positions the experimenter has supported in the past, and then update away from that to compensate for bias which you have good reason to think will show up in their data.
In the example, past results suggest that, even if the trial was double-blind, someone who is committed to achieving a good result for the treatment will get more favorable data than some other experimenter with no involvement.
And that’s on top of the trivial fact that someone with an interest in getting a successful trial is more likely to use a directionally-slanted stopping rule if they have doubts about the efficacy than if they are confident it will work, which is not explicitly relevant in Eliezer’s example.
I can’t say I disagree.