If there is a difference, it is not because the experiments went differently, it is because the experiments could have gone differently, and so the likelihoods of them happening the way they did happen is different.
The Monty Hall problem was mentioned above. I pick a door, Monty opens a door to reveal a goat, I can stick or switch (but can’t take the goat). Whether Monty is picking a random door or picking the door he knows doesn’t have the goat, the evidence is the same—Monty opened a door and revealed a goat. But if Monty what matters is what might have happened otherwise. If Monty always picks a door with a goat, then I win if I switch 2⁄3 of the time. If Monty might have picked the door with the car (and just happened not to), I win if I switch only 50% of the time.
Same evidence, different conclusions based solely on what someone might have done otherwise not based on what actually happened; and I am confident of the difference in the Monty Hall problem, as I have not only read about it but also simulated it.
In the situation given, Researcher 1 did stop at 100 experiments, but might have stopped at 49, or 280. Researcher 2 was sure to stop at 100. I am not unwilling to accept that this doesn’t change the meaning of the evidence, in this case, but I do not understand at all why it should be “obvious” that it can’t, given that it does in the case of the Monty Hall problem.
The difference is that depending on Monty’s algorithm, there is a different probability of getting the exact result we saw, namely seeing a goat. The exact event we actually saw happens with different probability depending on Monty’s rule, so Monty’s rule changes the meaning of that result.
The researchers don’t get a given exact sequence of 100 results with different probability depending on their state of mind—their state of mind is not part of the state of the world that the result sequence tells us about, the way Monty’s state of mind is part of the world that generates the exact goat.
To look at it another way, a spy watching Monty open doors and get goats would determine that Monty was deliberately avoiding the prize. Watching a researcher stop at 100 results doesn’t tell you anything about whether the researcher planned to stop at 100 or after getting a certain number of successes. So, just like that result doesn’t tell you anything about the researcher’s state of mind, knowing about the researcher’s state of mind doesn’t tell you anything about the result.
Suppose that the frequency of cures actually converges at 60%, and each researcher performs his experiment 100 times. Researcher A should end up with about 6000 cures out of 10000, and Researcher B should end up with .7n cures out of n. We would expect in advance, after being told that Researcher B ended up testing 10000 people, that he encountered 7000 cures.
It seems that once we know that Researcher B was not going to stop until the incidence of cures was 70%, we do not learn anything further about the efficacy of the treatment by looking at his data.
You are proposing to partition the data into two observations, namely “number of trials that were performed” and “the results of those trials”.
The information you have after observing the first part and not the second, does depend on the researcher’s stopping criterion. And the amount of information you gain from the second observation also so depends (since some of it was already known from the first observation in one case and not in the other). But your state of information after both observations does not depend on the stopping criterion.
Also, the most likely outcome of your proposed experiment is not .7n cures for some n. Rather, it’s that Researcher B never stops.
Did you read the chapter linked at the end of the post?
A hopefully intuitive explanation: A spy watching the experiments and using Bayesian methods to make his own conclusions about the results, will not see any different evidence in each case and so will end up with the same probability estimate regardless of which experimenter he watched.
While the second experimenter might be contributing to publication bias by using that method in general, he nonetheless should not have come up with a different result.
It seems worth noting the tension between this and bottom-line reasoning. Could the second experimenter have come up with the desired result no matter what, given infinite time? And if so, is there any further entanglement between his hypothesis and reality?
Monty Hall is analogous in that we are looking at evidence and trying to make conclusions about likelihoods. It is relevant because the likelihoods are different depending on what was in Monty’s head in the past, after observing the same physical evidence. Monty is not the experimenter; where does that make a difference? Could one reformulate it so that he was? He would be running two different experiments, surely—but then why isn’t that the case for the two researchers?
If there is a difference, it is not because the experiments went differently, it is because the experiments could have gone differently, and so the likelihoods of them happening the way they did happen is different.
The Monty Hall problem was mentioned above. I pick a door, Monty opens a door to reveal a goat, I can stick or switch (but can’t take the goat). Whether Monty is picking a random door or picking the door he knows doesn’t have the goat, the evidence is the same—Monty opened a door and revealed a goat. But if Monty what matters is what might have happened otherwise. If Monty always picks a door with a goat, then I win if I switch 2⁄3 of the time. If Monty might have picked the door with the car (and just happened not to), I win if I switch only 50% of the time.
Same evidence, different conclusions based solely on what someone might have done otherwise not based on what actually happened; and I am confident of the difference in the Monty Hall problem, as I have not only read about it but also simulated it.
In the situation given, Researcher 1 did stop at 100 experiments, but might have stopped at 49, or 280. Researcher 2 was sure to stop at 100. I am not unwilling to accept that this doesn’t change the meaning of the evidence, in this case, but I do not understand at all why it should be “obvious” that it can’t, given that it does in the case of the Monty Hall problem.
The difference is that depending on Monty’s algorithm, there is a different probability of getting the exact result we saw, namely seeing a goat. The exact event we actually saw happens with different probability depending on Monty’s rule, so Monty’s rule changes the meaning of that result.
The researchers don’t get a given exact sequence of 100 results with different probability depending on their state of mind—their state of mind is not part of the state of the world that the result sequence tells us about, the way Monty’s state of mind is part of the world that generates the exact goat.
To look at it another way, a spy watching Monty open doors and get goats would determine that Monty was deliberately avoiding the prize. Watching a researcher stop at 100 results doesn’t tell you anything about whether the researcher planned to stop at 100 or after getting a certain number of successes. So, just like that result doesn’t tell you anything about the researcher’s state of mind, knowing about the researcher’s state of mind doesn’t tell you anything about the result.
That makes sense. Thank you.
Suppose that the frequency of cures actually converges at 60%, and each researcher performs his experiment 100 times. Researcher A should end up with about 6000 cures out of 10000, and Researcher B should end up with .7n cures out of n. We would expect in advance, after being told that Researcher B ended up testing 10000 people, that he encountered 7000 cures.
It seems that once we know that Researcher B was not going to stop until the incidence of cures was 70%, we do not learn anything further about the efficacy of the treatment by looking at his data.
What’s wrong about this?
You are proposing to partition the data into two observations, namely “number of trials that were performed” and “the results of those trials”.
The information you have after observing the first part and not the second, does depend on the researcher’s stopping criterion. And the amount of information you gain from the second observation also so depends (since some of it was already known from the first observation in one case and not in the other). But your state of information after both observations does not depend on the stopping criterion.
Also, the most likely outcome of your proposed experiment is not .7n cures for some n. Rather, it’s that Researcher B never stops.
That explanation sounds plausible. Can you say it again in math?
Did you read the chapter linked at the end of the post?
A hopefully intuitive explanation: A spy watching the experiments and using Bayesian methods to make his own conclusions about the results, will not see any different evidence in each case and so will end up with the same probability estimate regardless of which experimenter he watched.
While the second experimenter might be contributing to publication bias by using that method in general, he nonetheless should not have come up with a different result.
It seems worth noting the tension between this and bottom-line reasoning. Could the second experimenter have come up with the desired result no matter what, given infinite time? And if so, is there any further entanglement between his hypothesis and reality?
Why would a spy watching Monty Hall be different?
Amongst other reasons, Monty isn’t the experimenter. I’m really not sure in precisely what way Monty Hall is analogous to these experiments.
Monty Hall is analogous in that we are looking at evidence and trying to make conclusions about likelihoods. It is relevant because the likelihoods are different depending on what was in Monty’s head in the past, after observing the same physical evidence. Monty is not the experimenter; where does that make a difference? Could one reformulate it so that he was? He would be running two different experiments, surely—but then why isn’t that the case for the two researchers?