It may also be worth mentioning that the experimental method is also “last line first”. That is, at the start you state the hypothesis that you’re about to test, and then you test the hypothesis—which test, depending on the result, may amount to an argument from evidence for the hypothesis.
No. When you state the hypothesis, it means that, depending on the evidence you are about to gather, your bottom line will be that the hypothesis is true or that the hypothesis is false (or that you can’t tell if the hypothesis is true or false). Writing the Bottom Line First would be deciding in advance to conclude that the hypothesis is true.
Deciding in advance to conclude that the hypothesis is true is not a danger if the way you decide to do that is by some means that in reality won’t let you do that if the hypothesis is false. Keep in mind: you can decide to do something and still be unable to do it.
Suppose I believe that a hypothesis is true. I believe it so strongly, that I believe a well-designed experiment will prove that it is true. So I decide in advance to conclude that the hypothesis is true by doing what I am positive in advance will prove the hypothesis, which is to run a well-designed experiment which will convince the doubters. So I do that, and (suppose) that the experiment supports my hypothesis. The fact that my intentions were to prove the hypothesis don’t invalidate the result of the experiment. The experiment is by its own good design protected from my intentions.
A well-designed experiment will yield truth whatever the intentions of the experimenter. What makes an experiment good isn’t good intentions on the part of the experimenter. That’s the whole point of the experiment: we can’t trust the experimenter, and so the experiment by design renders the experimenter powerless. (Of course, we can increase our confidence even further by replicating the experiment.)
Now let’s change both the intention and the method. Suppose you don’t know whether a hypothesis is true and decide to discover whether it is true by examining the evidence. The method you choose is “preponderance of evidence”. It is quite possible for you completely erroneously and unintentionally to in effect cherry-pick evidence for the hypothesis you were trying to test. People make procedural mistakes like this all the time without intending to do so. For example, you see one bit of evidence, and make note of the fact that this particular bit of evidence makes the the hypothesis appear to be true. But now, uh oh! You’re subject to confirmation bias! That means that you will automatically, without meaning to, start to pay attention to confirming and ignore disconfirming evidence. And you didn’t mean to!
Depending on where the hypothesis came from, the experimental method may be Privileging the Hypothesis
Absolutely, but privileging the hypothesis is a danger whether or not you have decided in advance to conclude the hypothesis. Look at Eliezer’s own description:
Then, one of the detectives says, “Well… we have no idea who did it… no particular evidence singling out any of the million people in this city… but let’s consider the hypothesis that this murder was committed by Mortimer Q. Snodgrass, who lives at 128 Ordinary Ln. It could have been him, after all.”
This detective has, importantly, not decided in advance to conclude that Snodgrass is the murderer.
No. When you state the hypothesis, it means that, depending on the evidence you are about to gather, your bottom line will be that the hypothesis is true or that the hypothesis is false (or that you can’t tell if the hypothesis is true or false). Writing the Bottom Line First would be deciding in advance to conclude that the hypothesis is true.
Depending on where the hypothesis came from, the experimental method may be Privileging the Hypothesis, which the social process of science compensates for by requiring lots of evidence.
Deciding in advance to conclude that the hypothesis is true is not a danger if the way you decide to do that is by some means that in reality won’t let you do that if the hypothesis is false. Keep in mind: you can decide to do something and still be unable to do it.
Suppose I believe that a hypothesis is true. I believe it so strongly, that I believe a well-designed experiment will prove that it is true. So I decide in advance to conclude that the hypothesis is true by doing what I am positive in advance will prove the hypothesis, which is to run a well-designed experiment which will convince the doubters. So I do that, and (suppose) that the experiment supports my hypothesis. The fact that my intentions were to prove the hypothesis don’t invalidate the result of the experiment. The experiment is by its own good design protected from my intentions.
A well-designed experiment will yield truth whatever the intentions of the experimenter. What makes an experiment good isn’t good intentions on the part of the experimenter. That’s the whole point of the experiment: we can’t trust the experimenter, and so the experiment by design renders the experimenter powerless. (Of course, we can increase our confidence even further by replicating the experiment.)
Now let’s change both the intention and the method. Suppose you don’t know whether a hypothesis is true and decide to discover whether it is true by examining the evidence. The method you choose is “preponderance of evidence”. It is quite possible for you completely erroneously and unintentionally to in effect cherry-pick evidence for the hypothesis you were trying to test. People make procedural mistakes like this all the time without intending to do so. For example, you see one bit of evidence, and make note of the fact that this particular bit of evidence makes the the hypothesis appear to be true. But now, uh oh! You’re subject to confirmation bias! That means that you will automatically, without meaning to, start to pay attention to confirming and ignore disconfirming evidence. And you didn’t mean to!
Absolutely, but privileging the hypothesis is a danger whether or not you have decided in advance to conclude the hypothesis. Look at Eliezer’s own description:
This detective has, importantly, not decided in advance to conclude that Snodgrass is the murderer.