As it turns out, it is perfectly rational to inspect evidence that contradicts your beliefs more closely than evidence that confirms them. If evidence goes against your beliefs it is more likely to be fake evidence. As the saying goes, your strength as a rationalist is your ability to detect fabricated evidence.
EDIT: As Ben Elliot points out in this post this argument really only applies if you happen to be a perfect Bayesian (and you aren’t). In real life you’re biased toward confirmatory evidence, and so often you really should check it over disconfirmatory evidence. However, it is worth keeping track of which things you’re doing to compensate for a human bias, and which things you’re doing because they’re optimal for Bayesians.
No. This is an easy mistake to make, but it has bad consequences. Detecting fake evidence is only an instrumental goal, in service to the more important goal of maximizing the accuracy of your beliefs. There is a common trap, where all the evidence you see, on both sides, would fall if you challenged it, but you only challenge the pieces you disagree with; and then you add up the remaining, also-invalid evidence, and conclude that you were right all along. It is theoretically possible to counter this bias directly, by discounting confirmatory evidence that you haven’t taken the time to challenge according to a well-calibrated prior probability that it’s invalid. However, I don’t know if anyone can actually do this in practice, and it seems like it would be very difficult to do without carefully formalizing and writing down everything.
I was worried by my own conclusion, so I built a mathematical model to check it*:
Suppose that there is an urn with 100 balls. You’re 99% sure that there are 99 white balls and 1 black, but there’s a 1% chance that there are 99 black balls and 1 white.
You’re about to be scored on the probability you assign to the correct state of the urn, using a logarithmic scoring method, but before that happens your friend takes a ball from the urn, looks at it, and puts it back. Your friend then tells you what colour it was.
Your prior that your friend would lie is 10%.
Suppose you are given the chance to check the colour of the ball your friend drew. How much are you willing to pay for this knowledge? Will you pay more or less if your friend said that the ball was black?
By my calculations** the expected utility
if your friend said “white”, and you don’t check is −0.013581774
if your friend said “white”, and you check is −0.0037359
if your friend said “black”, and you don’t check is −0.391529169
if your friend said “black”, and you check is −0.155101993
So your you will pay 0.0098 to check if your friend said “white” but 0.2364 if they said “black”.
You try harder to investigate unexpected evidence!
If you’re really sure of your conclusion, my maths is probably wrong somewhere. If you think the model itself is inappropriate, please point out how.
EDIT: Of course, if your friend lies 50% of the time, you care just as much about checking confirmatory evidence as disconfirmatory evidence, but then your friend isn’t really “offering you a fact”.
*And if I wasn’t worried, I wouldn’t have built a mathematical model!
I dig what you’ve done, but as a mathematician, my instinct would be to do calculations like this with variables instead of made-up numbers… that way you can be sure that your results hold In General.
Sure, I just couldn’t be bothered. If you ignore the complicating factor of having black balls in the white urn and vice-versa (i.e. we look at the problem where the urn is 100% white or 100% black). then the maths is easier, and we can quickly see that this always holds whatever variables we put in.
In fact, it also works for many different scoring algorithms. If our scoring algorithm is f(x) then it the proof works whenever xf(x)+(1-x)f(1-x) is increasing on [0.5,1].As far as I can tell this happens all of the time that f(x) is increasing on [0,1].
jimrandomh’s point might have been that you don’t try at all to investigate expected evidence. So you’d pay 0.2364 to check if they said ‘black’, but if your friend said ‘white’ you wouldn’t pay to check anything, you’d scoff at the suggestion.
Your result is important, though. Equal checking is wrong, lopsided checking is wrong. Only checking exactly as much as is required by the mathematics is okay.
As it turns out, it is perfectly rational to inspect evidence that contradicts your beliefs more closely than evidence that confirms them. If evidence goes against your beliefs it is more likely to be fake evidence. As the saying goes, your strength as a rationalist is your ability to detect fabricated evidence.
EDIT: As Ben Elliot points out in this post this argument really only applies if you happen to be a perfect Bayesian (and you aren’t). In real life you’re biased toward confirmatory evidence, and so often you really should check it over disconfirmatory evidence. However, it is worth keeping track of which things you’re doing to compensate for a human bias, and which things you’re doing because they’re optimal for Bayesians.
No. This is an easy mistake to make, but it has bad consequences. Detecting fake evidence is only an instrumental goal, in service to the more important goal of maximizing the accuracy of your beliefs. There is a common trap, where all the evidence you see, on both sides, would fall if you challenged it, but you only challenge the pieces you disagree with; and then you add up the remaining, also-invalid evidence, and conclude that you were right all along. It is theoretically possible to counter this bias directly, by discounting confirmatory evidence that you haven’t taken the time to challenge according to a well-calibrated prior probability that it’s invalid. However, I don’t know if anyone can actually do this in practice, and it seems like it would be very difficult to do without carefully formalizing and writing down everything.
I was worried by my own conclusion, so I built a mathematical model to check it*:
Suppose that there is an urn with 100 balls. You’re 99% sure that there are 99 white balls and 1 black, but there’s a 1% chance that there are 99 black balls and 1 white.
You’re about to be scored on the probability you assign to the correct state of the urn, using a logarithmic scoring method, but before that happens your friend takes a ball from the urn, looks at it, and puts it back. Your friend then tells you what colour it was.
Your prior that your friend would lie is 10%.
Suppose you are given the chance to check the colour of the ball your friend drew. How much are you willing to pay for this knowledge? Will you pay more or less if your friend said that the ball was black?
By my calculations** the expected utility
if your friend said “white”, and you don’t check is −0.013581774
if your friend said “white”, and you check is −0.0037359
if your friend said “black”, and you don’t check is −0.391529169
if your friend said “black”, and you check is −0.155101993
So your you will pay 0.0098 to check if your friend said “white” but 0.2364 if they said “black”.
You try harder to investigate unexpected evidence!
If you’re really sure of your conclusion, my maths is probably wrong somewhere. If you think the model itself is inappropriate, please point out how.
EDIT: Of course, if your friend lies 50% of the time, you care just as much about checking confirmatory evidence as disconfirmatory evidence, but then your friend isn’t really “offering you a fact”.
*And if I wasn’t worried, I wouldn’t have built a mathematical model!
**Using base 2 logs.
I dig what you’ve done, but as a mathematician, my instinct would be to do calculations like this with variables instead of made-up numbers… that way you can be sure that your results hold In General.
Sure, I just couldn’t be bothered. If you ignore the complicating factor of having black balls in the white urn and vice-versa (i.e. we look at the problem where the urn is 100% white or 100% black). then the maths is easier, and we can quickly see that this always holds whatever variables we put in.
In fact, it also works for many different scoring algorithms. If our scoring algorithm is f(x) then it the proof works whenever xf(x)+(1-x)f(1-x) is increasing on [0.5,1].As far as I can tell this happens all of the time that f(x) is increasing on [0,1].
jimrandomh’s point might have been that you don’t try at all to investigate expected evidence. So you’d pay 0.2364 to check if they said ‘black’, but if your friend said ‘white’ you wouldn’t pay to check anything, you’d scoff at the suggestion.
Your result is important, though. Equal checking is wrong, lopsided checking is wrong. Only checking exactly as much as is required by the mathematics is okay.