I was worried by my own conclusion, so I built a mathematical model to check it*:
Suppose that there is an urn with 100 balls. You’re 99% sure that there are 99 white balls and 1 black, but there’s a 1% chance that there are 99 black balls and 1 white.
You’re about to be scored on the probability you assign to the correct state of the urn, using a logarithmic scoring method, but before that happens your friend takes a ball from the urn, looks at it, and puts it back. Your friend then tells you what colour it was.
Your prior that your friend would lie is 10%.
Suppose you are given the chance to check the colour of the ball your friend drew. How much are you willing to pay for this knowledge? Will you pay more or less if your friend said that the ball was black?
By my calculations** the expected utility
if your friend said “white”, and you don’t check is −0.013581774
if your friend said “white”, and you check is −0.0037359
if your friend said “black”, and you don’t check is −0.391529169
if your friend said “black”, and you check is −0.155101993
So your you will pay 0.0098 to check if your friend said “white” but 0.2364 if they said “black”.
You try harder to investigate unexpected evidence!
If you’re really sure of your conclusion, my maths is probably wrong somewhere. If you think the model itself is inappropriate, please point out how.
EDIT: Of course, if your friend lies 50% of the time, you care just as much about checking confirmatory evidence as disconfirmatory evidence, but then your friend isn’t really “offering you a fact”.
*And if I wasn’t worried, I wouldn’t have built a mathematical model!
I dig what you’ve done, but as a mathematician, my instinct would be to do calculations like this with variables instead of made-up numbers… that way you can be sure that your results hold In General.
Sure, I just couldn’t be bothered. If you ignore the complicating factor of having black balls in the white urn and vice-versa (i.e. we look at the problem where the urn is 100% white or 100% black). then the maths is easier, and we can quickly see that this always holds whatever variables we put in.
In fact, it also works for many different scoring algorithms. If our scoring algorithm is f(x) then it the proof works whenever xf(x)+(1-x)f(1-x) is increasing on [0.5,1].As far as I can tell this happens all of the time that f(x) is increasing on [0,1].
jimrandomh’s point might have been that you don’t try at all to investigate expected evidence. So you’d pay 0.2364 to check if they said ‘black’, but if your friend said ‘white’ you wouldn’t pay to check anything, you’d scoff at the suggestion.
Your result is important, though. Equal checking is wrong, lopsided checking is wrong. Only checking exactly as much as is required by the mathematics is okay.
I was worried by my own conclusion, so I built a mathematical model to check it*:
Suppose that there is an urn with 100 balls. You’re 99% sure that there are 99 white balls and 1 black, but there’s a 1% chance that there are 99 black balls and 1 white.
You’re about to be scored on the probability you assign to the correct state of the urn, using a logarithmic scoring method, but before that happens your friend takes a ball from the urn, looks at it, and puts it back. Your friend then tells you what colour it was.
Your prior that your friend would lie is 10%.
Suppose you are given the chance to check the colour of the ball your friend drew. How much are you willing to pay for this knowledge? Will you pay more or less if your friend said that the ball was black?
By my calculations** the expected utility
if your friend said “white”, and you don’t check is −0.013581774
if your friend said “white”, and you check is −0.0037359
if your friend said “black”, and you don’t check is −0.391529169
if your friend said “black”, and you check is −0.155101993
So your you will pay 0.0098 to check if your friend said “white” but 0.2364 if they said “black”.
You try harder to investigate unexpected evidence!
If you’re really sure of your conclusion, my maths is probably wrong somewhere. If you think the model itself is inappropriate, please point out how.
EDIT: Of course, if your friend lies 50% of the time, you care just as much about checking confirmatory evidence as disconfirmatory evidence, but then your friend isn’t really “offering you a fact”.
*And if I wasn’t worried, I wouldn’t have built a mathematical model!
**Using base 2 logs.
I dig what you’ve done, but as a mathematician, my instinct would be to do calculations like this with variables instead of made-up numbers… that way you can be sure that your results hold In General.
Sure, I just couldn’t be bothered. If you ignore the complicating factor of having black balls in the white urn and vice-versa (i.e. we look at the problem where the urn is 100% white or 100% black). then the maths is easier, and we can quickly see that this always holds whatever variables we put in.
In fact, it also works for many different scoring algorithms. If our scoring algorithm is f(x) then it the proof works whenever xf(x)+(1-x)f(1-x) is increasing on [0.5,1].As far as I can tell this happens all of the time that f(x) is increasing on [0,1].
jimrandomh’s point might have been that you don’t try at all to investigate expected evidence. So you’d pay 0.2364 to check if they said ‘black’, but if your friend said ‘white’ you wouldn’t pay to check anything, you’d scoff at the suggestion.
Your result is important, though. Equal checking is wrong, lopsided checking is wrong. Only checking exactly as much as is required by the mathematics is okay.