I got 8 right. The ones I got wrong were the Great Lakes (upper bound was too small by a factor of 1e8) and the currency in circulation (my range was .75-1.25T, and the right answer was slightly lower than my lower bound).
The problem I have with this is while it’s structured well to punish overconfidence, it’s not structured well to publish underconfidence. You can do better than 95% of respondents (according to the graph the author posted) if you write down -inf for your lower bound and +inf for your upper bound, since all 10 will be within your range (and so your error magnitude is 1, just as if you got 8 right). The fact that you’re twice as bad at estimating as someone who writes down 0 as a lower bound and +inf as a upper bound shows up nowhere.
The best test of estimation ability would probably be: each guess costs you log(upper)-log(lower), and a guess that includes the answer gives you Y points. Y determines the minimum level of knowledge you need to guess, but if you narrow your guess you improve your score. You could then figure out what confidence interval people are using from the width they select, and see how that compares with their reported confidence.
This comment reflects how I realized I could game this… AFTER I took it though. Guess -inf to +inf for 9 of them, and something quite tight for the 10th. Then you’ve got your 90%.… but you don’t really.
I got 8 right. The ones I got wrong were the Great Lakes (upper bound was too small by a factor of 1e8) and the currency in circulation (my range was .75-1.25T, and the right answer was slightly lower than my lower bound).
The problem I have with this is while it’s structured well to punish overconfidence, it’s not structured well to publish underconfidence. You can do better than 95% of respondents (according to the graph the author posted) if you write down -inf for your lower bound and +inf for your upper bound, since all 10 will be within your range (and so your error magnitude is 1, just as if you got 8 right). The fact that you’re twice as bad at estimating as someone who writes down 0 as a lower bound and +inf as a upper bound shows up nowhere.
The best test of estimation ability would probably be: each guess costs you log(upper)-log(lower), and a guess that includes the answer gives you Y points. Y determines the minimum level of knowledge you need to guess, but if you narrow your guess you improve your score. You could then figure out what confidence interval people are using from the width they select, and see how that compares with their reported confidence.
This comment reflects how I realized I could game this… AFTER I took it though. Guess -inf to +inf for 9 of them, and something quite tight for the 10th. Then you’ve got your 90%.… but you don’t really.