This question was easy for me to answer by pattern-matching to the Law of Small Numbers, as outlined in Thinking, Fast and Slow. If I hadn’t read that, it’s hard to say whether I would have reasoned it out correctly. So if many respondents answer this question correctly, I hope that the survey authors don’t claim evidence that LW readers are better at statistical reasoning—it’d be more accurate to say that LW readers are more likely to have seen this very particular question before.
I don’t understand the distinction you are making here. If you can answer correctly more statistical questions, how is that not being ‘better at statistical reasoning’? Every area of thought draws heavily on memorization and caching.
If you can answer correctly more statistical questions, how is that not being ‘better at statistical reasoning’?
Those are related abilities, but there’s being able to answer specific questions and then there’s being able to apply what you’ve learned more generally. For me, this particular question triggered more “aha! I’ve seen this one before!” than it triggered statistical thought. A correct answer to the question might give you a smidgen of information on whether the answerer can reason about statistics, but it probably gives you a lot more information about whether the answerer has seen the question before.
One superficial example of dealing with this problem is how, in my college discrete math class, the professor gave us a problem involving placing pigeons in holes, with the solution having nothing to do with the pigeonhole principle. Even better than obfuscating a problem, of course, is stating a novel one that exercises the skills you’re testing for.
I don’t understand the distinction you are making here. If you can answer correctly more statistical questions, how is that not being ‘better at statistical reasoning’? Every area of thought draws heavily on memorization and caching.
Those are related abilities, but there’s being able to answer specific questions and then there’s being able to apply what you’ve learned more generally. For me, this particular question triggered more “aha! I’ve seen this one before!” than it triggered statistical thought. A correct answer to the question might give you a smidgen of information on whether the answerer can reason about statistics, but it probably gives you a lot more information about whether the answerer has seen the question before.
One superficial example of dealing with this problem is how, in my college discrete math class, the professor gave us a problem involving placing pigeons in holes, with the solution having nothing to do with the pigeonhole principle. Even better than obfuscating a problem, of course, is stating a novel one that exercises the skills you’re testing for.