I suggest that the students aren’t as irrational as they appear. After all, why would the designer of the test incorporate a “don’t know” option and a penalty for wrong answers, except to discourage guessing on questions that you’re clueless on? And if I were a random student (instead of someone especially interested in the mathematics of decision theory), why should I take the trouble to second guess the test designer, instead of assuming that (with high probability) he is rational and competent at his job?
ETA: Also, you’re supposed to maximize expected utility, not expected number of points. Increasing the variance of your score may decrease expected utility, even if it keeps the expected score the same. (I see that John Maxwell IV has made a similar point.)
I think students are happy to take advantage of amateur test design, like when one question’s back story reveals the answer to a different question. So this post is a good demonstration of how not being a rationalist can make you a sucker.
EDIT: Please ignore this comment, which was based on a misreading of Yvain’s post. I failed to notice that a wrong answer gives minus one half point, not minus one point.
But taking advantage of this test design is trickier than it looks. The best strategy is not necessarily “always guess”. (Actually it almost certainly isn’t “always guess”.)
For example, suppose that your expected score is above 50%, and you don’t care much about getting honors but really need to pass. Then you shouldn’t guess if you have no idea what the answer is, since guessing increases the probability that your score will fall below 50% by bad luck.
Here’s another example. Suppose you’re certain about 71% of the answers, and are slightly unsure about the rest. Then you should answer “don’t know” for all of the questions that you’re slightly unsure about, since there is no additional utility for getting more points above 70%, and by guessing you’re just decreasing the probability of getting “high honors” for no possibility of gain.
The students who refused to guess may actually have behaved rationally (even if they can’t articulate why). I think this story illustrates the dangers of overriding intuition with partial knowledge.
Edit: In sum, since there will be at least 30ish questions unknown, then losing any points at all by guessing is unlikely enough that you’d need to be quite unusually well calibrated to justify not-guessing to raise e.g. probability of getting over 70% given that you know (99% confidence) you’ve already got exactly 71%.
Interesting. I edited my comment to summarize it greatly, since I no longer think I’m trying to convince someone who is wildly wrong on the probability. :) But now my edit’s reverted.
What if UDT instances were taking the test? They should be able to conclude that everyone ever opting to exploit the rule is a net disadvantage (the passing score would, of course, have to be raised to compensate for the gambling points).
I suggest that the students aren’t as irrational as they appear. After all, why would the designer of the test incorporate a “don’t know” option and a penalty for wrong answers, except to discourage guessing on questions that you’re clueless on? And if I were a random student (instead of someone especially interested in the mathematics of decision theory), why should I take the trouble to second guess the test designer, instead of assuming that (with high probability) he is rational and competent at his job?
ETA: Also, you’re supposed to maximize expected utility, not expected number of points. Increasing the variance of your score may decrease expected utility, even if it keeps the expected score the same. (I see that John Maxwell IV has made a similar point.)
I think students are happy to take advantage of amateur test design, like when one question’s back story reveals the answer to a different question. So this post is a good demonstration of how not being a rationalist can make you a sucker.
EDIT: Please ignore this comment, which was based on a misreading of Yvain’s post. I failed to notice that a wrong answer gives minus one half point, not minus one point.
But taking advantage of this test design is trickier than it looks. The best strategy is not necessarily “always guess”. (Actually it almost certainly isn’t “always guess”.)
For example, suppose that your expected score is above 50%, and you don’t care much about getting honors but really need to pass. Then you shouldn’t guess if you have no idea what the answer is, since guessing increases the probability that your score will fall below 50% by bad luck.
Here’s another example. Suppose you’re certain about 71% of the answers, and are slightly unsure about the rest. Then you should answer “don’t know” for all of the questions that you’re slightly unsure about, since there is no additional utility for getting more points above 70%, and by guessing you’re just decreasing the probability of getting “high honors” for no possibility of gain.
The students who refused to guess may actually have behaved rationally (even if they can’t articulate why). I think this story illustrates the dangers of overriding intuition with partial knowledge.
Edit: In sum, since there will be at least 30ish questions unknown, then losing any points at all by guessing is unlikely enough that you’d need to be quite unusually well calibrated to justify not-guessing to raise e.g. probability of getting over 70% given that you know (99% confidence) you’ve already got exactly 71%.
You’re right. I’ve edited my comment with the cause of my error.
Interesting. I edited my comment to summarize it greatly, since I no longer think I’m trying to convince someone who is wildly wrong on the probability. :) But now my edit’s reverted.
What if UDT instances were taking the test? They should be able to conclude that everyone ever opting to exploit the rule is a net disadvantage (the passing score would, of course, have to be raised to compensate for the gambling points).
When I take a true-false test, I second-guess the author on every question.