A year back, I encountered a this kind of a test: binary multiple choice, one point for right answer, minus half a point for a wrong answer, zero points for no answer. (Multiple-choice exams of any kind are very rare in Finnish universities, so that’s pretty much the only time in my life when I’ve been faced with a test like that.) Looking at the scoring, I came to the same conclusion as you: my expected score would be higher if I’d just try guessing each of the questions I wasn’t sure on.
I didn’t follow my own advice. I now wish I had, as I failed that exam. I was under a pretty heavy workload at the time, so I never ended up retaking it. I suspect I’d have passed if I’d just shut up and multiplied.
Why didn’t I follow my own advice? I did have some kind of a conscious reason, but in retrospect it seems so flimsy that I have difficulty even formulating it here. It went something along the lines of “I might as well take all the questions I have absolutely no clue on and mark them all as ‘true’, which gives me a 50-50 chance to be right on each one assuming there are as many true as there are false questions. But what if the lecturer, forseeing that somebody would reason this way, wrote the questions in such a way that one alternative is more frequently correct than the other, and there isn’t a 50-50 chance for all questions to be ‘true’? Then my expected return calculation would be off, possibly costing me points!”
Yes, I’m aware of all the flaws in that line of thought, no need to point them out. I really didn’t think it through properly. That implies that the very thing you suggest happened to your friends, happened to me—I instinctively disliked the idea, and then rationalized myself a (bad) reason not to do it.
TANSTAAFL: There Ain’t No Such Thing As A Free Lunch
That’s still a better justifcation of your behaviour than the MIT students [edit: Yvain did not actually say MIT!] used—not to mention that you’re able to look back in retrospect and acknowledge the error of your decision.
This sort of suspicion is a good heuristic, if not the best heuristic. Scam artists (by which I mean casinos and carnies) are skilled at making things appear as if you’ve found the loophole in their game, and when you don’t have enough time to examine the loophole thoroughly you’re generally better off assuming it to be false. From the sound of things you were too busy to do this, not to mention that—being unfamiliar with multiple choice tests in general—it caught you with your pants down. You would have had to devote twice the analysis time as a typical North American, who is familiar with these sorts of exams.
Don’t discount the TANSTAAFL heuristic—you made a rational choice based on limited data and available processing time. Your error is wholly different from the errors at MIT.
Ooh, this is interesting. Eliezer says he hopes this wasn’t at MIT or somewhere, and now people are remembering the MIT reference and assuming I go to MIT. Reminds me of that bias where you try to debunk a rumor, and all people can remember is that they heard someone talking about the rumor somewhere and believe it more. What’s that called? There was an OB article on it somewhere, I think.
I should hire Eliezer to come by and make offhanded MIT references during my job interviews.
Dear lord, I just pooped myself. I’m thoroughly familiar with this bias—and I just fell into it.
Isn’t this sort of language manipulation exactly what the PUAs do? Hmm… a potential way of strengthening one’s arguments occurs to me. While in conversation with somebody IRL it should be more effective to phrase things as “Well, Eliezer said...” than “According to Eliezer’s article on...” so as to give the impression of possible first hand knowledge, or at least thorough familiarity with the relevant material.
This is a dark art no doubt, but with most people I find that this is the only way of dealing with them.
(I am not above name-dropping Eliezer to pick up chicks.)
I sincerely doubt that namedropping Eliezer is an effective chick-picking-up technique except under very unusual circumstances! OTOH I have pulled at least once by talking about cryptography, so you never know :-)
A year back, I encountered a this kind of a test: binary multiple choice, one point for right answer, minus half a point for a wrong answer, zero points for no answer. (Multiple-choice exams of any kind are very rare in Finnish universities, so that’s pretty much the only time in my life when I’ve been faced with a test like that.) Looking at the scoring, I came to the same conclusion as you: my expected score would be higher if I’d just try guessing each of the questions I wasn’t sure on.
I didn’t follow my own advice. I now wish I had, as I failed that exam. I was under a pretty heavy workload at the time, so I never ended up retaking it. I suspect I’d have passed if I’d just shut up and multiplied.
Why didn’t I follow my own advice? I did have some kind of a conscious reason, but in retrospect it seems so flimsy that I have difficulty even formulating it here. It went something along the lines of “I might as well take all the questions I have absolutely no clue on and mark them all as ‘true’, which gives me a 50-50 chance to be right on each one assuming there are as many true as there are false questions. But what if the lecturer, forseeing that somebody would reason this way, wrote the questions in such a way that one alternative is more frequently correct than the other, and there isn’t a 50-50 chance for all questions to be ‘true’? Then my expected return calculation would be off, possibly costing me points!”
Yes, I’m aware of all the flaws in that line of thought, no need to point them out. I really didn’t think it through properly. That implies that the very thing you suggest happened to your friends, happened to me—I instinctively disliked the idea, and then rationalized myself a (bad) reason not to do it.
TANSTAAFL: There Ain’t No Such Thing As A Free Lunch
That’s still a better justifcation of your behaviour than the MIT students [edit: Yvain did not actually say MIT!] used—not to mention that you’re able to look back in retrospect and acknowledge the error of your decision.
This sort of suspicion is a good heuristic, if not the best heuristic. Scam artists (by which I mean casinos and carnies) are skilled at making things appear as if you’ve found the loophole in their game, and when you don’t have enough time to examine the loophole thoroughly you’re generally better off assuming it to be false. From the sound of things you were too busy to do this, not to mention that—being unfamiliar with multiple choice tests in general—it caught you with your pants down. You would have had to devote twice the analysis time as a typical North American, who is familiar with these sorts of exams.
Don’t discount the TANSTAAFL heuristic—you made a rational choice based on limited data and available processing time. Your error is wholly different from the errors at MIT.
Ooh, this is interesting. Eliezer says he hopes this wasn’t at MIT or somewhere, and now people are remembering the MIT reference and assuming I go to MIT. Reminds me of that bias where you try to debunk a rumor, and all people can remember is that they heard someone talking about the rumor somewhere and believe it more. What’s that called? There was an OB article on it somewhere, I think.
I should hire Eliezer to come by and make offhanded MIT references during my job interviews.
Dear lord, I just pooped myself. I’m thoroughly familiar with this bias—and I just fell into it.
Isn’t this sort of language manipulation exactly what the PUAs do? Hmm… a potential way of strengthening one’s arguments occurs to me. While in conversation with somebody IRL it should be more effective to phrase things as “Well, Eliezer said...” than “According to Eliezer’s article on...” so as to give the impression of possible first hand knowledge, or at least thorough familiarity with the relevant material.
This is a dark art no doubt, but with most people I find that this is the only way of dealing with them.
(I am not above name-dropping Eliezer to pick up chicks.)
I sincerely doubt that namedropping Eliezer is an effective chick-picking-up technique except under very unusual circumstances! OTOH I have pulled at least once by talking about cryptography, so you never know :-)
That’s an interesting point. You may be right, now that I think of it that way.
Easy way out: Flip a coin for each answer.
Obviously the author of the test cannot possibly know how your coin falls, so you get a true probabilistic chance for each answer.
This is basically the same idea as for randomized quicksort: To guard against malicious data, make your algorithm unpredictable.