I get the feeling that the real problem here is repeatability. It’s one thing to design a test for rationality, it’s another to design a test that could not be gamed once the particulars are known. Since it probably isn’t possible to control the flow of information in that way, the next-best option might be to design a test so that the testing criteria would not be understood except by those who pass.
I’m thinking of a test I heard about years ago. The teacher passes out the test, stressing to the students to read the instructions before beginning. The instructions specify that the answer to every question is C. The actual questions on the test don’t matter, of course, but it’s a great test of reading comprehension and the ability to follow instructions. Plus, the test is completely repeatable. All of the test questions could leak out, and still only those who deserve to pass would do so. If you are willing to assume that people who pass would not be willing to cheat (unlikely in this test, possible in a rationality test), then you would have an ungameable test.
A rationality test in this model might be one where an impossible task is given, and the correct response would be to not play.
I don’t think that it’s reasonable to expect that secret criteria would stay secret once such a test would actually be used for anything. Sure, it could be kept a secret if there were a dozen people taking the test, of which the four who passed would get admitted to an exclusive club.
If there were ten thousand people taking the test, a thousand of which passed, I’d bet there’d be at least one who accidentally leaks it on the internet, from where it would immediately become public knowledge. (And at least a dozen who would willingly give up the answer if offered money for it, as would happen if there were anything at stake in this test.) It might work if such a test is obscure enough or not widely used, but not if it was used for anything that mattered to the test-takers and was open to many.
True, but I think that would be a problem with any test. I’m just trying to find a way around it since I think that as you add ways to avoid gaming, you both complicate and weaken the test. Perhaps a solution would be to test people without their knowledge, and reveal whether they succeeded or not at a later date.
“Psssst, when Mrs. P says to read the instructions, it’s because it’s a fake test! If you just follow the directions you can get an A without even trying!”
And that is that test ruined for all subsequent classes. People may not read instructions, but they will generally listen to peers highlighting that there is something unusual, or some easy way of cheating. Heck, it might become a weird group wisdom to always answer “C” because the answer scanning machine is broken or something. I’ve seen weirder in actual work places.
I get the feeling that the real problem here is repeatability. It’s one thing to design a test for rationality, it’s another to design a test that could not be gamed once the particulars are known. Since it probably isn’t possible to control the flow of information in that way, the next-best option might be to design a test so that the testing criteria would not be understood except by those who pass.
I’m thinking of a test I heard about years ago. The teacher passes out the test, stressing to the students to read the instructions before beginning. The instructions specify that the answer to every question is C. The actual questions on the test don’t matter, of course, but it’s a great test of reading comprehension and the ability to follow instructions. Plus, the test is completely repeatable. All of the test questions could leak out, and still only those who deserve to pass would do so. If you are willing to assume that people who pass would not be willing to cheat (unlikely in this test, possible in a rationality test), then you would have an ungameable test.
A rationality test in this model might be one where an impossible task is given, and the correct response would be to not play.
I don’t think that it’s reasonable to expect that secret criteria would stay secret once such a test would actually be used for anything. Sure, it could be kept a secret if there were a dozen people taking the test, of which the four who passed would get admitted to an exclusive club.
If there were ten thousand people taking the test, a thousand of which passed, I’d bet there’d be at least one who accidentally leaks it on the internet, from where it would immediately become public knowledge. (And at least a dozen who would willingly give up the answer if offered money for it, as would happen if there were anything at stake in this test.) It might work if such a test is obscure enough or not widely used, but not if it was used for anything that mattered to the test-takers and was open to many.
True, but I think that would be a problem with any test. I’m just trying to find a way around it since I think that as you add ways to avoid gaming, you both complicate and weaken the test. Perhaps a solution would be to test people without their knowledge, and reveal whether they succeeded or not at a later date.
Kobayashi Maru?
Global Thermonuclear War?
Well, only because the computer’s search tree didn’t include the “teleport giant psychic squid” action ;)
(spoilers behind link)
Thank you for making my day :)
^_^
“Psssst, when Mrs. P says to read the instructions, it’s because it’s a fake test! If you just follow the directions you can get an A without even trying!”
And that is that test ruined for all subsequent classes. People may not read instructions, but they will generally listen to peers highlighting that there is something unusual, or some easy way of cheating. Heck, it might become a weird group wisdom to always answer “C” because the answer scanning machine is broken or something. I’ve seen weirder in actual work places.
Isn’t that more a test of attention to detail and willingness to follow instructions rather than rationality per se?
Yes. I wasn’t offering that particular formulation as a rationality test, just the idea that you should hide from the testee the nature of the test.