Give the students sodium pentothal and ask if they’re one of the top 50% of rationalists in their school. However many out of 200 say ‘no’, that’s the school’s percentage score. Schools scoring over 100% are thrown out for cheating.
A school that reports to each student their class ranking easily games this test. The test could even favor schools that don’t teach students enough to question an arbitrary class rank.
Also, this doesn’t consider the possibility that students can be good rationalists, but don’t interact with enough of the other students to make a good assessment of their relative strengths.
Also, this doesn’t consider the possibility that students can be good rationalists, but don’t interact with enough of the other students to make a good assessment of their relative strengths.
Good rationalists, taken as a group, shouldn’t be systematically optimistic.
Good rationalists, taken as a group, shouldn’t be systematically optimistic.
They should be if they want to win in practice, as opposed to just getting theoretically-correct answers. See, e.g., the studies referenced in Seligman’s “Learned Optimism”, that show optimists consistently out-perform pessimists (i.e., realists) in a wide variety of fields and endeavors.
(Of course, Seligman’s definition of optimism may be different from yours.)
Perhaps we can still test for this systematic optimism, while filtering for the noise I objected to, by instead of asking a “yes” or “no” question, asking for the probability that the student is in the top 50%. Treat the sum of these probabilities as the count of “yes” answers in the original version. Then a rational student should be able to account for his ignorance of other students in his answer.
This is even easier to game: assuming the school has any merit, any individual you ask should have good incentive to simply say “50%” guaranteeing a perfect score. The very first time you used the test it might be okay, but only if nobody knew that the school’s reputation was at stake.
Give the students sodium pentothal and ask if they’re one of the top 50% of rationalists in their school. However many out of 200 say ‘no’, that’s the school’s percentage score. Schools scoring over 100% are thrown out for cheating.
A school that reports to each student their class ranking easily games this test. The test could even favor schools that don’t teach students enough to question an arbitrary class rank.
Also, this doesn’t consider the possibility that students can be good rationalists, but don’t interact with enough of the other students to make a good assessment of their relative strengths.
Good rationalists, taken as a group, shouldn’t be systematically optimistic.
They should be if they want to win in practice, as opposed to just getting theoretically-correct answers. See, e.g., the studies referenced in Seligman’s “Learned Optimism”, that show optimists consistently out-perform pessimists (i.e., realists) in a wide variety of fields and endeavors.
(Of course, Seligman’s definition of optimism may be different from yours.)
Perhaps we can still test for this systematic optimism, while filtering for the noise I objected to, by instead of asking a “yes” or “no” question, asking for the probability that the student is in the top 50%. Treat the sum of these probabilities as the count of “yes” answers in the original version. Then a rational student should be able to account for his ignorance of other students in his answer.
This is even easier to game: assuming the school has any merit, any individual you ask should have good incentive to simply say “50%” guaranteeing a perfect score. The very first time you used the test it might be okay, but only if nobody knew that the school’s reputation was at stake.
haha