Use small-scale, limited-term betting markets with play money.
Put the group of people you want to rank relative to each other into a room—without internet access. Everyone starts with 0 points. People are ranked on how many points they have at the end of the test.
Participants make bets (for points) with each other. There’s a time limit for settling those debts; all bets made have to be specified in a way that clearly determines the winner within a fixed period after the end of the test. Of course, bets that can be settled immediately (e.g. on current trivia, history or fiction) are also permissible.
Aside from that, there’s no limits: Any time two participants agree they want to bet against each other, on whatever they specify for however many points they choose, they can register that bet.
For instance, Alice and Bob bet on the temperature as reported by for at 6:00 local time, monday after the test:
Bob will pay Alice 5 points if the temperature is at most 20 degree Celsius
Otherwise, Alice will pay Bob 20 points.
After enough time has passed for all bets to be settled, have a trusted third party determine the winner for each, tally up the points and rank participants by final score.
This game is absolute zero-sum: the only way to earn points is by taking them from another participant. Test runs and outcomes can be published without obviously weakening the idea: If there’s something to be learned from previous rounds, all participants have a chance to learn it.
Studying obsesssively on certain subjects may help you, but only to the point that other participants don’t know you’ve done it: If everyone knows that you are a major Higurashi no Naku Koro ni fan, they’re unlikely to bet against you on that subject—or if they do, they won’t bet very much.
Edit: Thinking about this some more, this kind of test has a failure mode: There’s a strong incentive not to bet against people who are better at tests like this than you, so with sufficient information about the players the entire game may freeze up: For every possible bet, there’s somebody who expects to end up worse off, no bets get made and everyone always walks out with 0 points.
Possible solution: Keep participants anonymous to each other during each test. If nobody knows who they’re playing against, there’s a higher chance they’ll be willing to make some bets.
As an addendum, I think the whole thing could still work pretty well even if everyone is explicitly allowed to use the web (or any other data store) for research.
Bets that can be settled with immediately available information won’t be very useful in that context, of course; but you could still bet on near future events. Speed research would be a valuable skill in this variant. Nevertheless, if you have any significant domain specific knowledge useful for making a short-term prediction, that should give you an advantage over someone speed-researching the topic before deciding if they want to make a specific bet on it against you.
The real problem is that access to the internet (or any nontrivial subset) also allows you to do realtime communication with other humans, so you might convince/hire a master rationalist to offer you advice during the test, which would be an extremely effective way to cheat.
A fairly simple windows application could nearly eliminate the problem of research during the test—if it were timed. Each round being timed would allow little time to bypass the lockdowns that can be imposed through a windows API. Each time the test is given, a new version of the test software would be released Even the fastest hacker would be locked into taking the test!
Use small-scale, limited-term betting markets with play money.
Put the group of people you want to rank relative to each other into a room—without internet access. Everyone starts with 0 points. People are ranked on how many points they have at the end of the test.
Participants make bets (for points) with each other. There’s a time limit for settling those debts; all bets made have to be specified in a way that clearly determines the winner within a fixed period after the end of the test. Of course, bets that can be settled immediately (e.g. on current trivia, history or fiction) are also permissible.
Aside from that, there’s no limits: Any time two participants agree they want to bet against each other, on whatever they specify for however many points they choose, they can register that bet.
For instance, Alice and Bob bet on the temperature as reported by for at 6:00 local time, monday after the test:
Bob will pay Alice 5 points if the temperature is at most 20 degree Celsius
Otherwise, Alice will pay Bob 20 points.
After enough time has passed for all bets to be settled, have a trusted third party determine the winner for each, tally up the points and rank participants by final score.
This game is absolute zero-sum: the only way to earn points is by taking them from another participant. Test runs and outcomes can be published without obviously weakening the idea: If there’s something to be learned from previous rounds, all participants have a chance to learn it.
Studying obsesssively on certain subjects may help you, but only to the point that other participants don’t know you’ve done it: If everyone knows that you are a major Higurashi no Naku Koro ni fan, they’re unlikely to bet against you on that subject—or if they do, they won’t bet very much.
Edit: Thinking about this some more, this kind of test has a failure mode: There’s a strong incentive not to bet against people who are better at tests like this than you, so with sufficient information about the players the entire game may freeze up: For every possible bet, there’s somebody who expects to end up worse off, no bets get made and everyone always walks out with 0 points.
Possible solution: Keep participants anonymous to each other during each test. If nobody knows who they’re playing against, there’s a higher chance they’ll be willing to make some bets.
Good idea. It could work online if there’s enough trust between participants.
As an addendum, I think the whole thing could still work pretty well even if everyone is explicitly allowed to use the web (or any other data store) for research.
Bets that can be settled with immediately available information won’t be very useful in that context, of course; but you could still bet on near future events. Speed research would be a valuable skill in this variant. Nevertheless, if you have any significant domain specific knowledge useful for making a short-term prediction, that should give you an advantage over someone speed-researching the topic before deciding if they want to make a specific bet on it against you.
The real problem is that access to the internet (or any nontrivial subset) also allows you to do realtime communication with other humans, so you might convince/hire a master rationalist to offer you advice during the test, which would be an extremely effective way to cheat.
A fairly simple windows application could nearly eliminate the problem of research during the test—if it were timed. Each round being timed would allow little time to bypass the lockdowns that can be imposed through a windows API. Each time the test is given, a new version of the test software would be released Even the fastest hacker would be locked into taking the test!