For some reason, TrollBot always wins 500 / 0 against SmarterMirrorBot. DefectBot actually beats TrollBot by a narrow margin (1604 − 1575 = 29 points) on average, but there is quite a bit of randomness from RandomBot, so TrollBot often comes out ahead of even DefectBot, and they both come out over 100 points ahead of the next bot (TitForTatBot).
Since I built TrollBot as a sanity check on my actual bot to make sure that it would defect against TrollBot, I was definitely surprised by the fact that TrollBot outperformed not only my attempt at a serious bot, but also most of the other bots… :/
Oh! You’re also running your opponent playing a game against MirrorBot, not against TrollBot.
Which I still don’t understand… you run SMB versus MB, time limit 10000. SMB runs MB versus MB, time limit 10000. MB versus MB times out, which meas SMB runs for > 10000 us, which means that SMB should time out and you should cooperate.
Meanwhile, SMB runs you versus MB, time limit 10000. You run MB versus MB, time limit 10000. That times out, so you time out, so SMB cooperates.
But you’re defecting, which means that you’re running SMB versus MB to completion, which seems like it shouldn’t happen.
However, tetronian2′s “time” command catches all exceptions, which means that if multiple timeouts are nested, you can get some pretty strange behaviour.
In the TB vs SMB scenario, here’s what happens:
The world runs TB vs SMB
TB runs SMB vs MB (timeout #1, 10000us)
SMB runs MB vs MB (timeout #2, 10000us)
Timeout #1 triggers an exception, which is caught by timeout #2 and handled, causing (3) to return Nothing.
Before the exception for timeout #2 can actually trigger, (2) finishes normally and kills the timeout thread; this means that (2) return Just Cooperate.
Since (2) returned Just Cooperate, TrollBot returns Defect.
Thank you very much for catching this and for explaining it in detail here and on the github page; I’m going to play around with your proposed solution today and brainstorm any other ways of dealing with this.
Whoops. Effect goes away if I fix it, too.
Here are the average results for the first round:
http://pastebin.com/qN5H2A25
For some reason, TrollBot always wins 500 / 0 against SmarterMirrorBot. DefectBot actually beats TrollBot by a narrow margin (1604 − 1575 = 29 points) on average, but there is quite a bit of randomness from RandomBot, so TrollBot often comes out ahead of even DefectBot, and they both come out over 100 points ahead of the next bot (TitForTatBot).
Since I built TrollBot as a sanity check on my actual bot to make sure that it would defect against TrollBot, I was definitely surprised by the fact that TrollBot outperformed not only my attempt at a serious bot, but also most of the other bots… :/
Oh! You’re also running your opponent playing a game against MirrorBot, not against TrollBot.
Which I still don’t understand… you run SMB versus MB, time limit 10000. SMB runs MB versus MB, time limit 10000. MB versus MB times out, which meas SMB runs for > 10000 us, which means that SMB should time out and you should cooperate.
Meanwhile, SMB runs you versus MB, time limit 10000. You run MB versus MB, time limit 10000. That times out, so you time out, so SMB cooperates.
But you’re defecting, which means that you’re running SMB versus MB to completion, which seems like it shouldn’t happen.
That does seem exploitable, if one can figure out exactly what’s happening here.
Yes, it’s an issue with how the “time” command is set up. Basically, timeouts are done via threads an exceptions; there’s a detailed description of the “timeout” function in System.Timeout here: http://chimera.labs.oreilly.com/books/1230000000929/ch09.html#sec_timeout
However, tetronian2′s “time” command catches all exceptions, which means that if multiple timeouts are nested, you can get some pretty strange behaviour.
In the TB vs SMB scenario, here’s what happens:
The world runs TB vs SMB
TB runs SMB vs MB (timeout #1, 10000us)
SMB runs MB vs MB (timeout #2, 10000us)
Timeout #1 triggers an exception, which is caught by timeout #2 and handled, causing (3) to return Nothing.
Before the exception for timeout #2 can actually trigger, (2) finishes normally and kills the timeout thread; this means that (2) return Just Cooperate.
Since (2) returned Just Cooperate, TrollBot returns Defect.
I’ve brought this up as an issue on the GitHub issues page here: https://github.com/pdtournament/pdtournament/issues/3
I’ve suggested a solution that fixes this problem, but unfortunately race conditions are ultimately still present.
Thank you very much for catching this and for explaining it in detail here and on the github page; I’m going to play around with your proposed solution today and brainstorm any other ways of dealing with this.