Can I suggest running a second tournement, including all these bots and new bots that we can examine these bots in writing? After a few iterations, we might beat random. Or we might develop all sorts of signaling quirks, but that would also be interesting.
When I first announced the tournament, people came up with many suggestions for modifications to the rules, most of which I did not take for the sake of simplicity. Maybe we should try to figure out what rules will make it easier to write more effective bots, even if that would disqualify many of the bots already submitted to this tournament.
Am I the only one who sees a problem in that we’re turning a non-zero-sum game into a winner-take-all tournament? Perhaps instead of awarding a limited resource like bitcoins to the “winner”, each player should be awarded an unlimited resource such as karma or funny cat pictures according to their strategy’s performance.
Maybe we should have a prisoner’s dilemma meta-tournament in which we define a tournament-goodness metric and then everyone submits a tournament design that is run using the bot participants in the previous tournament, and then use the rankings of those designs.
Wait: does anyone know a good way to design the meta-tournament?
Conduct a statistically significant number of trials. With few trial, the highest ranked agents will be the ones with a high variance in possible score, rather than the agents with the highest average score.
Allow no libraries, except for a very small whitelist (e.g. random numbers).
No timing, no threads. They make analysis too hard. The payoff might have to be changed to make it never in your best interest to make your opponent diverge, e.g. (-1, −1).
The rule “This is about Prisoner’s dilemma, not the language you’re working in. If you violate the spirit of this rule, your agent will be disqualified.”
(Possible) Change language to a restricted dialect of Python. You can analyze your opponent bytecode, if you wish, or run your opponent as a black-box. Programs with side effects (such as writing to global variables) and programs that use Pythonic trickiness (such as inspecting the stack) are verboten.
Most of these suggestions are okay, but don’t move away from Lisp. A very restricted Scheme (e.g. one way to set variables, one looping construct (I really like foldl, as N shows), etc.) would be good; the lexical scoping and general immutability make it one of the best Lisps for our purposes.
+1 for restricted Scheme. I suggest restricted Python because it is more accessible and would allow more people to participate, but personally I like Scheme a bit more.
On the other hand, many many people were turned off by Scheme last time, and few of them will be reading this post. I’d like to give them a shot, if only to get a more diverse crowd.
On looping constructs: Would that buy us anything? What if people use recursion for looping? Should we restrict recursion?
Question for the audience: would there be a benefit to using a programming language that allows only primitive recursive functions?
I think a programming language that only allows primitive recursion is a bad idea. One common pattern (which I think we want to allow) was for bots to simulate their opponents, which entails the ability to simulate arbitrary valid code. This would not be possible in a language which restricts to primitive recursion.
For entries that only eval(opponent(x)), for simple x, that seems reasonable. Strategies that rely on deeper analysis would need to look for specific things about the code that I, as a non-programmer, cannot describe accurately.
I think that the bots already submitted should be considered for grandfathering if the rules change to prohibit them.
I would agree that they should compete even if they’re not awarded prizes for winning. Because we should at least have a comparison with performance against old bots. More data is pretty much always better.
Bring it on! Just to elaborate, in an iterated 1v1, a defectbot will reliably beat a randombot (4 to 1 if the randombot is unweighted). Or rather, any bot which detects a randombot and then always defects.
However, it’s not even necessary to beat the few randombots in a 1v1 to win the tournament, it is sufficient to beat a majority of other bots (including some cooperatebots for easy wins) to get ahead of the pack enough so that the results against the randombots don’t matter.
Can I suggest running a second tournement, including all these bots and new bots that we can examine these bots in writing? After a few iterations, we might beat random. Or we might develop all sorts of signaling quirks, but that would also be interesting.
I like this plan. I’d be willing to run it, unless AlexMennan wants to.
When I first announced the tournament, people came up with many suggestions for modifications to the rules, most of which I did not take for the sake of simplicity. Maybe we should try to figure out what rules will make it easier to write more effective bots, even if that would disqualify many of the bots already submitted to this tournament.
Am I the only one who sees a problem in that we’re turning a non-zero-sum game into a winner-take-all tournament? Perhaps instead of awarding a limited resource like bitcoins to the “winner”, each player should be awarded an unlimited resource such as karma or funny cat pictures according to their strategy’s performance.
No, not the only one. The same complaint was well received in the initial thread.
Maybe we should have a prisoner’s dilemma meta-tournament in which we define a tournament-goodness metric and then everyone submits a tournament design that is run using the bot participants in the previous tournament, and then use the rankings of those designs.
Wait: does anyone know a good way to design the meta-tournament?
I’ll list some suggestions.
Conduct a statistically significant number of trials. With few trial, the highest ranked agents will be the ones with a high variance in possible score, rather than the agents with the highest average score.
Allow no libraries, except for a very small whitelist (e.g. random numbers).
No timing, no threads. They make analysis too hard. The payoff might have to be changed to make it never in your best interest to make your opponent diverge, e.g. (-1, −1).
The rule “This is about Prisoner’s dilemma, not the language you’re working in. If you violate the spirit of this rule, your agent will be disqualified.”
(Possible) Change language to a restricted dialect of Python. You can analyze your opponent bytecode, if you wish, or run your opponent as a black-box. Programs with side effects (such as writing to global variables) and programs that use Pythonic trickiness (such as inspecting the stack) are verboten.
Most of these suggestions are okay, but don’t move away from Lisp. A very restricted Scheme (e.g. one way to set variables, one looping construct (I really like foldl, as N shows), etc.) would be good; the lexical scoping and general immutability make it one of the best Lisps for our purposes.
+1 for restricted Scheme. I suggest restricted Python because it is more accessible and would allow more people to participate, but personally I like Scheme a bit more.
On the other hand, many many people were turned off by Scheme last time, and few of them will be reading this post. I’d like to give them a shot, if only to get a more diverse crowd.
On looping constructs: Would that buy us anything? What if people use recursion for looping? Should we restrict recursion?
Question for the audience: would there be a benefit to using a programming language that allows only primitive recursive functions?
I think a programming language that only allows primitive recursion is a bad idea. One common pattern (which I think we want to allow) was for bots to simulate their opponents, which entails the ability to simulate arbitrary valid code. This would not be possible in a language which restricts to primitive recursion.
I think that the bots already submitted should be considered for grandfathering if the rules change to prohibit them.
I also have a specific entry in mind, but lack the expertise to go from ‘flowchart’ to ‘code’.
I think there would more people interested in playing if strategies could be submitted in pseudocode, so that would be great.
For entries that only eval(opponent(x)), for simple x, that seems reasonable. Strategies that rely on deeper analysis would need to look for specific things about the code that I, as a non-programmer, cannot describe accurately.
I would agree that they should compete even if they’re not awarded prizes for winning. Because we should at least have a comparison with performance against old bots. More data is pretty much always better.
It should also be trivial to calculate the score both in this heat and against all previously submitted bots.
Bring it on! Just to elaborate, in an iterated 1v1, a defectbot will reliably beat a randombot (4 to 1 if the randombot is unweighted). Or rather, any bot which detects a randombot and then always defects.
However, it’s not even necessary to beat the few randombots in a 1v1 to win the tournament, it is sufficient to beat a majority of other bots (including some cooperatebots for easy wins) to get ahead of the pack enough so that the results against the randombots don’t matter.