Similar to your lazy suggestion, challenging the subject to a novel (probably abstract-strategy) game seems like a possibly-fruitful approach.
On a similar note: Zendo-variations. I played a bit on a webcomic forum using natural numbers as koans, for example; this would be easy to execute over a chat interface, and a good test of both recall and problem-solving.
Nope; general game-playing is a well-studied area of AI; the AI’s aren’t great at it, but if you aren’t playing them for a long time they can certainly pass as a bad human. Zendo-like “analogy-finding” has also been studied.
By only demanding very structured action types, instead of a more free-flowing, natural-language based interaction, you are handicapping yourself as a judge immensely.
Similar to your lazy suggestion, challenging the subject to a novel (probably abstract-strategy) game seems like a possibly-fruitful approach.
On a similar note: Zendo-variations. I played a bit on a webcomic forum using natural numbers as koans, for example; this would be easy to execute over a chat interface, and a good test of both recall and problem-solving.
Maybe just do some roleplaying, with the judge as the DM.
Nope; general game-playing is a well-studied area of AI; the AI’s aren’t great at it, but if you aren’t playing them for a long time they can certainly pass as a bad human. Zendo-like “analogy-finding” has also been studied.
By only demanding very structured action types, instead of a more free-flowing, natural-language based interaction, you are handicapping yourself as a judge immensely.