One of the most interesting responses I’ve gotten from GPT-3 was to this prompt: “1.e4 c5 2.d4 cxd4”: the first moves of a chess game. GPT-3′s response: “” 3.c3 dxc3 4.Nxc3 g6 5.Nf3 Bg7 6.Bc4 Nc6 7.0-0 Qc7 8.Qe2 e6 9.Bg5 Nf6 10.Rd1 Ne5 11.Be3 0-0 12.a3 d5 13.Nb5 Qc8 14.Nxc7 Qxc7 15.e5 Nfd7 16.Nxd5 Nc4 17.Bxc4 Qxc4 18.Qe4 e5 19.Qxe5 Qc7 20.Rd3 Qc6 21.Qxc4 dxc4 22.Nd4 Rfe8 23.Rfd1 b5 24.Nb3 Rc8 25.Nd2 b4 26.cd3 f6 27.exf6 gxf6 28.Rd4 b3 29.Ke2 Be6 30.Ne4 Rd8 31.Nc5 Bg7 32.Rxd8+ Bxd8 33.a4 bxa3 34.Ra1 Kh8 35.Bf1 Bc3 36.Nxa6 f5 37.Ne6 Be”.
This looks like a chess game, though it is not: toward the end it veers into move-like notations that in fact don’t represent legal moves. But clearly much more than superficial pattern extension is going on. GPT-3 produced something that understands chess notation, has a model of the game, tracks state, and has learned to make good moves. After a few moves it deviates from any actual game I could find evidence of online, but it continues to make objectively excellent moves (for awhile). GPT-3 has generated something that by any standard is simulating chess gameplay (though I still can’t relate to calling GPT-3 itself a simulator here). This isn’t though a simulator in the sense that eg Stockfish is a simulator—Stockfish would never make an illegal move like GPT-3′s creation did. It does seem quite apt to me to speak of GPT-3′s production as speculative simulation, bearing in mind that there’s nothing to say that one day its speculations might not lead to gameplay that exceeds SOTA, human or machine, just as Einstein’s thought experiments speculated into existence a better physics. Similar things could be said about its productions of types other than simulator: pattern extensions, agents, oracles, and so on, in all of which cases we must account for the fact that its intelligence happily produces examples ranging from silly to sublime depending on how we prompt it...
This seems like a simulator in the same way the human imagination is a simulator. I could mentally simulate a few chess moves after the ones you prompted. After a while (probably a short while) I’d start losing track of things and start making bad moves. Eventually I’d probably make illegal moves, or maybe just write random move-like character strings if I was given some motivation for doing so and thought I could get away with it.
Yes, it sure felt like that. I don’t know whether you played through the game or not, but as a casual chess player, I’m very familiar with the experience of trying to follow a game from just the notation and experiencing exactly what you describe. Of course a master can do that easily and impeccably, and it’s easy to believe that GPT-3 could do that too with the right tuning and prompting. I don’t have the chops to try that, but if it’s correct it would make your ‘human imagination’ simile still more compelling. Similarly, the way GPT-3 “babbles” like a toddler just acquiring language sometimes, but then can become more coherent with better / more elaborate / recursive prompting is a strong rhyme with a human imagination maturing through its activity in a world of words.
Of course a compelling analogy is just a compelling analogy… but that’s not nothing!
It’s almost a cliche that a chess engine doesn’t “think like a human”, but we have here the suggestion not only that GPT could conceivably attain impeccable performance as a chess simulator, but perhaps also in such a way that it would “think like a human [grandmaster or better]”. Purely speculative, of course...
One of the most interesting responses I’ve gotten from GPT-3 was to this prompt: “1.e4 c5 2.d4 cxd4”: the first moves of a chess game. GPT-3′s response: “” 3.c3 dxc3 4.Nxc3 g6 5.Nf3 Bg7 6.Bc4 Nc6 7.0-0 Qc7 8.Qe2 e6 9.Bg5 Nf6 10.Rd1 Ne5 11.Be3 0-0 12.a3 d5 13.Nb5 Qc8 14.Nxc7 Qxc7 15.e5 Nfd7 16.Nxd5 Nc4 17.Bxc4 Qxc4 18.Qe4 e5 19.Qxe5 Qc7 20.Rd3 Qc6 21.Qxc4 dxc4 22.Nd4 Rfe8 23.Rfd1 b5 24.Nb3 Rc8 25.Nd2 b4 26.cd3 f6 27.exf6 gxf6 28.Rd4 b3 29.Ke2 Be6 30.Ne4 Rd8 31.Nc5 Bg7 32.Rxd8+ Bxd8 33.a4 bxa3 34.Ra1 Kh8 35.Bf1 Bc3 36.Nxa6 f5 37.Ne6 Be”.
This looks like a chess game, though it is not: toward the end it veers into move-like notations that in fact don’t represent legal moves. But clearly much more than superficial pattern extension is going on. GPT-3 produced something that understands chess notation, has a model of the game, tracks state, and has learned to make good moves. After a few moves it deviates from any actual game I could find evidence of online, but it continues to make objectively excellent moves (for awhile). GPT-3 has generated something that by any standard is simulating chess gameplay (though I still can’t relate to calling GPT-3 itself a simulator here). This isn’t though a simulator in the sense that eg Stockfish is a simulator—Stockfish would never make an illegal move like GPT-3′s creation did. It does seem quite apt to me to speak of GPT-3′s production as speculative simulation, bearing in mind that there’s nothing to say that one day its speculations might not lead to gameplay that exceeds SOTA, human or machine, just as Einstein’s thought experiments speculated into existence a better physics. Similar things could be said about its productions of types other than simulator: pattern extensions, agents, oracles, and so on, in all of which cases we must account for the fact that its intelligence happily produces examples ranging from silly to sublime depending on how we prompt it...
This seems like a simulator in the same way the human imagination is a simulator. I could mentally simulate a few chess moves after the ones you prompted. After a while (probably a short while) I’d start losing track of things and start making bad moves. Eventually I’d probably make illegal moves, or maybe just write random move-like character strings if I was given some motivation for doing so and thought I could get away with it.
Yes, it sure felt like that. I don’t know whether you played through the game or not, but as a casual chess player, I’m very familiar with the experience of trying to follow a game from just the notation and experiencing exactly what you describe. Of course a master can do that easily and impeccably, and it’s easy to believe that GPT-3 could do that too with the right tuning and prompting. I don’t have the chops to try that, but if it’s correct it would make your ‘human imagination’ simile still more compelling. Similarly, the way GPT-3 “babbles” like a toddler just acquiring language sometimes, but then can become more coherent with better / more elaborate / recursive prompting is a strong rhyme with a human imagination maturing through its activity in a world of words.
Of course a compelling analogy is just a compelling analogy… but that’s not nothing!
It’s almost a cliche that a chess engine doesn’t “think like a human”, but we have here the suggestion not only that GPT could conceivably attain impeccable performance as a chess simulator, but perhaps also in such a way that it would “think like a human [grandmaster or better]”. Purely speculative, of course...