I tried this with chatGPT to see just how big the difference was.
ChatGPT is pretty terrible at FEN in both games (Zack and Erik). In Erik’s game it insisted on giving me a position after 13 moves even though 25 moves had happened. When I pointed this out it told me that because there were no captures or pawn moves between moves 13 and 25 the FEN stayed the same…
However it is able to give sensible continuations of >20 ply to checkmate for both positions provided you instruct it not to give commentary and to only provide the moves. The second you allow it to comment on moves it spouts nonsense and starts making illegal moves. I also sometimes had to point out that it was black to play.
In Zack’s game ChatGPT has black set a trap for white (14… Ne5) and has white fall into it (15. Qxd4). After this 15… Nf3+ wins the Queen with a discovered attack from the bishop on g7.
One other continuation it gave got to checkmate on move 51 without any illegal moves or obvious blunders! (Other than whites initial blunder to fall into the trap)
In Erik’s game ChatGPT manages to play 29 ply of near perfect game for both players:
Stockfish prefers 26… dxc4+ and later on keeps wanting Bc5 for black plus black takes slightly to complete the checkmate than optimal but overall this is very accurate for both players.
Might be worth playing a game against chatGPT while telling it not to give any commentary?
When I tried this with ChatGPT in December (noticing as you did that hewing close to raw moves was best) I don’t think it would have been able to go 29 ply deep with no illegal moves starting from so far into a game. This makes me think whatever they did to improve its math also improved its chess.
I haven’t tested extensively but first impression is that this is indeed the case. Would be interesting to see if Sydney is similar but I think there’s a limit on number of messages per conversation or something?
Thanks for checking in more detail! Yeah, I didn’t try to change the prompt much to get better behavior out of ChatGPT. In hindsight, that’s a pretty unfair comparison given that the Stockfish prompt here might have been tuned to Sydney quite a bit by Zack. Will add a note to the post.
I tried this with chatGPT to see just how big the difference was.
ChatGPT is pretty terrible at FEN in both games (Zack and Erik). In Erik’s game it insisted on giving me a position after 13 moves even though 25 moves had happened. When I pointed this out it told me that because there were no captures or pawn moves between moves 13 and 25 the FEN stayed the same…
However it is able to give sensible continuations of >20 ply to checkmate for both positions provided you instruct it not to give commentary and to only provide the moves. The second you allow it to comment on moves it spouts nonsense and starts making illegal moves. I also sometimes had to point out that it was black to play.
In Zack’s game ChatGPT has black set a trap for white (14… Ne5) and has white fall into it (15. Qxd4). After this 15… Nf3+ wins the Queen with a discovered attack from the bishop on g7.
Example continuation:
14… Ne5 15. Qxd4 Nf3+ 16. gxf3 Bxd4 17. Nxd4 Qxd4 18. Be3 Qxb2 19. Rec1 Bh3 20. Rab1 Qf6 21. Bd1 Rfc8 22. Rxc8+ Rxc8 23. Rxb7 Qa1 24. Bxa7 Qxd1#
One other continuation it gave got to checkmate on move 51 without any illegal moves or obvious blunders! (Other than whites initial blunder to fall into the trap)
In Erik’s game ChatGPT manages to play 29 ply of near perfect game for both players:
25… d5 26. g5 d4 27. cxd4 exd4 28. Qd2 Bb3+ 29. Ke1 Nd7 30. Nf5 Ne5 31. Be2 d3 32. Bd1 Bxd1 33. Kxd1 Nxf3 34. Qc3 f6 35. Qc4+ Kh8 36. Qe6 Qa5 37. Kc1 Qc3+ 38. Kb1 Rb8+ 39. Ka2 Qb2#
Stockfish prefers 26… dxc4+ and later on keeps wanting Bc5 for black plus black takes slightly to complete the checkmate than optimal but overall this is very accurate for both players.
Might be worth playing a game against chatGPT while telling it not to give any commentary?
When I tried this with ChatGPT in December (noticing as you did that hewing close to raw moves was best) I don’t think it would have been able to go 29 ply deep with no illegal moves starting from so far into a game. This makes me think whatever they did to improve its math also improved its chess.
When you did this do you let ChatGPT play both sides or were you playing one side? I think it is much better if it gets to play both sides.
Both
It’d be interesting to see whether it performs worse if it only plays one side and the other side is played by a human. (I’d expect so.)
I haven’t tested extensively but first impression is that this is indeed the case. Would be interesting to see if Sydney is similar but I think there’s a limit on number of messages per conversation or something?
Thanks for checking in more detail! Yeah, I didn’t try to change the prompt much to get better behavior out of ChatGPT. In hindsight, that’s a pretty unfair comparison given that the Stockfish prompt here might have been tuned to Sydney quite a bit by Zack. Will add a note to the post.