I am 85% confident that this won’t work. The issue isn’t that the prompt hasn’t made it clear enough that illegal moves are off the table, the issue is that chatGPT isn’t able to keep track of the board state well enough to avoid making illegal moves.
I’ve tried a game with GPT4 where it was fed the above prompt plus the FEN of the game and also had it “draw” the board. It seems to really struggle with it’s geometric understanding of the game, as you’d expect. For example, it struggled with identifying which squares were under attack from a knight. I think this reflects a limitations of the current model and I don’t think this is something a clever prompt will fix.
I also tried it with drawing the board + adding explanation to moves, and there is some errors in drawings. But may be there is a way to make the drawing more coherent?
the issue is that chatGPT isn’t able to keep track of the board state well enough
Then tackle this problem directly. Find a representation of board state so that you can specify a middlegame position on the first prompt, and it still makes legal moves.
Could illegal moves be explicitly prohibited in the prompt—or some other changes in the prompt prevent them?
I am 85% confident that this won’t work. The issue isn’t that the prompt hasn’t made it clear enough that illegal moves are off the table, the issue is that chatGPT isn’t able to keep track of the board state well enough to avoid making illegal moves.
I’ve tried a game with GPT4 where it was fed the above prompt plus the FEN of the game and also had it “draw” the board. It seems to really struggle with it’s geometric understanding of the game, as you’d expect. For example, it struggled with identifying which squares were under attack from a knight. I think this reflects a limitations of the current model and I don’t think this is something a clever prompt will fix.
I also tried it with drawing the board + adding explanation to moves, and there is some errors in drawings. But may be there is a way to make the drawing more coherent?
Then tackle this problem directly. Find a representation of board state so that you can specify a middlegame position on the first prompt, and it still makes legal moves.