Interesting. My mental model says that by fine tuning a language model to be ready to output the contents of any square of a chess board, it should be possible to make it keep a model of that chess board even without making it output that model in inference mode.
I think I will have to try it out to see if it works.
What exactly do you mean by “without using a context”? If you mean “without the fine-tuned language model ever dumping the context into the output stream in practice in inference mode”, I would be extremely surprised if that was not possible.
If you mean “without the fine-tuned language model being trained to be able to dump the context into the output stream at any point”, I’m less confident.
For the sake of clarity, the approach I am trying is to fine-tune an openai language model (specifically babbage, since I’m not made of money) to simulate a command-line chess program, adding one command at a time, including several commands (get-square-content, is-legal-move, etc) that will ideally never show up in inference mode.
If things go as I expect, the end result will be a “language” model which would look like the following to interact with
[player] c4
[computer] e5
[player] g3
[computer] nf6
[player] bg2
[computer] d5
[player] bh1
Illegal move: h1 is occupied
[player] bxh1
Illegal move: cannot capture a piece of your own color
[player] cxd5
[computer] Nxd5
(this could be further fine-tuned to always chop off the first three lines, so it just starts with [player] always).
If control returns to the user whenever the model outputs [player], that becomes a chess chatbot.
Would a fine-tuned openai babbage that produced output like the above, and did not output illegal moves (outside of circumstances like “game too long for context window” or “prompt injection”) count as an instance of the thing you believe is not possible, or am I misunderstanding?
(note: my expectation is that such a thing is possible but would be wildly inefficient, since it would have to reconstruct the board state for each token it wants to output, and also not likely to play good chess. But I think legal chess would probably still be an attainable goal).
Interesting. My mental model says that by fine tuning a language model to be ready to output the contents of any square of a chess board, it should be possible to make it keep a model of that chess board even without making it output that model in inference mode.
I think I will have to try it out to see if it works.
What I’m asking with this particular test is, can an ai play blindfold chess, without using a context in order to recant every move in the game?
https://en.wikipedia.org/wiki/Blindfold_chess
What exactly do you mean by “without using a context”? If you mean “without the fine-tuned language model ever dumping the context into the output stream in practice in inference mode”, I would be extremely surprised if that was not possible.
If you mean “without the fine-tuned language model being trained to be able to dump the context into the output stream at any point”, I’m less confident.
For the sake of clarity, the approach I am trying is to fine-tune an openai language model (specifically babbage, since I’m not made of money) to simulate a command-line chess program, adding one command at a time, including several commands (
get-square-content,is-legal-move, etc) that will ideally never show up in inference mode.If things go as I expect, the end result will be a “language” model which would look like the following to interact with
(this could be further fine-tuned to always chop off the first three lines, so it just starts with
[player]always).If control returns to the user whenever the model outputs
[player], that becomes a chess chatbot.Would a fine-tuned openai babbage that produced output like the above, and did not output illegal moves (outside of circumstances like “game too long for context window” or “prompt injection”) count as an instance of the thing you believe is not possible, or am I misunderstanding?
(note: my expectation is that such a thing is possible but would be wildly inefficient, since it would have to reconstruct the board state for each token it wants to output, and also not likely to play good chess. But I think legal chess would probably still be an attainable goal).