I’m curious how long it’ll be until a general model can play Diplomacy at this level. Anyone fine-tuned an LLM like GPT-3 on chess yet? Chess should be simpler for an LLM to learn unless my intuition is misleading?
We never did ELO tests, but the 2.7B model trained from scratch on human games in PGN notation beat me and beat my colleague (~1800 ELO). But it would start making mistakes if the game went on very long (we hypothesized it was having difficulties constructing the board state from long PGN contexts), so you could beat it by drawing the game out.
I couldn’t find any actual elo estimates (nor code that lets me estimate the elo of a bot), but GPT-3 (at least, davinci and text-davinci-002) can play chess decently well using PGN notation without any finetuning.
I’m curious how long it’ll be until a general model can play Diplomacy at this level. Anyone fine-tuned an LLM like GPT-3 on chess yet? Chess should be simpler for an LLM to learn unless my intuition is misleading?
I’ve fine tuned LLMs on chess and it indeed is quite easy for them to learn.
Interesting, how good can they get? Any ELO estimates?
We never did ELO tests, but the 2.7B model trained from scratch on human games in PGN notation beat me and beat my colleague (~1800 ELO). But it would start making mistakes if the game went on very long (we hypothesized it was having difficulties constructing the board state from long PGN contexts), so you could beat it by drawing the game out.
I couldn’t find any actual elo estimates (nor code that lets me estimate the elo of a bot), but GPT-3 (at least,
davinciandtext-davinci-002) can play chess decently well using PGN notation without any finetuning.