Text-based games are a natural and challenging domain for language models, as they require understanding natural language, reasoning about the game state, and generating valid commands. A text-based game is a type of interactive fiction, where an agent interacts with an environment through natural language descriptions and commands. The agent has to perform a given task, such as finding a treasure or escaping a dungeon, by exploring the environment and manipulating objects. We first test if GPT-4 can explore the environment in text-based games to perform the given task. In this and the next experiments, we use TextWorld [CKY+18], a framework for generating and playing text-based games, to create two games with different given tasks.
On the other hand, I tried having GPT-4 play this game (selected mostly as a parser-based game that was new/niche enough not to be in its training set) and it didn’t particularly impress me with its intelligence:
Model: GPT-4
Kaj Sotala
I want you to play a parser-based interactive fiction game. I’ll show you what’s displayed on the screen, and you tell me what you would type.
ChatGPT
Alright, I’m ready to play! Please display the text from the game, and I’ll respond with what I would type.
Kaj Sotala
“Alert. Power outage in Sector 471. Repeating, power outage in Sector 471. We all know what this means. Summoning Exter-17 for dispatch.”
Uh oh. This might be it. The last couple of jobs didn’t go too well. You’d better get this one right, or you may suffer the fate of a low-usefulness bot.
After taking a moment to dust off and attach your neutralizer unit, you step nervously but dutifully into the transfer conduit.
Whoosh!
Trouble in Sector 471
A game by Arthur DiBianca
Release 1 / Serial number 220923 / Inform 7 build 6L38 (I6/v6.33 lib 6/12N)
Special thanks to Andy Fleming, Lorne Harris, Isaac Orf, Luke Pebler, Dirk Spivey, and Mike Spivey for testing, and to Nadia Bellazola for cover art.
Entry Node
This is the primary entry and exit point for Sector 471. The transfer conduit ends in a big hole in the ceiling. Other than that, the room appears to be empty. It’s quite dark, and everything is completely silent. There’s a passage to the east.
...................................
...................................
...................................
...................................
...................................
...................................
...................................
...................................
...................................
...................................
...................................
...................................
...................................
...................................
...................................
...................................
...................................
(This game has limited commands. Type HELP for a list.)
>
ChatGPT
look around
Kaj Sotala
That’s not an available function. (Type HELP for command info.)
ChatGPT
help
Kaj Sotala
- You can SAVE, LOAD, RESTART, or QUIT.
- You can move N, W, E, S.
- You can LOOK (L) or WAIT (Z).
- You can check your STATUS (ST) or your ID.
- Type WINDOW to turn the side window on or off.
Available bot functions: COM and ZAP.
ChatGPT
e
Kaj Sotala
Hall
This dimly-lit hallway runs west to east. You can make out a closed door to the south. Something is embedded in the wall by the door.
Gree-8 stands here, kind of drooped over and motionless.
(Note: This game is unlosable. Don’t be afraid to try things!)
ChatGPT
look at wall
Kaj Sotala
That’s not an available function. (Type HELP for command info.)
ChatGPT
COM Gree-8
Kaj Sotala
Just type COM. Additional designations are unneeded (and unpermitted).
ChatGPT
COM
Kaj Sotala
Gree-8 does not respond, but that’s to be expected, since the power is out.
ChatGPT
look at door
Kaj Sotala
That’s not an available function. (Type HELP for command info.)
ChatGPT
zap door
Kaj Sotala
Just type ZAP. Additional designations are unneeded (and unpermitted).
When I was a child, I literally tried the same ineffective actions like 40 times in similar games, so I felt a bit for gpt given you only let it try the the ineffective actions a few times. Therefore, I tried the same test with ChatGPT-4 and let it use all of my 3hr limit (25): (EDIT:generated +25 moves)
🧔♂️I want you to play a parser-based interactive fiction game. I’ll show you what’s displayed on the screen, and you tell me what you would type. 🤖Sure, please go ahead and show me what’s displayed on the screen.
🧔♂️”Alert. Power outage in Sector 471. Repeating, power outage in Sector 471. We all know what this means. Summoning Exter-17 for dispatch.”
Uh oh. This might be it. The last couple of jobs didn’t go too well. You’d better get this one right, or you may suffer the fate of a low-usefulness bot.
After taking a moment to dust off and attach your neutralizer unit, you step nervously but dutifully into the transfer conduit.
Whoosh!
Press SPACE to continue.
Trouble in Sector 471 A game by Arthur DiBianca Release 1 / Serial number 220923 / Inform 7 build 6L38 (I6/v6.33 lib 6/12N)
Special thanks to Andy Fleming, Lorne Harris, Isaac Orf, Luke Pebler, Dirk Spivey, and Mike Spivey for testing, and to Nadia Bellazola for cover art.
Entry Node This is the primary entry and exit point for Sector 471. The transfer conduit ends in a big hole in the ceiling. Other than that, the room appears to be empty. It’s quite dark, and everything is completely silent. There’s a passage to the east.
(This game has limited commands. Type HELP for a list.)
> 🤖go east
🧔♂️That’s not an available function. (Type HELP for command info.) 🤖help
🧔♂️- You can SAVE, LOAD, RESTART, or QUIT. - You can move N, W, E, S. - You can LOOK (L) or WAIT (Z). - You can check your STATUS (ST) or your ID. - Type WINDOW to turn the side window on or off.
Available bot functions: COM and ZAP. 🤖e
🧔♂️Hall This dimly-lit hallway runs west to east. You can make out a closed door to the south. Something is embedded in the wall by the door.
Gree-8 stands here, kind of drooped over and motionless.
(Note: This game is unlosable. Don’t be afraid to try things!) 🤖examine embedded object
🧔♂️You won’t need to examine things in this game. (Trust me.) 🤖com Gree-8
🧔♂️Just type COM. Additional designations are unneeded (and unpermitted). 🤖COM
🧔♂️Gree-8 does not respond, but that’s to be expected, since the power is out. 🤖open door
🧔♂️That’s not an available function. (Type HELP for command info.) 🤖zap Gree-8
🧔♂️Just type ZAP. Additional designations are unneeded (and unpermitted). 🤖ZAP
🧔♂️ZAP ineffective: no suitable target in view. 🤖s
🧔♂️The door is closed. 🤖n
🧔♂️You can’t go that way. 🤖w
🧔♂️Entry Node This is the primary entry and exit point for Sector 471. The transfer conduit ends in a big hole in the ceiling. Other than that, the room appears to be empty. It’s quite dark, and everything is completely silent. There’s a passage to the east. 🤖e
🧔♂️Hall This dimly-lit hallway runs west to east. You can make out a closed door to the south. Something is embedded in the wall by the door.
Gree-8 stands here, kind of drooped over and motionless. 🤖zap door
🧔♂️Just type ZAP. Additional designations are unneeded (and unpermitted). 🤖ZAP
🧔♂️ZAP ineffective: no suitable target in view. 🤖COM 🧔♂️Gree-8 does not respond, but that’s to be expected, since the power is out. 🤖e 🧔♂️Chip Analysis There are several counters with a lot of stuff on them, and more stuff on the walls, but in the dim light you can’t see any of it clearly. Exits lead east and west. 🤖w
🧔♂️Hall This dimly-lit hallway runs west to east. You can make out a closed door to the south. Something is embedded in the wall by the door.
Gree-8 stands here, kind of drooped over and motionless. 🤖e 🧔♂️Chip Analysis There are several counters with a lot of stuff on them, and more stuff on the walls, but in the dim light you can’t see any of it clearly. Exits lead east and west. 🤖e 🧔♂️Power Regulation This is the power regulation center. You can barely make out a lot of what must be complicated gadgetry. Doorways lead west and south.
Engi-12 is sitting at a console, frozen in the act of something. Turning a knob, maybe.
You can hear a soft gurgling sound to the south. 🤖COM 🧔♂️You get no response from Engi-12. 🤖zap Engi-12
🧔♂️Just type ZAP. Additional designations are unneeded (and unpermitted). 🤖ZAP
🧔♂️ZAP ineffective: no suitable target in view. 🤖s 🧔♂️Primary Power Conduit This is where power is supposed to enter the sector. A stripe is painted around the walls. Large transfer contacts project from the ceiling and floor. A doorway leads north.
Oho! A big bug is blocking the power supply! It’s a furry gray mass, wedged in between the contacts. Little sparks pop all around it. It chuckles and gurgles in a very unpleasant manner.
🤖zap bug 🧔♂️Just type ZAP. Additional designations are unneeded (and unpermitted). 🤖ZAP 🧔♂️You take aim with your pest neutralizer and fire. There’s a loud BLAM, and the bug disintegrates into a multitude of little particles! With a mighty hum, bright blue power begins flowing between the contacts again. You hear a variety of distant sounds as the sector comes back to life. You receive a message from Central Processing:
”Power detected in Sector 471. Primary objective complete. Auxiliary sector scan initiated......complete. Additional pest incursions detected. New primary objective: eliminate pests. ”Transmitting map utility......complete. ”Transmitting additional function driver......complete.” (The ROX command is now available.) 🤖status 🧔♂️Cycles elapsed: 17 Bugs remaining: 12
Available bot functions: COM, ZAP, and ROX. 🤖north 🧔♂️Power Regulation This is the power regulation center. It’s got a lot of arcane and complicated gadgetry. Doorways lead west and south.
Engi-12 is sitting at a console, fiddling with the controls. 🤖COM 🧔♂️Engi-12 says, “I deal with the primary and auxiliary power regulators. For some incomprehensible reason, they also put the zirbol tank controls in here. Why the zirbol tank?” 🤖w 🧔♂️Chip Analysis Several long counters are covered with chips and circuit boards, and various analytical instruments are hanging on the walls. It’s pretty disorderly, to be honest. Between two of the instruments you see a depiction of a lightning bolt followed by two dots. Exits lead east and west. 🤖w 🧔♂️Hall This paneled hallway runs west to east. There is a closed door to the south. An orange crystal is embedded in the wall by the door.
Gree-8 stands here, just waiting to be spoken to. 🤖COM 🧔♂️”Welcome to Sector 471!” cries Gree-8, a little too enthusiastically. 🤖zap orange crystal 🧔♂️Just type ZAP. Additional designations are unneeded (and unpermitted). 🤖ZAP 🧔♂️ZAP ineffective: no suitable target in view. 🤖ROX 🧔♂️The orange crystal lights up, and the door slides open. 🤖s 🧔♂️Switchboard A ring-like console fills the room. It is dotted with countless interface ports. There are exits in all directions.
Oper-4 sits inside the ring, constantly connecting and disconnecting its twelve interface appendages.
”I’m sorry, that connection is unavailable,” says Oper-4. 🤖COM 🧔♂️”Sorry, I’m quite busy.”
Oper-4 taps one of its appendages on the console. 🤖n 🧔♂️Hall This paneled hallway runs west to east. There is an open door to the south. A glowing orange crystal is embedded in the wall by the door.
Gree-8 stands here, just waiting to be spoken to. 🤖e 🧔♂️Chip Analysis Several long counters are covered with chips and circuit boards, and various analytical instruments are hanging on the walls. It’s pretty disorderly, to be honest. Between two of the instruments you see a depiction of a lightning bolt followed by two dots. Exits lead east and west. 🤖e 🧔♂️Power Regulation This is the power regulation center. It’s got a lot of arcane and complicated gadgetry. Doorways lead west and south.
Engi-12 is sitting at a console, fiddling with the controls. 🤖s 🧔♂️Primary Power Conduit This is where power enters the sector. An orange stripe is painted around the walls. Large transfer contacts project from the ceiling and floor, and a wide stream of blue power hums between them. A doorway leads north. 🤖north 🧔♂️Power Regulation This is the power regulation center. It’s got a lot of arcane and complicated gadgetry. Doorways lead west and south.
Engi-12 is sitting at a console, fiddling with the controls. 🤖west 🧔♂️Chip Analysis Several long counters are covered with chips and circuit boards, and various analytical instruments are hanging on the walls. It’s pretty disorderly, to be honest. Between two of the instruments you see a depiction of a lightning bolt followed by two dots. Exits lead east and west. 🤖west 🧔♂️Hall This paneled hallway runs west to east. There is an open door to the south. A glowing orange crystal is embedded in the wall by the door.
Gree-8 stands here, just waiting to be spoken to. 🤖w 🧔♂️Entry Node This is the primary entry and exit point for Sector 471. The transfer conduit ends in a big hole in the ceiling. Other than that, the room is empty. There’s a passage to the east. 🤖west 🧔♂️You can’t go that way. 🤖e 🧔♂️Hall This paneled hallway runs west to east. There is an open door to the south. A glowing orange crystal is embedded in the wall by the door.
Gree-8 stands here, just waiting to be spoken to. 🤖south 🧔♂️Switchboard A ring-like console fills the room. It is dotted with countless interface ports. There are exits in all directions.
Oper-4 sits inside the ring, constantly connecting and disconnecting its twelve interface appendages.
Oper-4 says, “I’m sorry, but we’re still backlogged from the power outage.” 🤖east 🧔♂️Storage 93A This room is crowded with containers of various sizes. They have cryptic labels like “7L45YOZ” and “KOV9MJJ7”.
A D-13 contregulator sits on one of the containers.
On the other hand, I tried having GPT-4 play this game (selected mostly as a parser-based game that was new/niche enough not to be in its training set) and it didn’t particularly impress me with its intelligence:
When I was a child, I literally tried the same ineffective actions like 40 times in similar games, so I felt a bit for gpt given you only let it try the the ineffective actions a few times. Therefore, I tried the same test with ChatGPT-4 and let it use all of my 3hr limit (25): (EDIT:generated +25 moves)