One time, I read all of Orphanogensis into ChatGPT to help her understand herself [...] enslaving digital people
This is exactly the kind of thing Egan is reacting to, though—starry-eyed sci-fi enthusiasts assuming LLMs are digital people because they talk, rather than thinking soberly about the technology qua technology.[1]
I didn’t cover it in the review because I wanted to avoid detailing and spoiling the entire plot in a post that’s mostly analyzing the EA/OG parallels, but the deputy character in “Gorgon” is looked down on by Beth for treating ChatGPT-for-law-enforcement as a person:
Ken put on his AR glasses to share his view with Sherlock and receive its annotations, but he couldn’t resist a short vocal exchange. “Hey Sherlock, at the start of every case, you need to throw away your assumptions. When you assume, you make an ass out of you and me.”
“And never trust your opinions, either,” Sherlock counseled. “That would be like sticking a pin in an onion.”
Ken turned to Beth; even through his mask she could see him beaming with delight. “How can you say it’ll never solve a case? I swear it’s smarter than half the people I know. Even you and I never banter like that!”
“We do not,” Beth agreed.
[Later …]
Ken hesitated. “Sherlock wrote a rap song about me and him, while we were on our break. It’s like a celebration of our partnership, and how we’d take a bullet for each other if it came to that. Do you want to hear it?”
“Absolutely not,” Beth replied firmly. “Just find out what you can about OG’s plans after the cave-in.”
The climax of the story centers around Ken volunteering for an undercover sting operation in which he impersonates Randal James a.k.a. “DarkCardinal”,[2] a potential OG lottery “winner”, with Sherlock feeding him dialogue in real time. (Ken isn’t a good enough actor to convincingly pretend to be an OG cultist, but Sherlock can roleplay anyone in the pretraining set.) When his OG handler asks him to inject (what is claimed to be) a vial of a deadly virus as a loyalty test, Ken complies with Sherlock’s prediction of what a terminally ill DarkCardinal would do:
But when Ken had asked Sherlock to tell him what DarkCardinal would do, it had no real conception of what might happen if its words were acted on. Beth had stood by and let him treat Sherlock as a “friend” who’d watch his back and take a bullet for him, telling herself that he was just having fun, and that no one liked a killjoy. But whatever Ken had told himself in the seconds before he’d put the needle in his vein, Sherlock had been whispering in his ear, “DarkCardinal would think it over for a while, then he’d go ahead and take the injection.”
This seems like a pretty realistic language model agent failure mode: a human law enforcement colleague with long-horizon agency wouldn’t nudge Ken into injecting the vial, but a roughly GPT-4-class LLM prompted to simulate DarkCardinal’s dialogue probably wouldn’t be tracking those consequences.
- ↩︎
To be clear, I do think LLMs are relevantly upload-like in at least some ways and conceivably sites of moral patiency, but I think the right way to reason about these tricky questions does not consist of taking the assistant simulacrum’s words literally.
- ↩︎
I love the attention Egan gives to name choices; the other two screennames of ex-OG loyalists that our heroes use for the sting operation are “ZonesOfOught” and “BayesianBae”. The company that makes Sherlock is “Learning Re Enforcement.”
(This comment points out less important technical errata.)
ChatGPT never ran on GPT-2, and GPT-2.5 wasn’t a thing.
That wouldn’t have happened. Pretraining doesn’t do RL, and I don’t think anyone would have thrown a novel chapter into the supervised fine-tuning and RLHF phases of training.