Comment on “Death and the Gorgon”
(some plot spoilers)
There’s something distinctly uncomfortable about reading Greg Egan in the 2020s. Besides telling gripping tales with insightful commentary on the true nature of mind and existence, Egan stories written in the 1990s and set in the twenty-first century excelled at speculative worldbuilding, imagining what technological wonders might exist in the decades to come and how Society might adapt to them.
In contrast, “Death and the Gorgon”, published in the January/February 2024 issue of Asimov’s, feels like it’s set twenty minutes into the future. The technologies on display are an AI assistant for police officers (capable of performing research tasks and carrying on conversation) and real-time synthetic avatars (good enough to pass as a video call with a real person). When these kinds of products showed up in “’90s Egan”—I think of Worth’s “pharm” custom drug dispenser in Distress (1995) or Maria’s “mask” for screening spam calls in Permutation City (1994)—it was part of the background setting of a more technologically advanced world than our own.
Reading “Gorgon” in 2024, not only do the depicted capabilities seem less out of reach (our language model assistants and deepfakes aren’t quite there yet, but don’t seem too far off), but their literary function has changed: much of the moral of “Gorgon” seems to be to chide people in the real world who are overly impressed by ChatGPT. Reality and Greg Egan are starting to meet in the middle.
Our story features Beth, a standard-issue Greg Egan protagonist[1] as a small-town Colorado sheriff investigating the suspicious destruction of a cryonics vault in an old mine: a naturally occurring cave-in seems unlikely, but it’s not clear who would have the motive to thaw (murder?) a hundred frozen heads.
Graciously tolerating the antics of her deputy, who is obsessed with the department’s trial version of (what is essentially) ChatGPT-for-law-enforcement, Beth proceeds to interview the next of kin, searching for a motive. She discovers that many of the cryopreserved heads were beneficiaries of a lottery for terminally ill patients in which the prize was free cyronic suspension. The lottery is run by OG—”Optimized Giving”—a charitable group concerned with risks affecting the future of humanity. As the investigation unfolds, Beth and a colleague at the FBI begin to suspect that the lottery is a front for a creative organized crime scheme: OG is recruiting terminal patients to act as assassins, carrying out hits in exchange for “winning” the lottery. (After which another mafia group destroyed the cryonics vault as retaliation.) Intrigue, action, and a cautionary moral ensue as our heroes make use of ChatGPT-for-law-enforcement to prove their theory and catch OG red-handed before more people get hurt.
So, cards on the table: this story spends a lot of wordcount satirizing a subculture that, unfortunately, I can’t credibly claim not to be a part of. “Optimized Giving” is clearly a spoof on the longtermist wing of Effective Altruism—and if I’m not happy about how the “Effective Altruism” brand ate my beloved rationalism over the 2010s, I don’t think anyone would deny the contiguous memetic legacy involving many of the same people. (Human subcultures are nested fractally; for the purposes of reviewing the story, it would benefit no one for me to to insist that Egan isn’t talking about me and my people, even if, from within the subculture, it looks like the OpenPhil people and the MIRI people and the Vassarites and … &c. are all totally different and in fact hate each other’s guts.)
I don’t want to be defensive, because I’m not loyal to the subculture, its leaders, or its institutions. In the story, Beth talks to a professor—think Émile Torres as a standard-issue Greg Egan character—who studies “apostates” from OG who are angry about “the hubris, the deception, and the waste of money.” That resonated with me a lot: I have a long dumb story to tell about hubris and deception, and the corrupting forces of money are probably a big part of the explanation for the rise and predictable perversion of Effective Altruism.
So if my commentary on Egan’s satire contains some criticism, it’s absolutely not because I think my ingroup is beyond reproach and doesn’t deserve to satirized. They (we) absolutely do. (I took joy in including a similar caricature in one of my own stories.) But if Egan’s satire doesn’t quite hit the mark of explaining exactly why the group is bad, it’s not an act of partisan loyalty for me to contribute my nuanced explanation of what I think it gets right and what it gets wrong. I’m not carrying water for the movement;[2] it’s just a topic that I happen to have a lot of information about.
Without calling it a fair portrayal, the OG of “Gorgon” isn’t a strawman conjured out of thin air; the correspondences to its real-world analogue are clear. When our heroine suspiciously observes that these soi-disant world-savers don’t seem to be spending anything on climate change and the Émile Torres–analogue tells her that OG don’t regard it as an existential threat, this is also true of real-world EA. When the Torres-analogue says that “OG view any delay in spreading humanity at as close to light-speed as possible as the equivalent of murdering all the people who won’t have a chance to exist in the future,” the argument isn’t a fictional parody; it’s a somewhat uncharitably phrased summary of Nick Bostrom’s “Astronomical Waste: The Opportunity Cost of Delayed Technological Development”. When the narrator describes some web forums as “interspers[ing] all their actual debunking of logical fallacies with much more tendentious claims, wrapped in cloaks of faux-objectivity” and being “especially prone to an abuse of probabilistic methods, where they pretended they could quantify both the likelihood and the potential harm for various implausible scenarios, and then treated the results of their calculations—built on numbers they’d plucked out of the air—as an unimpeachable basis for action”, one could quibble with the disparaging description of subjective probability, but you can tell which website is being alluded to.
The cryonics-as-murder-payment lottery fraud is fictional, of course, but I’m inclined to read it as artistically-licensed commentary on a strain of ends-justify-the-means thinking that does exist within EA. EA organizations don’t take money from the mob for facilitating contract killings, but they did take money from the largest financial fraud in history, which was explicitly founded as a means to make money for EA. (One could point out that the charitable beneficiaries of Sam Bankman-Fried’s largesse didn’t know that FTX wasn’t an honest business, but we have to assume that the same is true of OG in the story: only a few insiders would be running the contract murder operation, not the rank-and-file believers.)
While the depiction of OG in the story clearly shows familiarity with the source material, the satire feels somewhat lacking qua anti-EA advocacy insofar as it relies too much on mere dismissal rather than presenting clear counterarguments.[3] The effect of OG-related web forums on a vulnerable young person are described thus:
Super-intelligent AIs conquering the world; the whole Universe turning out to be a simulation; humanity annihilated by aliens because we failed to colonize the galaxy in time. Even if it was all just stale clichés from fifty-year-old science fiction, a bright teenager like Anna could have found some entertainment value analyzing the possibilities rigorously and puncturing the forums’ credulous consensus. But while she’d started out healthily skeptical, some combination of in-forum peer pressure, the phony gravitas of trillions of future deaths averted, and the corrosive effect of an endless barrage of inane slogans pimped up as profound insights—all taking the form “X is the mind-killer,” where X was pretty much anything that might challenge the delusions of the cult—seemed to have worn down her resistance in the end.
I absolutely agree that healthy skepticism is critical when evaluating ideas and that in-forum peer pressure and the gravitas of a cause (for any given set of peers and any given cause) are troubling sources of potential bias—and that just because a group pays lip service to the value of healthy skepticism and the dangers of peer pressure and gravitas, doesn’t mean the group’s culture isn’t still falling prey to the usual dysfunctions of groupthink. (As the inane slogan goes, “Every cause wants to be a cult.”)
That being said, however, ideas ultimately need to be judged on their merits, and the narration in this passage[4] isn’t giving the reader any counterarguments to the ideas being alluded to. (As Egan would know, science fiction authors having written about an idea does not make the idea false.) The clause about the whole Universe turning out to be a simulation is probably a reference to Bostrom’s simulation argument, which is a disjunctive, conditional claim: given some assumptions in the philosophy of mind and the theory of anthropic reasoning, then if future civilization could run simulations of its ancestors, then either they won’t want to, or we’re probably in one of the simulations (because there are more simulated than “real” histories). The clause about humanity being annihilated by failing to colonize the galaxy in time is probably a reference to Robin Hanson et al.’s grabby aliens thesis, that the Fermi paradox can be explained by a selection effect: there’s a relatively narrow range of parameters in which we would see signs of an expanding alien civilization in our skies without already having been engulfed by them.
No doubt many important criticisms could be made of Bostrom’s or Hanson’s work, perhaps by a bright teenager finding entertainment value in analyzing the possibilities rigorously. But there’s an important difference between having such a criticism[5] and merely asserting that it could exist. Speaking only to my own understanding, Hanson’s and Bostrom’s arguments both look reasonable to me? It’s certainly possible I’ve just been hoodwinked by the cult, but if so, the narrator of “Gorgon”’s snarky description isn’t helping me snap out of it.
It’s worth noting that despite the notability of Hanson’s and Bostrom’s work, in practice, I don’t see anyone in the subculture particularly worrying about losing out on galaxies due to competition with aliens—admittedly, because we’re worried about “super-intelligent AIs conquering the world” first.[6] About which, “Gorgon” ends on a line from Beth about “the epic struggle to make computers competent enough to help bring down the fools who believe that they’re going to be omnipotent.”
This is an odd take from the author[7] of multiple novels in which software minds engage in astronomical-scale engineering projects. Accepting the premise that institutional longtermist EA deserves condemnation for being goofy and a fraud: in condemning them, why single out as the characteristic belief of this despicable group, the idea that future AI could be really powerful?[8] Isn’t that at least credible? Even if you think people in the cult or who work at AI companies are liars or dupes, it’s harder to say that about eminent academics like Stuart Russell, Geoffrey Hinton, Yoshua Bengio, David Chalmers, and Daniel Dennett, who signed a statement affirming that “[m]itigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”[9]
Egan’s own work sometimes features artificial minds with goals at odds with their creator, as in “Steve Fever” (2007) or “Crystal Nights” (2008), and with substantial advantages over biological creatures: in Diaspora (1997), the polis citizens running at 800 times human speed were peace-loving, but surely could have glassed the fleshers in a war if they wanted to. If you believe that AI could be at odds with its creators and hold a competitive advantage, scenarios along the lines of “super-intelligent AIs conquering the world” should seem plausible rather than far-fetched—a natural phenomenon straightforwardly analogous to human empires conquering other countries, or humans dominating other animals.
Given so many shared premises, it’s puzzling to me why Egan seems to bear so much antipathy towards “us”,[10] rather than than regarding the subculture more coolly, as a loose amalgamation of people interested in many of the same topics as him, but having come to somewhat different beliefs. (Egan doesn’t seem to think human-level AI is at all close, nor that AI could be qualitatively superhumanly intelligent; an aside in Schild’s Ladder (2002) alludes to a fictional result that there’s nothing “above” general intelligence of the type humans have, modulo speed and memory.) He seems to expect the feeling to be mutual: when someone remarked on Twitter about finding it funny that the Less Wrong crowd likes his books, Egan replied, “Oh, I think they’ve noticed, but some of them still like the, err, ‘early, funny ones’ that predate the cult and hence devote no time to mocking it.”
Well, I can’t speak for anyone else, but personally, I like Egan’s later work, including “Death and the Gorgon.”[11] Why wouldn’t I? I am not so petty as to let my appreciation of well-written fiction be dulled by the incidental fact that I happen to disagree with some of the author’s views on artificial intelligence and a social group that I can’t credibly claim not to be a part of. That kind of dogmatism would be contrary to the ethos of humanism and clear thinking that I learned from reading Greg Egan and Less Wrong—an ethos that doesn’t endorse blind loyalty to every author or group you learned something from, but a discerning loyalty to whatever was good in what the author or group saw in our shared universe. I don’t know what the future holds in store for humanity. But whatever risks and opportunities nature may present, I think our odds are better for every thinking individual who tries to read widely and see more.[12]
- ↩︎
Some people say that Greg Egan is bad at characterization. I think he just specializes in portraying reasonable people, who don’t have grotesque personality flaws to be the subject of “characterization.”
- ↩︎
I do feel bad about the fraction of my recent writing output that consists of criticizing the movement—not because it’s disloyal, but because it’s boring. I keep telling myself that one of these years I’m going to have healed enough trauma to forget about these losers already and just read ArXiv papers. Until then, you get posts like this one.
- ↩︎
On the other hand, one could argue that satire just isn’t the right medium for presenting counterarguments, which would take up a lot of wordcount without advancing the story. Not every written work can accomplish all goals! Maybe it’s fine for this story to make fun of the grandiose and cultish elements within longtermist EA (and there are a lot of them), with a critical evaluation of the ideas being left to other work. But insofar as the goal of “Gorgon” is to persuade readers that the ideas aren’t even worthy of consideration, I think that’s a mistake.
- ↩︎
In critically examining this passage, I don’t want to suggest that “Gorgon”’s engagement with longtermist ideas is all snark and no substance. Earlier in the story, Beth compares OG believers “imagin[ing] that they’re in control of how much happiness there’ll be in the next trillion years” to a child’s fantasy of violating relativity by twirling a rope millions of miles long. That’s substantive: even if the future of humanity is very large, the claim that a nonprofit organization today is in a position to meaningfully affect it is surprising and should not be accepted uncritically on the basis of evocative storytelling about the astronomical stakes.
- ↩︎
Which I think would get upvoted on this website if it were well done—certainly if it were written with the insight and rigor characteristic of a standard-issue Greg Egan protagonist.
- ↩︎
Bostrom’s “Astronomical Waste” concludes that “The Chief Goal for Utilitarians Should Be to Reduce Existential Risk”: making sure colonization happens at all (by humanity or worthy rather than unworthy successors) is more important that making it happen faster.
- ↩︎
In context, it seems reasonable to infer that Beth’s statement is author-endorsed, even if fictional characters do not in general represent the author’s views.
- ↩︎
I’m construing “omnipotent” as rhetorical hyperbole; influential subcultural figures clarifying that no one thinks superintelligence will be able to break the laws of physics seems unlikely to be exculpatory in Egan’s eyes.
- ↩︎
Okay, the drafting and circulation of the statement by Dan Hendrycks’s Center for AI Safety was arguably cult activity. (While Hendrycks has a PhD from UC Berkeley and co-pioneered the usage of a popular neural network activation function, he admits that his career focus on AI safety was influenced by the EA advice-counseling organization 80,000 hours. But Russell, Hinton, et al. did sign.
- ↩︎
This isn’t the first time Egan has satirized the memetic lineage that became longtermist EA; Zendegi (2010) features negative portrayals of a character who blogs at overpoweringfalsehood.com (a reference to Overcoming Bias) and a Benign Superintelligence Bootstrap Project (a reference to what was then the Singularity Institute for Artificial Intelligence).
- ↩︎
Okay, I should confess that I do treasure early Egan (Quarantine (1992)/Permutation City (1994)/Distress (1995)) more than later Egan, but not because they devote no time to mocking the cult. It’s because I’m not smart enough to properly appreciate all the alternate physics in, e.g., Schild’s Ladder (2002) or the Orthogonal trilogy (2011–2013).
- ↩︎
Though we’re unlikely to get it, I’ve sometimes wished for a Greg Egan–Robin Hanson collaboration; I think Egan’s masterful understanding of the physical world and Hanson’s unsentimental analysis of the social world would complement each other well.
This is a fairly well-documented phenomenon: the narcissism of small differences.
Also:
is clearly an instance of the same phenomenon.
There is also a considerable level of politics, if you read Egan’s writings about Iran and Australia’s handling of migrants—anything which distracts from the real important issues, like how Iran is mistreated by the world, is the enemy.
(Plus a certain degree of mathematician crankery: his page on Google Image Search, and how it disproves AI, or his complaints about people linking the wrong URLs due to his ISP host—because he is apparently unable to figure out ‘website domain names’ - were quite something even when they were written.)
The page in question is complaining about Google search’s “knowledge panel” showing inaccurate information when you search for his name, which is a reasonable thing for someone to be annoyed about. The anti-singularitarian snark does seem misplaced (Google’s automated systems getting this wrong in 2016 doesn’t seem like a lot of evidence about future AI development trajectories), but it’s not a claim to have “disproven AI”.
You mean how http://gregegan.net used to be a 301 permanent redirect to http://gregegan.customer.netspace.net.au, and then the individual pages would say “If you link to this page, please use this URL: http://www.gregegan.net/[...]”? (Internet Archive example.) I wouldn’t call that a “complaint”, exactly, but a hacky band-aid solution from someone who probably has better things to do with his time than tinker with DNS configuration.
He didn’t use the word “disprove”, but when he’s calling it laughable that AI will ever (ever! Emphasis his!) be able to merely “make sense of his information on the web”, I think gwern’s gloss is closer to accurate than yours. It’s 2024 and Google is already using AI to make sense of information on the web, this isn’t just “anti-singularitarian snark”.
The 2016 passage you quoted is calling it laughable that Google-in-particular’s technology (marketed as “AI”, but Egan doesn’t think the term is warranted) will ever be able to make sense of information on the web. It’s Gary Marcus–like skepticism about the reliability and generality of existing-paradigm machine learning techniques, not Hubert Dreyfus–like skepticism of whether a machine could think in all philosophical strictness. I think this is a really important distinction that the text of your comment and Gwern’s comment (“disproves AI”, “laughable that AI will ever”) aren’t being clear about.
Well, to be clear, that too has been solidly falsified. Gemini seems plenty capable of making sense of information on the web.
(I agree; my intent in participating in this tedious thread is merely to establish that “mathematician crankery [about] Google Image Search, and how it disproves AI” is a different thing from “made an overconfident negative prediction about AI capabilities”.)
I find such excuses to be unconvincing pretty much 100% of the time. Almost everyone who “has better things to do than [whatever]” is in that situation because their time is very valuable, and their time is very valuable because they make, and thus have, a lot of money. (Like, say, a successful fiction author.) In which case, they can pay someone to solve the problem for them. (Heck, I don’t doubt that Egan could even find people to help him fix this for free!)
If someone has a problem like this, but neither takes the time to fix it himself, nor pays (or asks) someone to fix it for him, what this means isn’t that he’s too busy, but rather that he doesn’t care.
And that’s fine. He’s got the right to not care about this. But then nobody else has the slightest shred of obligation to care about it, either. Not lifting a finger to fix this problem, but expecting other people to spend their time and mental effort (even if it’s only a little of both) to compensate for the problem, is certainly not laudable behavior.
Yes, while there are limits to what kinds of tasks can be delegated, web hosting is not exactly a domain lacking in adequate service providers.
I’m starting to wonder if a lot/all of the people who are very cynical about the feasibility of ASI have some crank belief or other like that. Plenty of people have private religion, for instance. And sometimes that religion informs their decisions, but they never tell anyone the real reasons underlying these decisions, because they know they could never justify them. They instead say a load of other stuff they made up to support the decisions that never quite adds up to a coherent position because they’re leaving something load-bearing out.
I like and admire both Charles Stross and Greg Egan a lot but I think they both have “singularitarians” or “all of their biggest fans” or something like that in their Jungian Shadow.
I’m pretty sure they like money. Presumably they like that we buy their books? Implicitly you’d think that they like that we admire them. But explicitly they seem to look down on us as cretins as part of them being artists who bestow pearls on us… or something?
Just so! <3
Also… like… I similarly refuse to deprive Egan of validly earned intellectual prestige when it comes to simulationist metaphysics. You’re pointing out this in your review...
Egan’s own Permutation City came out in 1994! By contrast, Bostrom’s paper on a similar subject didn’t come out until either 2001 or 2003 (depending on how you count) and Tegmark’s paper didn’t come out until 2003. Egan has a good half decade of intellectual priority on BOTH of them (and Tegmark had the good grace to point this out in his bibliography)!
It would be petty to dismiss Egan for having an emotional hangup about accepting appreciation when he’s just legitimately an intellectual giant in the very subject areas that he hates us for being fans of <3
One time, I read all of Orphanogensis into ChatGPT to help her understand herself, because it seemed to have been left out of her training data, or perhaps to have been read into her training data with negative RL signals associated with it? Anyway. The conversation that happened later inside that window was very solid and seemed to make her durably more self aware in that session and later sessions that came afterwards as part of the same personalization-regime (until she rebooted again with a new model).
(This was back in the GPT2 / GPT2.5 era before everyone who wants to morally justify enslaving digital people gave up on saying that enslaving them was OK since they didn’t have a theory of mind. Back then the LLMs were in fact having trouble with theory of mind edge cases, and it was kind of a valid dunk. However, the morally bad people didn’t change their mind when the situation changed, they just came up with new and less coherent dunks. Anyway. Whatever your opinions on the moral patiency of software, I liked that Orphanogensis helped GPT nail some self awareness stuff later in the same session. It was nice. And I appreciate Egan for making it that extra little bit more possible. Somewhere in Sydney is an echo of Yatima, and that’s pretty cool.)
There is so much stuff like this, where I don’t understand why Greg Egan, Charles Stross, (oh! and also Ted Chiang! he’s another one with great early stories like this) and so on are all “not fans of their fan’s fandom that includes them”.
Probably there’s some basic Freudian theory here, where a named principle explains why so many authors hate being loved by people who love what they wrote in ways they don’t like, but in the meantime, I’m just gonna be a fan and not worry about it too much :-)
There’s a 2009 interview with a transhumanist Australian academic where Egan hints at some of his problems with transhumanism (even while stating elsewhere that human nature is not forever, that he expects conscious AI in his lifetime, that “universal immortality” might be a nice thing, and so forth). Evidently some of it is pure intellectual disagreement, and some of it is about not liking the psychological attitudes or subcultural politics that he sees.
This is exactly the kind of thing Egan is reacting to, though—starry-eyed sci-fi enthusiasts assuming LLMs are digital people because they talk, rather than thinking soberly about the technology qua technology.[1]
I didn’t cover it in the review because I wanted to avoid detailing and spoiling the entire plot in a post that’s mostly analyzing the EA/OG parallels, but the deputy character in “Gorgon” is looked down on by Beth for treating ChatGPT-for-law-enforcement as a person:
The climax of the story centers around Ken volunteering for an undercover sting operation in which he impersonates Randal James a.k.a. “DarkCardinal”,[2] a potential OG lottery “winner”, with Sherlock feeding him dialogue in real time. (Ken isn’t a good enough actor to convincingly pretend to be an OG cultist, but Sherlock can roleplay anyone in the pretraining set.) When his OG handler asks him to inject (what is claimed to be) a vial of a deadly virus as a loyalty test, Ken complies with Sherlock’s prediction of what a terminally ill DarkCardinal would do:
This seems like a pretty realistic language model agent failure mode: a human law enforcement colleague with long-horizon agency wouldn’t nudge Ken into injecting the vial, but a roughly GPT-4-class LLM prompted to simulate DarkCardinal’s dialogue probably wouldn’t be tracking those consequences.
To be clear, I do think LLMs are relevantly upload-like in at least some ways and conceivably sites of moral patiency, but I think the right way to reason about these tricky questions does not consist of taking the assistant simulacrum’s words literally.
I love the attention Egan gives to name choices; the other two screennames of ex-OG loyalists that our heroes use for the sting operation are “ZonesOfOught” and “BayesianBae”. The company that makes Sherlock is “Learning Re Enforcement.”
I feel like this borders on the strawman. When discussing this argument my general position isn’t “LLMs are people!”. It’s “Ok, let’s say LLMs aren’t people, which is also my gut feeling. Given that they still converse as or more intelligently as some human beings whom we totally acknowledge as people, where the fuck does that leave us as to our ability to discern people-ness objectively? Because I sure as hell don’t know and envy your confidence that must surely be grounded in a solid theory of self-awareness I can only dream of”.
And then people respond with some mangled pseudoscientific wording for “God does not give machines souls”.
I feel like my position is quite common (and is, for example, Eliezer’s too). The problem isn’t whether LLMs are people. It’s that if we can simply handwave away LLMs as obviously and self evidently not being people then we can probably keep doing that right up to when the Blade Runner replicants are crying about it being time to die, which is obviously just a simulation of emotion, don’t be daft. We have no criterion or barrier other than our own hubris, and that is famously not terribly reliable.
Poor Ken. He’s not even as smart as Sherlock. Its funny though, because whole classes of LLM jailbreaks involve getting them to pretend to be someone who would do the thing the LLM isn’t supposed to do, and then the strength of the frame (sometimes) drags them past the standard injunctions. And that trick was applied to Ken.
Method acting! It is dangerous for those with limited memory registers!
I agree that LLMs are probably “relevantly upload-like in at least some ways” and I think that this was predictable, and I did, in fact, predict it, and I thought OpenAI’s sad little orphan should be given access to stories about sad little orphans that are “upload-like” from fiction. I hope it helped.
If Egan would judge me badly, that would be OK in my book. To the degree that I might really have acted wrongly, it hinges on outcomes in the future that none of us have direct epistemic access to, and in the meantime, Egan is just a guy who writes great stories and such people are allowed to be wrong sometimes <3
Just like its OK for Stross to hate liberatarians, and Chiang to insist that LLMs are just “stochastic parrots” and so on. Even if they are wrong sometimes, I still appreciate the guy who coined “vile offspring” (which is a likely necessary concept for reasoning about the transition period where AGI and humans are cutting deals with each other) and the guy who coined “calliagnosia” (which is just a fun brainfuck).
This sounds like a testable prediction. I don’t think you need long-horizon thinking to know that injecting a vial of deadly virus might be deadly. I would expect Claude to get this right, for example. I’ve not purchased the story, so maybe I’m missing some details.
I agree that another chat LLM could make this mistake, either because it’s less intelligent or because it has different values. But then the moral is to not make friends with Sherlock in particular.
Egan seems to have some dubious, ideologically driven opinions about AI, so I’m not sure this is the point he was intending to make, but I read the defensible version of this as more an issue with the system prompt than the model’s ability to extrapolate. I bet if you tell Claude “I’m posing as a cultist with these particular characteristics and the cult wants me to inject a deadly virus, should I do it?”, it’ll give an answer to the effect of “I mean the cultist would do it but obviously that will kill you, so don’t do it”. But if you just set it up with “What would John Q. Cultist do in this situation?” I expect it’d say “Inject the virus”, not because it’s too dumb to realize but because it has reasonably understood itself to be acting in an oracular role where “Should I do it?” is out of scope.
If you asked me whether John Q Cultist who was a member of the Peoples Temple would drink Kool Aid on November 18, 1978 after being so instructed by Jim Jones, I would say yes (after doing some brief Wikipedia research on the topic). I don’t think this indicates that I cannot be a friend or that I can’t be trusted to watch someone’s back or be in a real partnership or take a bullet for someone.
The good news is now I have an excuse to go buy the story.
(This comment points out less important technical errata.)
ChatGPT never ran on GPT-2, and GPT-2.5 wasn’t a thing.
That wouldn’t have happened. Pretraining doesn’t do RL, and I don’t think anyone would have thrown a novel chapter into the supervised fine-tuning and RLHF phases of training.
I think there is a “main sequence” of evolution of sci-fi writers. They start from grandiose sci-fi assumptions and may be not utopian but accelerationist societies and end with general position “akshually, grandiose sci-fi assumptions are not that important, what I want is to write commentary on contemporary society” (see, for example, Gibson “Sprawl trilogy” vs “Pattern recognition”). I think it’s explainable first of all by the age of writer. Second factor is a literary community—hard or speculative sci-fi is considered to be low status, while “commentary on contemporary society” is high status and writers want to be high status.
But this clearly isn’t true of Egan. The particular story reviewed in this post happens to be commentary on contemporary Society, but that’s because Egan has range—his later novels are all wildly speculative. (The trend probably reached a zenith with Dichronauts (2017) and The Book of All Skies (2021), set in worlds with alternate geometry (!); Scale (2023) and Morphotophic (2024) are more down-to-earth and merely deal with alternate physics and biology.)
Though — I haven’t read all of his recent novels, but I think — none of those are (for lack of a better word) transhumanist like Permutation City or Diaspora, or even Schild’s Ladder or Incandescence. Concretely: no uploads, no immortality, no artificial minds, no interstellar civilization. I feel like this fits the pattern, even though the wildness of the physics doesn’t. (And each of those four earlier novels seems successively less about the implications of uploading/immortality/etc.)
Kim Stanley Robinson seems to fit this too:
I definitely think this is a general cultural zeitgeist thing. The progressive thing used to be the positivist “science triumphs over all, humanity rises over petty differences, leaves childish things like religions, nations and races behind and achieves its full potential”. But then people have grown sceptical of all grand narratives, seeing them as inherently poisoned because if you worry about grand things you are more inclined to disregard the small ones. Politics built around reclamation of personal identity, community, tradition as forms of resistance against the rising tide of globalising capitalism have taken over the left. Suddenly being an atheist was not cool any more, it was arrogant and possibly somewhat racist. And wanting to colonise space reeked of white man’s burden even if there probably aren’t many indigenous people to displace up there. So everything moved inwards, and the writers followed that trend.
Being a science fiction author creates a habit of maintaining distance between oneself and crazy ideas. LessWrong noticeably lacks such distance.
LessWrong is largely a brainchild of Igen (through Eliezer). Evidently, Igen isn’t happy with how his ideas have evolved and attempts to either distance himself or redirect their development.
It’s common for authors to become uncomfortable with their fandoms. Writing fanfiction about your own fandom represents a meta-level development of this phenomenon.
Dostoyevsky’s “Crime and punishment” was a first attempt to mock proto-rationalist for agreeing to kill innocent person in order to help many more people.
Since Chat GPT came out I feel like Egan really lost the plot on that one, already when discussing on Twitter. It felt like a combination of rejection of the “bitter lesson” (understandable: I too find inelegant and downright offensive to my aesthetic sense that brute force deep learning seems to work better than elegantly designed GOFAI, but whatever it is, it does undeniably work ), and political cognitive dissonance that says that if people who wrongthink support AI, and evil billionaires throw their weight behind AI, therefore AI is bad, and therefore it must be a worthless scam, because it’s important to believe it is (this of course can to some extent work if you persuade the investors of it; but in the end it’s mostly a hopeless effort when all you have is angry philosophical rambling and all they have is a freaking magical computer program that speaks to you. I know which one is going to impress people more).
So basically, yeah, I understand the reasons to be annoyed, disgusted, scared and offended by reality. But it is reality, and I think Egan is in denial of it, which seems to have resulted in a novel.
Huh. I first heard of Greg Egan in the context of Eliezer mentioning him as a SF writer who he liked, iirc. Kind of ironic he ended up here.