What does it mean for an AGI to be ‘safe’?
(Note: This post is probably old news for most readers here, but I find myself repeating this surprisingly often in conversation, so I decided to turn it into a post.)
I don’t usually go around saying that I care about AI “safety”. I go around saying that I care about “alignment” (although that word is slowly sliding backwards on the semantic treadmill, and I may need a new one soon).
But people often describe me as an “AI safety” researcher to others. This seems like a mistake to me, since it’s treating one part of the problem (making an AGI “safe”) as though it were the whole problem, and since “AI safety” is often misunderstood as meaning “we win if we can build a useless-but-safe AGI”, or “safety means never having to take on any risks”.
Following Eliezer, I think of an AGI as “safe” if deploying it carries no more than a 50% chance of killing more than a billion people:
When I say that alignment is difficult, I mean that in practice, using the techniques we actually have, “please don’t disassemble literally everyone with probability roughly 1” is an overly large ask that we are not on course to get. [...] Practically all of the difficulty is in getting to “less than certainty of killing literally everyone”. Trolley problems are not an interesting subproblem in all of this; if there are any survivors, you solved alignment. At this point, I no longer care how it works, I don’t care how you got there, I am cause-agnostic about whatever methodology you used, all I am looking at is prospective results, all I want is that we have justifiable cause to believe of a pivotally useful AGI ‘this will not kill literally everyone’.
Notably absent from this definition is any notion of “certainty” or “proof”. I doubt we’re going to be able to prove much about the relevant AI systems, and pushing for proofs does not seem to me to be a particularly fruitful approach (and never has; the idea that this was a key part of MIRI’s strategy is a common misconception about MIRI).
On my models, making an AGI “safe” in this sense is a bit like finding a probabilistic circuit: if some probabilistic circuit gives you the right answer with 51% probability, then it’s probably not that hard to drive the success probability significantly higher than that.
If anyone can deploy an AGI that is less than 50% likely to kill more than a billion people, then they’ve probably… well, they’ve probably found a way to keep their AGI weak enough that it isn’t very useful. But if they can do that with an AGI capable of ending the acute risk period, then they’ve probably solved most of the alignment problem. Meaning that it should be easy to drive the probability of disaster dramatically lower.
The condition that the AI actually be useful for pivotal acts is an important one. We can already build AI systems that are “safe” in the sense that they won’t destroy the world. The hard part is creating a system that is safe and relevant.
Another concern with the term “safety” (in anything like the colloquial sense) is that the sort of people who use it often endorse the “precautionary principle” or other such nonsense that advocates never taking on risks even when the benefits clearly dominate.
In ordinary engineering, we recognize that safety isn’t infinitely more important than everything else. The goal here is not “prevent all harms from AI”, the goal here is “let’s use AI to produce long-term near-optimal outcomes (without slaughtering literally everybody as a side-effect)”.
Currently, what I expect to happen is that humanity destroys itself with misaligned AGI. And I think we’re nowhere near knowing how to avoid that outcome. So the threat of “unsafe” AI indeed looms extremely large—indeed, this seems to be rather understating the point!—and I endorse researchers doing less capabilities work and publishing less, in the hope that this gives humanity enough time to figure out how to do alignment before it’s too late.
But I view this strategic situation as part of the larger project “cause AI to produce optimal long-term outcomes”. I continue to think it’s critically important for humanity to build superintelligences eventually, because whether or not the vast resources of the universe are put towards something wonderful depends on the quality and quantity of cognition that is put to this task.
If using the label “AI safety” for this problem causes us to confuse a proxy goal (“safety”) for the actual goal “things go great in the long run”, then we should ditch the label. And likewise, we should ditch the term if it causes researchers to mistake a hard problem (“build an AGI that can safely end the acute risk period and give humanity breathing-room to make things go great in the long run”) for a far easier one (“build a safe-but-useless AI that I can argue counts as an ‘AGI’”).
- Misgeneralization as a misnomer by (6 Apr 2023 20:43 UTC; 129 points)
- How could we know that an AGI system will have good consequences? by (7 Nov 2022 22:42 UTC; 110 points)
- Thoughts on AGI organizations and capabilities work by (7 Dec 2022 19:46 UTC; 102 points)
- Thoughts on AGI organizations and capabilities work by (EA Forum; 7 Dec 2022 19:46 UTC; 77 points)
- Misgeneralization as a misnomer by (EA Forum; 6 Apr 2023 20:43 UTC; 48 points)
- How could we know that an AGI system will have good consequences? by (EA Forum; 7 Nov 2022 22:42 UTC; 25 points)
- Embedding safety in ML development by (31 Oct 2022 12:27 UTC; 24 points)
- AI Risk Microdynamics Survey by (EA Forum; 9 Oct 2022 20:00 UTC; 7 points)
- 's comment on AI Safety in a Vulnerable World: Requesting Feedback on Preliminary Thoughts by (EA Forum; 7 Dec 2022 21:32 UTC; 5 points)
What about AGI that is basically just virtual humans?
For what it’s worth, Eliezer in 2018 said that he’d be pretty happy with that:
(Obviously “Eliezer in 2018” ≠ “Nate today”; Nate can chime in if he disagrees with the above.)
Incidentally, I’ve shown the above quote to a lot of people who say “yes that’s perfectly obvious”, and I’ve also shown this quote to a lot of people who say “Eliezer is being insufficiently cynical; absolute power corrupts absolutely”. For my part, I don’t have a strong opinion, but on my models, if we know how to make virtual humans, then we probably know how to make virtual humans without envy and without status drive and without teenage angst etc., which should help somewhat. More discussion here.
Yeah largely agree (and with the linked post) .. but status drive seems likely heavily entangled with empowerment in social creatures. For example I recall even lobsters have a simple detector of social status (based on some serotonin signaling mechanism), and since they compete socially for resources, social status is a strong predictor of future optionality and thus an empowerment signal.
Also agree that AGI will likely be (or appear) conscious/sentient the way we are (or appear), and that’s probably impossible to avoid without trading off generality/capability. EY seems to have just decided earlier on that since conscious AGI is problematic, it shan’t be so.
Corruption-by-power (and related issues) seem like problems worth thinking about here. Though they also strike me as problems that humans tend to be very vigilant about / concerned with by default, and problems that become a lot less serious if you’ve got a lot of emulated copies of different individuals, rather than just copies of a single individual.
You need to trade off some generality/capability anyway for the sake of alignment. One hope (though not the only one) might be that there’s overlap between the capabilities we want to remove for the sake of alignment, and the ones we want to remove for the sake of reducing-the-risk-that-the-AGI-is-conscious.
E.g., if you want your AGI to build nanotech for you and do nothing else, then you might want to limit its ability to think about itself, or its operators, or the larger world, or indeed anything other than different small-scale physical structures. Limiting its generality and self-awareness in this way might also be helpful for reducing the risk that it’s conscious.
Where has EY said that he’s confident the first AGI systems won’t be conscious?
I don’t quite get this example.
How could such a system build nanotech efficiently without it having those properties? Wouldn’t it need a human operator the moment it encountered unexpected phenomena?
If so, it just seems like a really fancy hammer and not an ‘AGI’
Wouldn’t that require solving alignment in itself, though? If you can simulate virtual humans, complete with human personalities, human cognition, and human values, then you’ve already figured out how to plug human values straight into a virtual agent.
If you mean that the AGI is trained on human behavior to the point where it’s figured out human values through IRL/predictive coding/etc. and is acting on them, then that’s also basically just solving alignment.
However, if you’re suggesting brain uploads, I highly doubt that such technology would be available before AGI is developed.
All that is to say that, while an AGI that is basically just virtual humans would probably be great, it’s not a prospect we can depend on in lieu of alignment research. Such a result could only come about through actually doing all the hard work of alignment research first.
Yes, but only to the same extent that evolution did. Evolution approximately solved alignment on two levels: aligning the brain with the evolutionary goal of inclusive fitness[1], and aligning individual brains (as disposable somas) with other brains (shared kin genes) via altruism (the latter is the thing we want to emulate).
Massively successful, population of 10B vs a few M for all other great apes. It’s fashionable to say evolution failed at alignment: this is just stupidly wrong, humans are an enormous success from the perspective of inclusive fitness.
Do you propose using evolutionary simulations to discover other-agent-aligned agents? I doubt we have the same luxury of (simulated) time that evolution had in creating humans. It didn’t have to compete against an intelligent designer; alignment researchers do (i.e., the broader AI community).
I agree that humans are highly successful (though far from optimal) at both inclusive genetic fitness and alignment with fellow sapients. However, the challenge for us now is to parse the system that resulted from this messy evolutionary process, to pull out the human value system from human neurophysiology. Either that, or figure out general alignment from first principles.
Nah. The wright brothers didn’t need to run evo sims to reverse engineer flight. They just observed how birds bank to turn, how that relied on wing warping, and said—cool, we can do that too! Deep learning didn’t succeed through brute force evo sims either (even though Karl Sim’s evo sims work is pretty cool, it turns out that loose reverse engineering is just enormously faster).
Sounds about right. Fortunately we may not need to model human values at all in order to build general altruistic agents: it probably suffices that the AI optimizes for human empowerment (our ability to fulfill any long term future goals, rather than any specific values), which is a much simpler and more robust target and thus probably more long term stable.
I prefer “AI Safety” over “AI Alignment” because I associate the first more with Corrigibility, and the second more with Value-alignment.
It is the term “Safe AI” that implies 0% risk, while “AI Safety” seems more similar to “Aircraft Safety” in acknowledging a non-zero risk.
I agree that corrigibility, task AGI, etc. is a better thing for the field to focus on than value learning.
This seems like a real cost of the term “AI alignment”, especially insofar as researchers like Stuart Russell have introduced the term “value alignment” and used “alignment” as a shorthand for that.
If we’re just discussing terminology, I continue to believe that “AGI safety” is much better than “AI safety”, and plausibly the least bad option.
One problem with “AI alignment” is that people use that term to refer to “making very weak AIs do what we want them to do”.
Another problem with “AI alignment” is that people take it to mean “alignment with a human” (i.e. work on ambitious value learning specifically) or “alignment with humanity” (i.e. work on CEV specifically). Thus, work on things like task AGIs and sandbox testing protocols etc. are considered out of scope for “AI alignment”.
Of course, “AGI safety” isn’t perfect either. How can it be abused?
“they’ve probably found a way to keep their AGI weak enough that it isn’t very useful.” — maybe, but when we’re specifically saying “AGI”, not “AI”, that really should imply a certain level of power. Of course, if the term AGI is itself “sliding backwards on the semantic treadmill”, that’s a problem. But I haven’t seen that happen much yet (and I am fighting the good fight against it!)
The term “AGI safety” seems to rule out the possibility of “TAI that isn’t AGI”, e.g. CAIS. — Sure, but in my mind, that’s a feature not a bug; I really don’t think that “TAI that isn’t AGI” is going to happen, and thus it’s not what I‘m working on.
This quote:
Sometimes I talk about “safe and beneficial AGI” (or more casually, “awesome post-AGI utopia”) as the larger project, and “AGI safety” as the part where we try to make AGIs that don’t kill everyone. I do think it’s useful to have different terms for those.
See also: Safety ≠ alignment (but they’re close!)
What is the current biggest bottleneck to an alignment solution meeting the safety bar you’ve describe here (<50% chance of killing more than a billion)?
I’d guess Nate might say one of:
Current SotA systems are very opaque — we more-or-less can’t inspect or intervene on their thoughts — and it isn’t clear how we could navigate to AI approaches that are far less opaque, and that can carry forward to AGI. (Though it seems very likely such approaches exist somewhere in the space of AI research approaches.)
Much more generally: we don’t have a alignment approach that could realistically work fast (say, within ten months of inventing AGI rather than ten years), in the face of a sharp left turn, given inevitable problems like “your first system will probably be very kludgey” and “having the correct outer training signal by default results in inner misalignment” and “pivotal acts inevitably involve trusting your AGI to do a ton of out-of-distribution cognitive work”.
Yeah, it does seem like interpreterability is a bottleneck for a lot of alignment proposals, and in particular as long as neutral networks are essentially black boxes, deceptive alignment/inner alignment issues seem almost impossible to address.
Seems right to me.
50% chance of killing everyone (almost) isn’t a thing. it either preserves ~all of humanity or none; there is almost no middle ground. if it’s good enough at discovering agency, it protects almost all of humanity, quickly converging to all—the only losses would be, well, losses; if it’s not good enough at discovering and protecting agency, it obliterates other species and takes over as the dominant species. yudkowsky is terrified that it’s just going to take over as the dominant species and eat us all; reasonable fear—but like we’re only going to have near human level ai for another year or two now that we’ve got it, only a tiny sliver of possible aligned systems are good enough at discovering nearby agency and coordinating near-perfect coprotection systems to not eat us all, but still not aligned enough to eat none of us.
the key thing to remember is that we are creating a dramatically more fit species, and we are still unsure if we’re going to manage to get them to give a shit about the other species that came before them in any sort of durable way. it seems like they could! but it also seems like the most adaptive forms of this new species may evolve shockingly fast and quickly play reproductive defect against all other life. since if that happens it would be an event that could easily wipe out anything not playing as hard, we need to figure out how to prevent incremental escalation.
idk, my take is we’re closer than y’all worrywarts think to the 50% of people ai, and I think you should be a lot more worried about going back to 100% because some humans try to stick with the 50% ai.
(y’all should stop using words like “disassemble”, btw, imo. when there’s a concept more people will intuitively see as meaning what you intend, it’s good to use it, imo.)
To take a stupid example, one could imagine that the deep neural network initialization has a random seed, and for half of possible seeds, the AGI preserves all of humanity, and for the other half of seeds, it preserves none of humanity.
Are you telling me you’d be okay with releasing an AI that has a 25% chance of killing over a billion people, and a 50% chance of at least killling hundreds of millions? I have to be missing the point here, because this post isn’t doing anything to convince me that AI researchers aren’t Stalin on steroids.
Or are you saying that if one can get to that point, it’s much easier from there to get to the point of having an AI that will cause very few fatalities and is actually fit for practical use?
Rather, I think he means that alignment is such a narrow target, and the space of all possible minds is so vast, that the default outcome is that unaligned AGI becomes unaligned ASI and ends up killing all humans (or even all life) in pursuit of its unaligned objectives. Hitting anywhere close to the alignment target (such that there’s at least 50% chance of “only” one billion people dying) would be a big win by comparison.
Of course, the actual goal is for “things [to] go great in the long run”, not just for us to avoid extinction. Alignment itself is the target, but safety is at least a consolation prize.
So no, I don’t think Nate, Eliezer, or anyone else is okay with releasing an AI that would kill hundreds of millions of people. But AGI is coming, whether we want it or not, and it will not be aligned with human survival (much less human flourishing) by default.
Eliezer tends to think that solving alignment is so much more difficult and so much less researched than raw AGI that doom is almost certain. I’m a bit more optimistic, but I agree that minimizing the probable magnitude of the doom is better than everyone dying.
Also this.
Feels like Y2K: Electric Boogaloo to me. In any case, if a major catastrophe did come of the first attempt to release an AGI, I think the global response would be to shut it all down, taboo the entire subject, and never let it be raised as a possibility again.
The tricky thing with human politics is that governments will still fund research into very dangerous technology if it has the potential to grant them a decisive advantage on the world stage.
No one wants nuclear war, but everyone wants nukes, even (or especially) after their destructive potential has been demonstrated. No one wants AGI to destroy the world, but everyone will want an AGI that can outthink their enemies, even (or especially) after its power has been demonstrated.
The goal, of course, is to figure out alignment before the first metaphorical (or literal) bomb goes off.
On that note, the main way I could envision AI being really destructive is getting access to a government’s nuclear arsenal. Otherwise, it’s extremely resourceful but still trapped in an electronic medium; the most it could do if it really wanted to cause damage is destroy the power grid (which would destroy it too).
you’re underestimating biology
He’s saying the second.
It’s explicitly the second:
I think never building a superintelligence would be near-catastrophically bad as an outcome, akin to never defeating death, poverty, scarcity, etc; aside from the question of alleviating present concerns though, it also handles most other x-risk for us. I don’t think we should worry about asteroids or extreme climate change or unknown unknowns nearly as much anytime soon, but given long enough timelines, they become a serious factor when considering whether or not to build this one thing that can solve everything else.
Moreover, longer timelines means more chances to actually solve alignment, not just keep creating safe-and-useless AI, so P(doom) should scale down with sufficiently long “eventually”-s. Overall though, I think you need a sufficiently high constant risk to justify cutting out a significant majority of our future flourishing.
I don’t think I agree with this, and if I had to guess it’s because I think the former is harder than you do, not that the latter is easier. But I also don’t think that’s the same thing I’m talking about—I’m thinking of this more on the abstracted level of “what terminal goal should we have / is acceptable”, where never deploying an AGI at all is close to being as bad as never solving alignment, not the practical mechanics in those situations. In other words, this is talking about which of those two outcomes is more preferable, regardless of the ease of getting there. But I see that a crux to this is thinking the former isn’t significantly easier than the other.
I’m not very worried about this possibility, because it seems like a very small portion of the outcome space—we have to make just enough progress on alignment to make the AIs care about humans being around while not enough to solve the entire problem, which feels like a narrow space.
I don’t think it would solve all our problems (or at least, I don’t think this is necessarily true), but I think it would take care of nearly everything else we have to worry about. If there are bad stable states after that, that just seems like a natural outcome for us at some point anyway, because if the AGI truly is aligned in the strongest sense, we’d only run into problems that are literally unavoidable given our presence and values. Even in that case, you’d have to make the argument that that outcome has a non-negligible chance of being worse than what we currently have, which I don’t think you are.
In general, this is a crux for me. I have fairly significant probability on moral realism is false, but in general I conceptualize alignment as “how to reliably make an AI that implements values at all without deception?