[...] In very broad terms, however, our approach to global risk mitigation is to think in terms of desired outcomes, and to ask: “What is the likeliest way that the outcome in question might occur?” We then repeat this process until we backchain to interventions that actors can take today. [...]
1. Long-run good outcomes. Ultimately, we want humanity to figure out the best possible long-run future and enact that kind of future, factoring in good outcomes for all sentient beings. However, there is currently very little we can say with confidence about what desirable long-term outcomes look like, or how best to achieve them; and if someone rushes to lock in a particular conception of “the best possible long-run future,” they’re likely to make catastrophic mistakes both in how they envision that goal and in how they implement it.
In order to avoid making critical decisions in haste and locking in flawed conclusions, humanity needs:
2. A stable period during which relevant actors can accumulate whatever capabilities and knowledge are required to reach robustly good conclusions about long-run outcomes. This might involve decisionmakers developing better judgment, insight, and reasoning skills in the future, solving the full alignment problem for fully autonomous AGI systems, and so on.
Given the difficulty of the task, we expect a successful stable period to require:
3. A preceding end to the acute risk period. If AGI carries a significant chance of causing an existential catastrophe over the next few decades, this forces a response under time pressure; but if actors attempt to make irreversible decisions about the long-term future under strong time pressure, we expect the result to be catastrophically bad. Conditioning on good outcomes, we therefore expect a two-step process where addressing acute existential risks takes temporal priority.
To end the acute risk period, we expect it to be necessary for actors to make use of:
4. A risk-mitigating technology. On our current view of the technological landscape, there are a number of plausible future technologies that could be leveraged to end the acute risk period.
We believe that the likeliest way to achieve a technology in this category sufficiently soon is through:
5. AGI-empowered technological development carried out by task-directed AGI systems. Depending on early AGI systems’ level of capital-intensiveness, on whether AGI is a late-paradigm or early-paradigm invention, and on a number of other factors, AGI might be developed by anything from a small Silicon Valley startup to a large-scale multinational collaboration. Regardless, we expect AGI to be developed before any other (meta)technology that can be employed to end the acute risk period, and if early AGI systems can be used safely at all, then we expect it to be possible for an AI-empowered project to safely automate a reasonably small set of concrete science and engineering tasks that are sufficient for ending the risk period. This requires:
6. Construction of minimal aligned AGI. We specify “minimal” because we consider success much more likely if developers attempt to build systems with the bare minimum of capabilities for ending the acute risk period. We expect AGI alignment to be highly difficult, and we expect additional capabilities to add substantially to this difficulty.
Added: “Minimal aligned AGI” means “aligned AGI that has the minimal necessary capabilities”; be sure not to misread it as “minimally aligned AGI”. Rob Bensinger adds: “The MIRI view isn’t ‘rather than making alignment your top priority and working really hard to over-engineer your system for safety, try to build a system with the bare minimum of capabilities’. It’s: ‘in addition to making alignment your top priority and working really hard to over-engineer your system for safety, also build the system to have the bare minimum of capabilities’.”
If an aligned system of this kind were developed, we would expect two factors to be responsible:
7a. A technological edge in AGI by an operationally adequate project. By “operationally adequate” we mean a project with strong opsec, research closure, trustworthy command, a commitment to the common good, security mindset, requisite resource levels, and heavy prioritization of alignment work. A project like this needs to have a large enough lead to be able to afford to spend a substantial amount of time on safety measures, as discussed at FLI’s Asilomar conference.
7b. A strong white-boxed system understanding on the part of the operationally adequate project during late AGI development. By this we mean that developers go into building AGI systems with a good understanding of how their systems decompose and solve particular cognitive problems, of the kinds of problems different parts of the system are working on, and of how all of the parts of the system interact.
On our current understanding of the alignment problem, developers need to be able to give a reasonable account of how all of the AGI-grade computation in their system is being allocated, similar to how secure software systems are built to allow security professionals to give a simple accounting of why the system has no unforeseen vulnerabilities. See “Security Mindset and Ordinary Paranoia” for more details.
Developers must be able to explicitly state and check all of the basic assumptions required for their account of the system’s alignment and effectiveness to hold. Additionally, they need to design and modify AGI systems only in ways that preserve understandability — that is, only allow system modifications that preserve developers’ ability to generate full accounts of what cognitive problems any given slice of the system is solving, and why the interaction of all of the system’s parts is both safe and effective.
Our view is that this kind of system understandability will in turn require:
8. Steering toward alignment-conducive AGI approaches. Leading AGI researchers and developers need to deliberately direct research efforts toward ensuring that the earliest AGI designs are relatively easy to understand and align.
We expect this to be a critical step, as we do not expect most approaches to AGI to be alignable after the fact without long, multi-year delays.
We then added:
We plan to say more in the future about the criteria for operationally adequate projects in 7a. We do not believe that any project meeting all of these conditions currently exists, though we see various ways that projects could reach this threshold.
The above breakdown only discusses what we view as the “mainline” success scenario. If we condition on good long-run outcomes, the most plausible explanation we can come up with cites an operationally adequate AI-empowered project ending the acute risk period, and appeals to the fact that those future AGI developers maintained a strong understanding of their system’s problem-solving work over the course of development, made use of advance knowledge about which AGI approaches conduce to that kind of understanding, and filtered on those approaches.
… and we said that MIRI’s strategy is to do research aimed at making “8. Steering toward alignment-conducive AGI approaches” easier:
For that reason, MIRI does research to intervene on 8 from various angles, such as by examining holes and anomalies in the field’s current understanding of real-world reasoning and decision-making. We hope to thereby reduce our own confusion about alignment-conducive AGI approaches and ultimately help make it feasible for developers to construct adequate “safety-stories” in an alignment setting. As we improve our understanding of the alignment problem, our aim is to share new insights and techniques with leading or up-and-coming developer groups, who we’re generally on good terms with. [...]
Replying to your points with that in mind, Mitchell:
I find this a little surprising. If someone had asked me what MIRI’s strategy is, I would have said that the core of it was still something like CEV, with topics like logical induction and new decision theory paradigms as technical framework issues.
Assuming the long-run future goes well, I expect humanity to eventually build an AGI system that does something vaguely CEV-like. (This will plausibly be the main goal of step 2 in the backchained list, and therefore the main way we get to step 1.)
But I wouldn’t say that MIRI’s research is at all about CEV. Before humanity gets to steps 1-2 (‘use CEV or something to make the long-term future awesome’), it needs to get past steps 3-6 (‘use limited task AGI to ensure that humanity doesn’t kill itself with AGI so we can proceed to take our time with far harder problems like “what even is CEV” and “how even in principle would one get an AI system to robustly do anything remotely like that, without some subtle or not-so-subtle disaster resulting”’).
We’ll have plenty of time to worry about steps 1-2 if we can figure out 3-6, so almost all of the alignment field’s attention should be on how to achieve 3-6 (or plausible prerequisites for 3-6, like 7-8).
I mean, part of the MIRI paradigm has always been that AGI alignment is grounded in how the human brain works, right? The mechanics of decision-making in human brains, are the starting point in constructing the mechanics of decision-making in an AGI that humans would call ‘aligned’.
Sounds false to me, and I’m not sure how this relates to CEV, logical induction, logical decision theory, etc.
It’s true that humans reason under logical uncertainty (and logical induction may be getting at a core thing about why that works), and it’s true that some of our instincts and reasoning are very likely to look FDT-ish insofar as they evolved in an environment where agents model each other and use similar reasoning processes.
But if humans somehow couldn’t apply their normal probablistic reasoning to math (and couldn’t do FDT-ish reasoning), and yet we could still somehow do math/AI research at all, then logical uncertainty and FDT would still be just as important, because they’d still be progress toward understanding how sufficiently smart AI systems reason about the world.
Think less “LI and FDT are how humans work, and we’re trying to make a machine that works like a human brain”, more “there’s such a thing as general-purpose problem-solving, some undiscovered aspects of this thing are probably simple, and something like LI and FDT may be (idealized versions of) how certain parts of this general-purpose problem-solving works, or may be steps toward finding such parts”.
And I would have thought that identifying how to do this, was still just research in progress in many directions, rather than something that had hit a dead end.
Last I checked, there are still MIRI researchers working on the major research programs we’ve done in the past (or working on research programs along those lines). But the organization as a whole isn’t trying to concentrate effort on any one of those lines of research, and Nate and Eliezer aren’t focusing their own efforts on any of the ongoing MIRI research projects, because Nate and Eliezer think AGI timelines are too short relative to those projects’ rates of progress to date.
(Or, ‘a critical mass of MIRI’s research leadership (which includes more people than Nate and Eliezer but assigns a lot of weight to Nate’s and Eliezer’s views) thinks AGI timelines are too short relative to those projects’ rates of progress to date’.)
From Nate/EY’s perspective, as I understand it, the problem isn’t ‘none of the stuff MIRI’s tried has borne fruit, or is currently bearing fruit’; it’s ‘the fruit is being borne too slowly for us to feel like MIRI’s current or past research efforts are on the critical path to humanity surviving and flourishing’.
They don’t know what is on the critical path, but they feel sufficiently pessimistic about the current things they’ve tried (relative to their timelines) that they’d rather work on things like aumanning right now and keep an eye out for better future alignment research ideas, rather than keep their focus on the ‘things we’ve already tried putting lots of effort into’ bucket.
(As with the vast majority of my comments, EY and Nate haven’t reviewed this, so I may be getting stuff about their views wrong.)
Before humanity gets to steps 1-2 (‘use CEV or something to make the long-term future awesome’), it needs to get past steps 3-6 (‘use limited task AGI to ensure that humanity doesn’t kill itself with AGI so we can proceed to take our time with far harder problems like “what even is CEV” and “how even in principle would one get an AI system to robustly do anything remotely like that, without some subtle or not-so-subtle disaster resulting”’).
I want to register my skepticism about this claim. Whereas it might naively seem that “put a strawberry on a plate” is easier to align than “extrapolated volition”, on a closer look there are reasons why it might be the other way around. Specifically, the notion of “utility function of given agent” is a natural concept that we should expect to have a relatively succinct mathematical description. This intuition is supported by the AIT definition of intelligence. On the other hand, “put a strawberry on a plate without undesirable side effects” is not a natural concept, since a lot of complexity is packed into the “undesirable side effects”. Therefore, while I see some lines of attack on both task AGI and extrapolated volition, the latter might well turn out easier.
And if humans had a utility function and we knew what that utility function was, we would not need CEV. Unfortunately extracting human preferences over out-of-distribution options and outcomes at dangerously high intelligence, using data gathered at safe levels of intelligence and a correspondingly narrower range of outcomes and options, when there exists no sensory ground truth about what humans want because human raters can be fooled or disassembled, seems pretty complicated. There is ultimately a rescuable truth about what we want, and CEV is my lengthy informal attempt at stating what that even is; but I would assess it as much, much, much more difficult than ‘corrigibility’ to train into a dangerously intelligent system using only training and data from safe levels of intelligence. (As is the central lethally difficult challenge of AGI alignment.)
If we were paperclip maximizers and knew what paperclips were, then yes, it would be easier to just build an offshoot paperclip maximizer.
I agree that it’s a tricky problem, but I think it’s probably tractable. The way PreDCA tries to deal with these difficulties is:
The AI can tell that, even before the AI was turned on, the physical universe was running certain programs.
Some of those programs are “agentic” programs.
Agentic programs have approximately well-defined utility functions.
Disassembling the humans doesn’t change anything, since it doesn’t affect the programs that were already running[1] before the AI was turned on.
Since we’re looking at agent-programs rather than specific agent-actions, there is much more ground for inference about novel situations.
Obviously, the concepts I’m using here (e.g. which programs are “running” or which programs are “agentic”) are non-trivial to define, but infra-Bayesian physicalism does allow us the define them (not without some caveats, but hopefully at least to a 1st approximation).
Yeah, I’m very interested in hearing counter-arguments to claims like this. I’ll say that although I think task AGI is easier, it’s not necessarily strictly easier, for the reason you mentioned.
Maybe a cruxier way of putting my claim is: Maybe corrigibility / task AGI / etc. is harder than CEV, but it just doesn’t seem realistic to me to try to achieve full, up-and-running CEV with the very first AGI systems you build, within a few months or a few years of humanity figuring out how to build AGI at all.
And I do think you need to get CEV up and running within a few months or a few years, if you want to both (1) avoid someone else destroying the world first, and (2) not use a “strawberry-aligned” AGI to prevent 1 from happening.
All of the options are to some extent a gamble, but corrigibility, task AGI, limited impact, etc. strike me as gambles that could actually realistically work out well for humanity even under extreme time pressure to deploy a system within a year or two of ‘we figure out how to build AGI’. I don’t think CEV is possible under that constraint. (And rushing CEV and getting it only 95% correct poses far larger s-risks than rushing low-impact non-operator-modeling strawberry AGI and getting it only 95% correct.)
Maybe corrigibility / task AGI / etc. is harder than CEV, but it just doesn’t seem realistic to me to try to achieve full, up-and-running CEV with the very first AGI systems you build, within a few months or a few years of humanity figuring out how to build AGI at all.
The way I imagine the win scenario is, we’re going to make a lot of progress in understanding alignment before we know how to build AGI. And, we’re going to do it by prioritizing understanding alignment modulo capability (the two are not really possible to cleanly separate, but it might be possible to separate them enough for this purpose). For example, we can assume the existence of algorithms with certain properties, s.t. these properties arguably imply the algorithms can be used as building-blocks for AGI, and then ask: given such algorithms, how would we build aligned AGI? Or, we can come up with some toy setting where we already know how to build “AGI” in some sense, and ask, how to make it aligned in that setting? And then, once we know how to build AGI in the real world, it would hopefully not be too difficult to translate the alignment method.
One caveat in all this is, if AGI is going to use deep learning, we might not know how to apply the lesson from the “oracle”/toy setting, because we don’t understand what deep learning is actually doing, and because of that, we wouldn’t be sure where to “slot” it in the correspondence/analogy s.t. the alignment method remains sound. But, mainstream researchers have been making progress on understanding what deep learning is actually doing, and IMO it’s plausible we will have a good mathematical handle on it before AGI.
And rushing CEV and getting it only 95% correct poses far larger s-risks than rushing low-impact non-operator-modeling strawberry AGI and getting it only 95% correct.
I’m not sure whether you mean “95% correct CEV has a lot of S-risk” or “95% correct CEV has a little S-risk, and even a tiny amount of S-risk is terrifying”? I think I agree with the latter but not with the former. (How specifically does 95% CEV produce S-risk? I can imagine something like “AI realizes we want non-zero amount of pain/suffering to exist, somehow miscalibrates the amount and creates a lot of pain/suffering” or “AI realizes we don’t want to die, and focuses on this goal on the expense of everything else, preserving us forever in a state of complete sensory deprivation”. But these scenarios don’t seem very likely?)
I’m not sure whether you mean “95% correct CEV has a lot of S-risk” or “95% correct CEV has a little S-risk, and even a tiny amount of S-risk is terrifying”?
Insofar as humans care about their AI being corrigible, we should expect some degree of corrigibility even from a CEV-maximizer. That, in turn, suggests at least some basin-of-attraction for values (at least along some dimensions), in the same way that corrigibility yields a basin-of-attraction.
(Though obviously that’s not an argument we’d want to make load-bearing without both theoretical and empirical evidence about how big the basin-of-attraction is along which dimensions.)
Conversely, it doesn’t seem realistic to define limited impact or corrigibility or whatever without relying on an awful lot of values information—like e.g. what sort of changes-to-the-world we do/don’t care about, what thing-in-the-environment the system is supposed to be corrigible with, etc.
Values seem like a necessary-and-sufficient component. Corrigibility/task architecture/etc doesn’t.
And rushing CEV and getting it only 95% correct poses far larger s-risks than rushing low-impact non-operator-modeling strawberry AGI and getting it only 95% correct.
Small but important point here: an estimate of CEV which is within 5% error everywhere does reasonably well; that gets us within 5% of our best possible outcome. The problem is when our estimate is waaayyy off in 5% of scenarios, especially if it’s off in the overestimate direction; then we’re in trouble.
Conversely, it doesn’t seem realistic to define limited impact or corrigibility or whatever without relying on an awful lot of values information—like e.g. what sort of changes-to-the-world we do/don’t care about, what thing-in-the-environment the system is supposed to be corrigible with, etc.
I suspect you could do this in a less value-loaded way if you’re somehow intervening on ‘what the AGI wants to pay attention to’, as opposed to just intervening on ‘what sorts of directions it wants to steer the world in’.
‘Only spend your cognition thinking about individual physical structures smaller than 10 micrometers’, ‘only spend your cognition thinking about the physical state of this particular five-cubic-foot volume of space’, etc. could eliminate most of the risk of ‘high-impact’ actions without forcing us to define human conceptions of ‘impact’, and without forcing the AI to do a bunch of human-modeling. But I don’t know what research path would produce the ability to do things like that.
(There’s still of course something that we’re trying to get the AGI to do, like make a nanofactory or make a scanning machine for WBE or make improved computing hardware. That part strikes me as intuitively more value-loaded than ‘only think about this particular volume of space’.
The difficulty with ‘only think about this particular volume of space’ is that it requires the ability to intervene on thoughts rather than outputs.)
‘Only spend your cognition thinking about individual physical structures smaller than 10 micrometers’, ‘only spend your cognition thinking about the physical state of this particular five-cubic-foot volume of space’, etc. could eliminate most of the risk of ‘high-impact’ actions without forcing us to define human conceptions of ‘impact’, and without forcing the AI to do a bunch of human-modeling.
I do not think that would do what you seem to think it would do. If something optimizes one little chunk of the world really hard, ignoring everything else, that doesn’t mean the rest of the world is unchanged; by default there are lots of side effects. E.g. if something is building nanotech in a 1m cube, ignoring everything outside the cube, at the very least I’d expect that dump nuke levels of waste heat into its immediate surroundings.
I agree that this would be scary if the system is, for example, as smart as physically possible. What I’m imagining is:
(1) if you find a way to ensure that the system is only weakly superhuman (e.g., it performs vast amounts of low-level-Google-engineer-quality reasoning, only rare short controlled bursts of von-Neumann-quality reasoning, and nothing dramatically above the von-Neumann level), and
(2) if you get the system to only care about thinking about this cube of space, and
(3) if you also somehow get the system to want to build the particular machine you care about,
then you can plausibly save the world, and (importantly) you’re not likely to destroy the world if you fail, assuming you really are correctly confident in 1, 2, and 3.
I think you can also get more safety margin if the cube is in Antarctica (or on the Moon?), if you’ve tried to seal it off from the environment to some degree, and if you actively monitor for things like toxic waste products, etc.
Notably, the “only care about thinking about this cube of space” part is important for a lot of the other safety features to work, like:
It’s a lot harder to get guarantees about the system’s intelligence if it’s optimizing the larger world (since it might then improve itself, or build a smart successor in its environment—good luck closing off all possible loopholes for what kinds of physical systems an AGI might build that count as “smart successors”, while still leaving it able to build nanotech!).
Likewise, it’s a lot harder to get guarantees that the system stably is optimizing what you want it to optimize, or stably has any specific internal property, if it’s willing and able to modify itself.
Part of why we can hope to notice, anticipate, and guard against bad side-effects like “waste products” is that the waste products aren’toptimized to have any particular effect on the external environment, and aren’t optimized to evade our efforts to notice, anticipate, or respond to the danger. For that reason, “An AGI that only terminally cares about the state of a certain cube of space, but does spend time thinking about the larger world”, is vastly scarier than an AGI that just-doesn’t-think in those directions.
If the system does start going off the rails, we’re a lot more likely to be able to shut it down if it isn’t thinking about us or about itself.
This makes me think that the “only care about thinking about certain things” part may be relatively important in order for a lot of other safety requirements to be tractable. It feels more “(realistically) necessary” than “sufficient” to me; but I do personally have a hunch (which hopefully we wouldn’t have to actually rely on as a safety assumption!) that the ability to do things in this reference class would get us, like, 80+% of the way to saving the world? (Dunno whether Eliezer or anyone else at MIRI would agree.)
Thank you for the long reply. The 2017 document postulates an “acute risk period” in which people don’t know how to align, and then a “stable period” once alignment theory is mature.
So if I’m getting the gist of things, rather than focus outright on the creation of a human-friendly superhuman AI, MIRI decided to focus on developing a more general theory and practice of alignment; and then once alignment theory is sufficiently mature and correct, one can focus on applying that theory to the specific crucial case, of aligning superhuman AI with extrapolated human volition.
But what’s happened is that we’re racing towards superhuman AI while the general theory of alignment is still crude, and this is a failure for the strategy of prioritizing general theory of alignment over the specific task of CEV.
The 2017 document postulates an “acute risk period” in which people don’t know how to align, and then a “stable period” once alignment theory is mature.
“Align” is a vague term. Let’s distinguish “strawberry alignment” (where we can safely and reliably use an AGI to execute a task like “Place, onto this particular plate here, two strawberries identical down to the cellular but not molecular level.”) from “CEV alignment” (where we can safely and reliably use an AGI to carry out a CEV-like procedure.)
Strawberry alignment seems vastly easier than CEV alignment to me, and I think it’s a similar task (in both difficulty and kind) to what we’ll need AGI to do in order to prevent humanity from killing itself with other AGIs.
The “acute risk period” is the period where we’re at risk of someone immediately destroying the world once they figure out how to build AGI (or once hardware scales to the required level, or whatever).
Figuring out how to do strawberry alignment isn’t sufficient for ending the acute risk period, since humanity then has to actually apply this knowledge and build and deploy an aligned AGI to execute some pivotal act. But I do think that figuring out strawberry alignment is the main obstacle; if we knew how to do that, I think humanity would have double-digit odds of surviving and flourishing.
The “stable period” is the period between “humanity successfully makes it the case that no one can destroy the world with AGI” and “humanity figures out how to ensure the long-term future is awesome”.
This stable period is very similar to the idea of a “long reflection” posited by Toby Ord and Will MacAskill, though the lengths of time they cite sound far too long to me, at least if we’re measuring in sidereal time. (With fast-running human whole-brain emulations, I think you could complete the entire “long reflection” in just a few sidereal years, without cutting any corners or taking any serious risks.)
So if I’m getting the gist of things, rather than focus outright on the creation of a human-friendly superhuman AI
“Human-friendly” and “superhuman” are both vague—strawberry-aligned task AGI is less robustly friendly, and less broadly capable, than CEV AGI. But strawberry-aligned AGI is still superhuman in at least some respects—heck, a pocket calculator is too—and it’s still friendly enough to do some impressive things without killing us.
Alignment is a matter of degree, and more ambitious tasks can be much harder to align.
MIRI decided to focus on developing a more general theory and practice of alignment;
Strawberry alignment is more “general” in the sense that we’re not trying to impart as many human-specific values into the AGI (though we still need to impart some).
But it’s less “general” in the sense that strawberry-grade alignment is likely to be much more brittle than CEV-grade alignment, and strawberry-grade alignment is much more dependent on us carefully picking exactly the right tasks and procedures to make the alignment work.
But what’s happened is that we’re racing towards superhuman AI while the general theory of alignment is still crude, and this is a failure for the strategy of prioritizing general theory of alignment over the specific task of CEV.
No. If we’d focused on CEV-grade alignment over strawberry-grade alignment, we’d be in even worse shape if anything.
The problem is that timelines look short, so it’s looking more difficult to figure out strawberry alignment in time to prevent human extinction. We should nonetheless make strawberry alignment humanity’s top priority, and put an enormous effort into it, because there isn’t a higher-probability path to good outcomes. (AFAICT, anyway. Having at least some people try to prove me wrong here obviously seems worthwhile too.)
CEV alignment is even harder than strawberry alignment (by a large margin), so short timelines are much more of a problem for the ‘rush straight to CEV alignment’ plan than for the ‘do strawberry alignment first, then CEV afterwards’ plan.
The “stable period” is supposed to be a period in which AGI already exists, but nothing like CEV has yet been implemented, and yet “no one can destroy the world with AGI”. How would that work? How do you prevent everyone in the whole wide world from developing unsafe AGI during the stable period?
Use strawberry alignment to melt all the computing clusters containing more than 4 GPUs. (Not actually the best thing to do with strawberry alignment, IMO, but anything you can do here is outside the Overton Window, so I picked something of which I could say “Oh but I wouldn’t actually do that” if pressed.)
I think there are multiple viable options, like the toy example EY uses:
I think that after AGI becomes possible at all and then possible to scale to dangerously superhuman levels, there will be, in the best-case scenario where a lot of other social difficulties got resolved, a 3-month to 2-year period where only a very few actors have AGI, meaning that it was socially possible for those few actors to decide to not just scale it to where it automatically destroys the world.
During this step, if humanity is to survive, somebody has to perform some feat that causes the world to not be destroyed in 3 months or 2 years when too many actors have access to AGI code that will destroy the world if its intelligence dial is turned up. This requires that the first actor or actors to build AGI, be able to do something with that AGI which prevents the world from being destroyed; if it didn’t require superintelligence, we could go do that thing right now, but no such human-doable act apparently exists so far as I can tell.
So we want the least dangerous, most easily aligned thing-to-do-with-an-AGI, but it does have to be a pretty powerful act to prevent the automatic destruction of Earth after 3 months or 2 years. It has to “flip the gameboard” rather than letting the suicidal game play out. We need to align the AGI that performs this pivotal act, to perform that pivotal act without killing everybody.
Parenthetically, no act powerful enough and gameboard-flipping enough to qualify is inside the Overton Window of politics, or possibly even of effective altruism, which presents a separate social problem. I usually dodge around this problem by picking an exemplar act which is powerful enough to actually flip the gameboard, but not the most alignable act because it would require way too many aligned details: Build self-replicating open-air nanosystems and use them (only) to melt all GPUs.
Since any such nanosystems would have to operate in the full open world containing lots of complicated details, this would require tons and tons of alignment work, is not the pivotal act easiest to align, and we should do some other thing instead. But the other thing I have in mind is also outside the Overton Window, just like this is. So I use “melt all GPUs” to talk about the requisite power level and the Overton Window problem level, both of which seem around the right levels to me, but the actual thing I have in mind is more alignable; and this way, I can reply to anyone who says “How dare you?!” by saying “Don’t worry, I don’t actually plan on doing that.”
It’s obviously a super core question; there’s no point aligning your AGI if someone else just builds unaligned AGI a few months later and kills everyone. The “alignment problem” humanity has as its urgent task is exactly the problem of aligning cognitive work that can be leveraged to prevent the proliferation of tech that destroys the world. Once you solve that, humanity can afford to take as much time as it needs to solve everything else.
The “alignment problem” humanity has as its urgent task is exactly the problem of aligning cognitive work that can be leveraged to prevent the proliferation of tech that destroys the world. Once you solve that, humanity can afford to take as much time as it needs to solve everything else.
OK, I disagree very much with that strategy. You’re basically saying, your aim is not to design ethical/friendly/aligned AI, you’re saying your aim is to design AI that can take over the world without killing anyone. Then once that is accomplished, you’ll settle down to figure out how that unlimited power would best be used.
To put it another way: Your optimistic scenario is one in which the organization that first achieves AGI uses it to take over the world, install a benevolent interim regime that monopolizes access to AGI without itself making a deadly mistake, and which then eventually figures out how to implement CEV (for example); and then it’s finally safe to have autonomous AGI.
I have a different optimistic scenario: We definitively figure out the theory of how to implement CEV before AGI even arises, and then spread that knowledge widely, so that whoever it is in the world that first achieves AGI, they will already know what they should do with it.
Both these scenarios are utopian in different ways. The first one says that flawed humans can directly wield superintelligence for a protracted period without screwing things up. The second one says that flawed humans can fully figure out how to safely wield superintelligence before it even arrives.
Meanwhile, in reality, we’ve already proceeded an unknown distance up the curve towards superintelligence, but none of the organizations leading the way has much of a plan for what happens, if their creations escape their control.
In this situation, I say that people whose aim is to create ethical/friendly/aligned superintelligence, should focus on solving that problem. Leave the techno-military strategizing to the national security elites of the world. It’s not a topic that you can avoid completely, but in the end it’s not your job to figure out how mere humans can safely and humanely wield superhuman power. It’s your job to design an autonomous superhuman power that is intrinsically safe and humane. To that end we have CEV, we have June Ku’s work, and more. We should be focusing there, while remaining engaged with the developments in mainstream AI, like language models. That’s my manifesto.
You’re basically saying, your aim is not to design ethical/friendly/aligned AI [...]
My goal is an awesome, eudaimonistic long-run future. To get there, I strongly predict that you need to build AGI that is fully aligned with human values. To get there, I strongly predict that you need to have decades of experience actually working with AGI, since early generations of systems will inevitably have bugs and limitations and it would be catastrophic to lock in the wrong future because we did a rush job.
(I’d also expect us to need the equivalent of subjective centuries of further progress on understanding stuff like “how human brains encode morality”, “how moral reasoning works”, etc.)
If it’s true that you need decades of working experience with AGI (and solutions to moral philosophy, psychology, etc.) to pull off CEV, then something clearly needs to happen to prevent humanity from destroying itself in those intervening decades.
I don’t like the characterization “your aim is not to design ethical/friendly/aligned AI”, because it’s picking an arbitrary cut-off for which parts of the plan count as my “aim”, and because it makes it sound like I’m trying to build unethical, unfriendly, unaligned AI instead. Rather, I think alignment is hard and we need a lot of time (including a lot of time with functioning AGIs) to have a hope of solving the maximal version of the problem. Which inherently requires humanity to do something about that dangerous “we can build AGI but not CEV-align it” time window.
I don’t think the best solution to that problem is for the field to throw up their hands and say “we’re scientists, it’s not our job to think about practicalities like that” and hope someone else takes care of it. We’re human beings, not science-bots; we should use our human intelligence to think about which course of action is likeliest to produce good outcomes, and do that.
[...] I have a different optimistic scenario: We definitively figure out the theory of how to implement CEV before AGI even arises, and then spread that knowledge widely, so that whoever it is in the world that first achieves AGI, they will already know what they should do with it. [...]
How long are your AGI timelines? I could imagine endorsing a plan like that if I were confident AGI is 200+ years away; but in fact I think it’s very unlikely to even be 100 years away, and my probability is mostly on scenarios like “it’s 8 years away” or “it’s 25 years away”.
I do agree that we’re likelier to see better outcomes if alignment knowledge is widespread, rather than being concentrated at a few big orgs. (All else equal, anyway. E.g., you might not want to do this if it somehow shortens timelines a bunch.)
But the kind of alignment knowledge I think matters here is primarily strawberry-grade alignment. It’s good if people widely know about things like CEV, but I wouldn’t advise a researcher to spend their 2022 working on advancing abstract CEV theory instead of advancing strawberry-grade alignment, if they’re equally interested in both problems and capable of working on either.
[...] To put it another way: Your optimistic scenario is one in which the organization that first achieves AGI uses it to take over the world, install a benevolent interim regime that monopolizes access to AGI without itself making a deadly mistake, and which then eventually figures out how to implement CEV (for example); and then it’s finally safe to have autonomous AGI. [...]
Talking about “taking over the world” strikes me as inviting a worst-argument-in-the-world style of reasoning. All the past examples of “taking over the world” weren’t cases where there’s some action A such that:
if no one does A, then all humans die and the future’s entire value is lost.
by comparison, it doesn’t matter much to anyone who does A; everyone stands to personally gain or lose a lot based on whether A is done, but they accrue similar value regardless of which actor does A. (Because there are vastly more than enough resources in the universe for everyone. The notion that this is a zero-sum conflict to grab a scarce pot of gold is calibrated to a very different world than the “ASI exists” world.)
doing A doesn’t necessarily mean that your idiosyncratic values will play a larger role in shaping the long-term future than anyone else’s, and in fact you’re bought into a specific plan aimed at preventing this outcome. (Because CEV, no-pot-of-gold, etc.)
I do think there are serious risks and moral hazards associated with a transition to that state of affairs. (I think this regardless of whether it’s a government or a private actor or an intergovernmental collaboration or whatever that’s running the task AGI.)
But I think it’s better for humanity to try to tackle those risks and moral hazards, than for humanity to just give up and die? And I haven’t heard a plausible-sounding plan for what humanity ought to do instead of addressing AGI proliferation somehow.
[...] you’re saying your aim is to design AI that can take over the world without killing anyone. Then once that is accomplished, you’ll settle down to figure out how that unlimited power would best be used. [...]
The ‘rush straight to CEV’ plan is exactly the same, except without the “settling down to figure out” part. Rushing straight to CEV isn’t doing any less ‘grabbing the world’s steering wheel’; it’s just taking less time to figure out which direction to go, before setting off.
This is the other reason it’s misleading to push on “taking over the world” noncentral fallacies here. Neither the rush-to-CEV plan nor the strawberries-followed-by-CEV plan is very much like people’s central prototypes for what “taking over the world” looks like (derived from the history of warfare or from Hollywood movies or what-have-you).
I’m tempted to point out that “rush-to-CEV” is more like “taking over the world” in many ways than “strawberries-followed-by-CEV” is. (Especially if “strawberries-followed-by-CEV” includes a step where the task-AGI operators engage in real, protracted debate and scholarly inquiry with the rest of the world to attempt to reach some level of consensus about whether CEV is a good idea, which version of CEV is best, etc.)
But IMO it makes more sense to just not go down the road of arguing about connotations, given that our language and intuitions aren’t calibrated to this totally-novel situation.
The first one says that flawed humans can directly wield superintelligence for a protracted period without screwing things up. The second one says that flawed humans can fully figure out how to safely wield superintelligence before it even arrives.
There’s clearly some length of time such that the cost of waiting that long to implement CEV outweighs the benefits. I think those are mostly costs of losing negentropy in the universe at large, though (as stars burn their fuel and/or move away from us via expansion), not costs like ‘the AGI operators get corrupted or make some major irreversible misstep because they waited an extra five years too long’.
I don’t know why you think the corruption/misstep risk of “waiting for an extra three years before running CEV” (for example) is larger than the ‘we might implement CEV wrong’ risk of rushing to implement CEV after zero years of tinkering with working AGI systems.
It seems like the sensible thing to do in this situation is to hope for the best, but plan for realistic outcomes that fall short of “the best”:
Realistically, there’s a strong chance (I would say: overwhelmingly strong) that we won’t be able to fully solve CEV before AGI arrives. So since our options in that case will be “strawberry-grade alignment, or just roll over and die”, let’s start by working on strawberry-grade alignment. Once we solve that problem, sure, we can shift resources into CEV. If you’re optimistic about ‘rush to CEV’, then IMO you should be even more optimistic that we can nail down strawberry alignment fast, at which point we should have made a lot of headway toward CEV alignment without gambling the whole future on our getting alignment perfect immediately and on the first try.
Likewise, realistically, there’s a strong chance (I would say overwhelming) that there will be some multi-year period where humanity can build AGI, but isn’t yet able to maximally align it. It would be good if we don’t just roll over and die in those worlds; so while we might hope for there to be no such period, we should make plans that are robust to such a period occurring.
There’s nothing about the strawberry plan that requires waiting, if it’s not net-beneficial to do so. You can in fact execute a ‘no one else can destroy the world with AGI’ pivotal act, start working on CEV, and then surprise yourself with how fast CEV falls into place and just go implement that in relatively short order.
What strawberry-ish actions do is give humanity the option of waiting. I think we’ll desperately need this option, but even if you disagree, I don’t think you should consider it net-negative to have the option available in the first place.
Meanwhile, in reality, we’ve already proceeded an unknown distance up the curve towards superintelligence, but none of the organizations leading the way has much of a plan for what happens, if their creations escape their control.
I agree with this.
In this situation, I say that people whose aim is to create ethical/friendly/aligned superintelligence, should focus on solving that problem. Leave the techno-military strategizing to the national security elites of the world. It’s not a topic that you can avoid completely, but in the end it’s not your job to figure out how mere humans can safely and humanely wield superhuman power. It’s your job to design an autonomous superhuman power that is intrinsically safe and humane.
This is the place where I say: “I don’t think the best solution to that problem is for the field to throw up their hands and say ‘we’re scientists, it’s not our job to think about practicalities like that’ and hope someone else takes care of it. We’re human beings, not science-bots; we should use our human intelligence to think about which course of action is likeliest to produce good outcomes, and do that.”
We aren’t helpless victims of our social roles, who must look at Research Path A and Research Path B and go, “Hmm, there’s a strategic consideration that says Path A is much more likely to save humanity than Path B… but thinking about practical strategic considerations is the kind of thing people wearing military uniforms do, not the kind of thing people wearing lab coats do.” You don’t have to do the non-world-saving thing just because it feels more normal or expected-of-your-role. You can actually just do the thing that makes sense.
(Again, maybe you disagree with me about what makes sense to do here. But the debate should be had at the level of ‘which factual beliefs suggest that A versus B is likeliest to produce a flourishing long-run future?‘, not at the level of ‘is it improper for scientists to update on information that sounds more geopolitics-y than linear-algebra-y?’. The real world doesn’t care about literary genres and roles; it’s all just facts, and disregarding a big slice of the world’s facts will tend to produce worse decisions.)
Very short. Longer timelines are logically possible, but I wouldn’t count on them.
As for this notion that something like CEV might require decades of thought to be figured out, or even might require decades of trial and error with AGI—that’s just a guess. I may be monotonous by saying June Ku over and over again (there are others whose work I intend to study too), but metaethical.ai is an extremely promising schema. If a serious effort was made to fill out that schema, while also critically but constructively examining its assumptions from all directions, who knows how far we’d get, and how quickly?
Another argument for shorter CEV timelines, is that AI itself may help complete the theory of CEV alignment. Along with the traditional powers of computation—calculation, optimization, deduction, etc—language models, despite their highly uneven output, are giving us a glimpse of what it will be like, to have AI contributing even to discussions like this. That day isn’t far off at all.
So from my perspective, long CEV timelines don’t actually seem likely. The other thing that I have great doubts about, is the stability of any world order in which a handful of humans - even if it were the NSA or the UN Security Council—use tool AGI to prevent everyone else from developing unsafe AGI. Targeting just one thing like GPUs won’t work forever because you can do computation in other ways; there will be great temptations to use tool AGI to carry out interventions that have nothing to do with stopping unsafe AGI… Anyone in such a position becomes a kind of world government.
The problem of “world government” or “what the fate of the world should be”, is something that CEV is meant to solve comprehensively, by providing an accurate first-principles extrapolation of humanity’s true volition, etc. But here, the scenario is an AGI-powered world takeover where the problems of governance and normativity have not been figured out. I’m not at all opposed to thinking about such scenarios; the next chapter of human affairs may indeed be one in which autonomous superhuman AI does not yet exist, but there are human elites possessing tool AGI. I just think that’s a highly unstable situation; making it stable and safe for humans would be hard to do without having something like CEV figured out; and because of the input of AI itself, one shouldn’t expect figuring out CEV to take a long time. I propose that it will be done relatively quickly, or not at all.
Another argument for shorter CEV timelines, is that AI itself may help complete the theory of CEV alignment.
I agree with this part. That’s why I’ve been saying ‘maybe we can do this in a few subjective decades or centuries’ rather than ‘maybe we can do this in a few subjective millennia.’ 🙂
But I’m mostly imagining AGI helping us get CEV theory faster. Which obviously requires a lot of prior alignment just to make use of the AGI safely, and to trust its outputs.
The idea is to keep ratcheting up alignment so we can safely make use of more capabilities—and then, in at least some cases, using those new capabilities to further improve and accelerate our next ratcheting-up of alignment.
Along with the traditional powers of computation—calculation, optimization, deduction, etc—language models, despite their highly uneven output, are giving us a glimpse of what it will be like, to have AI contributing even to discussions like this. That day isn’t far off at all.
… And that makes you feel optimistic about the rush-to-CEV option? ‘Unaligned AGIs or proto-AGIs generating plausble-sounding arguments about how to do CEV’ is not a scenario that makes me update toward humanity surviving.
there will be great temptations to use tool AGI to carry out interventions that have nothing to do with stopping unsafe AGI...
I share your pessimism about any group that would feel inclined to give in to those temptations, when the entire future light cone is at stake.
The scenario where we narrowly avoid paperclips by the skin of our teeth, and now have a chance to breathe and think things through before taking any major action, is indeed a fragile one in some respects, where there are many ways to rapidly destroy all of the future’s value by overextending. (E.g., using more AGI capabilities than you can currently align, or locking in contemporary human values that shouldn’t be locked in, or hastily picking the wrong theory or implementation of ‘how to do moral progress’.)
I don’t think we necessarily disagree about anything except ‘how hard is CEV’? It sounds to me like we’d mostly have the same intuitions conditional on ‘CEV is very hard’; but I take this very much for granted, so I’m freely focusing my attention on ‘OK, how could we make things go well given that fact?’.
I don’t think we necessarily disagree about anything except ‘how hard is CEV’? It sounds to me like we’d mostly have the same intuitions conditional on ‘CEV is very hard’
I disagree on the plausibility of a stop-the-world scheme. The way things are now is as safe or stable as they will ever be. I think it’s a better plan to use the rising tide of AI capabilities to flesh out CEV. In particular, since the details of CEV depend on not-yet-understood details of human cognition, one should think about how to use near-future AI power to extract those details from the available data regarding brains and behavior. But AI can contribute everywhere else in the development of CEV theory and practice too.
But I won’t try to change your thinking. In practice, MIRI’s work on simpler forms of alignment (and its work on logical induction, decision theory paradigms, etc) is surely relevant to a “CEV now” approach too. What I am wondering is, where is the de-facto center of gravity for a “CEV now” effort? I’ve said many times that June Ku’s blueprint is the best I’ve seen, but I don’t see anyone working to refine it. And there are other people whose work seems relevant and has a promising rigor, but I’m not sure how it best fits together.
edit: I do see a value to non-CEV alignment work, distinct from figuring out how to stop the world safely: and that is to reduce the risk arising from the general usage of advanced AI systems. So it is a contribution to AI safety.
CEV seems much much more difficult than strawberry alignment and I have written it off as a potential option for a baby’s first try at constructing superintelligence.
To be clear, I also expect that strawberry alignment is too hard for these babies and we’ll just die. But things can always be even more difficult, and with targeting CEV on a first try, it sure would be.
There’s zero room, there is negative room, to give away to luxury targets like CEV. They’re not even going to be able to do strawberry alignment, and if by some miracle we were able to do strawberry alignment and so humanity survived, that miracle would not suffice to get CEV right on the first try.
I highly doubt anything in this universe could be ‘intrinsically’ safe and humane. I’m not sure what such an object would even look like or how we could verify that.
Even purposefully designed safety systems for deep pocketed customers for very simple operations in a completely controlled environment, such as car manufacturer’s assembly lines with a robot arm assigned solely to tightening nuts, have safety systems that are not presumed to be 100% safe in any case. That’s why they have multiple layers of interlocks, panic buttons, etc.
And that’s for something millions of times simpler than a superhuman agi in ideal conditions.
Perhaps before even the strawberry, it would be interesting to consider the difficulty of the movable arm itself.
A useful distinction. Yet of the rare outcomes that follow current timelines without ending in ruin, I expect the most likely one falls into neither category. Instead it’s an AGI that behaves like a weird supersmart human that bootstrapped its humanity from language models with relatively little architecture support (for alignment), as a result of a theoretical miracle where things like that are the default outcome. Possibly from giving a language model autonomy/agency to debug its thinking while having notebooks and a working memory, tuning the model in the process. It’s not going to reliably do as it’s told, could be deceptive, yet possibly doesn’t turn everything into paperclips. Arguably it’s aligned, but only the way weird individual humans are aligned, which is noncentrally strawberry-aligned, and too-indirectly-to-use-the-term CEV-aligned.
Quoting from the “strategic background” summary we shared in 2017:
We then added:
… and we said that MIRI’s strategy is to do research aimed at making “8. Steering toward alignment-conducive AGI approaches” easier:
Replying to your points with that in mind, Mitchell:
Assuming the long-run future goes well, I expect humanity to eventually build an AGI system that does something vaguely CEV-like. (This will plausibly be the main goal of step 2 in the backchained list, and therefore the main way we get to step 1.)
But I wouldn’t say that MIRI’s research is at all about CEV. Before humanity gets to steps 1-2 (‘use CEV or something to make the long-term future awesome’), it needs to get past steps 3-6 (‘use limited task AGI to ensure that humanity doesn’t kill itself with AGI so we can proceed to take our time with far harder problems like “what even is CEV” and “how even in principle would one get an AI system to robustly do anything remotely like that, without some subtle or not-so-subtle disaster resulting”’).
We’ll have plenty of time to worry about steps 1-2 if we can figure out 3-6, so almost all of the alignment field’s attention should be on how to achieve 3-6 (or plausible prerequisites for 3-6, like 7-8).
Re logical induction and decision theory: the main goal of this kind of research has been “de-confusion” in the sense described in The Rocket Alignment Problem and our 2018 strategy update.
Sounds false to me, and I’m not sure how this relates to CEV, logical induction, logical decision theory, etc.
It’s true that humans reason under logical uncertainty (and logical induction may be getting at a core thing about why that works), and it’s true that some of our instincts and reasoning are very likely to look FDT-ish insofar as they evolved in an environment where agents model each other and use similar reasoning processes.
But if humans somehow couldn’t apply their normal probablistic reasoning to math (and couldn’t do FDT-ish reasoning), and yet we could still somehow do math/AI research at all, then logical uncertainty and FDT would still be just as important, because they’d still be progress toward understanding how sufficiently smart AI systems reason about the world.
Think less “LI and FDT are how humans work, and we’re trying to make a machine that works like a human brain”, more “there’s such a thing as general-purpose problem-solving, some undiscovered aspects of this thing are probably simple, and something like LI and FDT may be (idealized versions of) how certain parts of this general-purpose problem-solving works, or may be steps toward finding such parts”.
Last I checked, there are still MIRI researchers working on the major research programs we’ve done in the past (or working on research programs along those lines). But the organization as a whole isn’t trying to concentrate effort on any one of those lines of research, and Nate and Eliezer aren’t focusing their own efforts on any of the ongoing MIRI research projects, because Nate and Eliezer think AGI timelines are too short relative to those projects’ rates of progress to date.
(Or, ‘a critical mass of MIRI’s research leadership (which includes more people than Nate and Eliezer but assigns a lot of weight to Nate’s and Eliezer’s views) thinks AGI timelines are too short relative to those projects’ rates of progress to date’.)
From Nate/EY’s perspective, as I understand it, the problem isn’t ‘none of the stuff MIRI’s tried has borne fruit, or is currently bearing fruit’; it’s ‘the fruit is being borne too slowly for us to feel like MIRI’s current or past research efforts are on the critical path to humanity surviving and flourishing’.
They don’t know what is on the critical path, but they feel sufficiently pessimistic about the current things they’ve tried (relative to their timelines) that they’d rather work on things like aumanning right now and keep an eye out for better future alignment research ideas, rather than keep their focus on the ‘things we’ve already tried putting lots of effort into’ bucket.
(As with the vast majority of my comments, EY and Nate haven’t reviewed this, so I may be getting stuff about their views wrong.)
I want to register my skepticism about this claim. Whereas it might naively seem that “put a strawberry on a plate” is easier to align than “extrapolated volition”, on a closer look there are reasons why it might be the other way around. Specifically, the notion of “utility function of given agent” is a natural concept that we should expect to have a relatively succinct mathematical description. This intuition is supported by the AIT definition of intelligence. On the other hand, “put a strawberry on a plate without undesirable side effects” is not a natural concept, since a lot of complexity is packed into the “undesirable side effects”. Therefore, while I see some lines of attack on both task AGI and extrapolated volition, the latter might well turn out easier.
And if humans had a utility function and we knew what that utility function was, we would not need CEV. Unfortunately extracting human preferences over out-of-distribution options and outcomes at dangerously high intelligence, using data gathered at safe levels of intelligence and a correspondingly narrower range of outcomes and options, when there exists no sensory ground truth about what humans want because human raters can be fooled or disassembled, seems pretty complicated. There is ultimately a rescuable truth about what we want, and CEV is my lengthy informal attempt at stating what that even is; but I would assess it as much, much, much more difficult than ‘corrigibility’ to train into a dangerously intelligent system using only training and data from safe levels of intelligence. (As is the central lethally difficult challenge of AGI alignment.)
If we were paperclip maximizers and knew what paperclips were, then yes, it would be easier to just build an offshoot paperclip maximizer.
I agree that it’s a tricky problem, but I think it’s probably tractable. The way PreDCA tries to deal with these difficulties is:
The AI can tell that, even before the AI was turned on, the physical universe was running certain programs.
Some of those programs are “agentic” programs.
Agentic programs have approximately well-defined utility functions.
Disassembling the humans doesn’t change anything, since it doesn’t affect the programs that were already running[1] before the AI was turned on.
Since we’re looking at agent-programs rather than specific agent-actions, there is much more ground for inference about novel situations.
Obviously, the concepts I’m using here (e.g. which programs are “running” or which programs are “agentic”) are non-trivial to define, but infra-Bayesian physicalism does allow us the define them (not without some caveats, but hopefully at least to a 1st approximation).
More precisely, I am looking at agents which could prevent the AI from becoming turned on, this is what I call “precursors”.
Yeah, I’m very interested in hearing counter-arguments to claims like this. I’ll say that although I think task AGI is easier, it’s not necessarily strictly easier, for the reason you mentioned.
Maybe a cruxier way of putting my claim is: Maybe corrigibility / task AGI / etc. is harder than CEV, but it just doesn’t seem realistic to me to try to achieve full, up-and-running CEV with the very first AGI systems you build, within a few months or a few years of humanity figuring out how to build AGI at all.
And I do think you need to get CEV up and running within a few months or a few years, if you want to both (1) avoid someone else destroying the world first, and (2) not use a “strawberry-aligned” AGI to prevent 1 from happening.
All of the options are to some extent a gamble, but corrigibility, task AGI, limited impact, etc. strike me as gambles that could actually realistically work out well for humanity even under extreme time pressure to deploy a system within a year or two of ‘we figure out how to build AGI’. I don’t think CEV is possible under that constraint. (And rushing CEV and getting it only 95% correct poses far larger s-risks than rushing low-impact non-operator-modeling strawberry AGI and getting it only 95% correct.)
The way I imagine the win scenario is, we’re going to make a lot of progress in understanding alignment before we know how to build AGI. And, we’re going to do it by prioritizing understanding alignment modulo capability (the two are not really possible to cleanly separate, but it might be possible to separate them enough for this purpose). For example, we can assume the existence of algorithms with certain properties, s.t. these properties arguably imply the algorithms can be used as building-blocks for AGI, and then ask: given such algorithms, how would we build aligned AGI? Or, we can come up with some toy setting where we already know how to build “AGI” in some sense, and ask, how to make it aligned in that setting? And then, once we know how to build AGI in the real world, it would hopefully not be too difficult to translate the alignment method.
One caveat in all this is, if AGI is going to use deep learning, we might not know how to apply the lesson from the “oracle”/toy setting, because we don’t understand what deep learning is actually doing, and because of that, we wouldn’t be sure where to “slot” it in the correspondence/analogy s.t. the alignment method remains sound. But, mainstream researchers have been making progress on understanding what deep learning is actually doing, and IMO it’s plausible we will have a good mathematical handle on it before AGI.
I’m not sure whether you mean “95% correct CEV has a lot of S-risk” or “95% correct CEV has a little S-risk, and even a tiny amount of S-risk is terrifying”? I think I agree with the latter but not with the former. (How specifically does 95% CEV produce S-risk? I can imagine something like “AI realizes we want non-zero amount of pain/suffering to exist, somehow miscalibrates the amount and creates a lot of pain/suffering” or “AI realizes we don’t want to die, and focuses on this goal on the expense of everything else, preserving us forever in a state of complete sensory deprivation”. But these scenarios don’t seem very likely?)
The latter, as I was imagining “95%”.
Insofar as humans care about their AI being corrigible, we should expect some degree of corrigibility even from a CEV-maximizer. That, in turn, suggests at least some basin-of-attraction for values (at least along some dimensions), in the same way that corrigibility yields a basin-of-attraction.
(Though obviously that’s not an argument we’d want to make load-bearing without both theoretical and empirical evidence about how big the basin-of-attraction is along which dimensions.)
Conversely, it doesn’t seem realistic to define limited impact or corrigibility or whatever without relying on an awful lot of values information—like e.g. what sort of changes-to-the-world we do/don’t care about, what thing-in-the-environment the system is supposed to be corrigible with, etc.
Values seem like a necessary-and-sufficient component. Corrigibility/task architecture/etc doesn’t.
Small but important point here: an estimate of CEV which is within 5% error everywhere does reasonably well; that gets us within 5% of our best possible outcome. The problem is when our estimate is waaayyy off in 5% of scenarios, especially if it’s off in the overestimate direction; then we’re in trouble.
I suspect you could do this in a less value-loaded way if you’re somehow intervening on ‘what the AGI wants to pay attention to’, as opposed to just intervening on ‘what sorts of directions it wants to steer the world in’.
‘Only spend your cognition thinking about individual physical structures smaller than 10 micrometers’, ‘only spend your cognition thinking about the physical state of this particular five-cubic-foot volume of space’, etc. could eliminate most of the risk of ‘high-impact’ actions without forcing us to define human conceptions of ‘impact’, and without forcing the AI to do a bunch of human-modeling. But I don’t know what research path would produce the ability to do things like that.
(There’s still of course something that we’re trying to get the AGI to do, like make a nanofactory or make a scanning machine for WBE or make improved computing hardware. That part strikes me as intuitively more value-loaded than ‘only think about this particular volume of space’.
The difficulty with ‘only think about this particular volume of space’ is that it requires the ability to intervene on thoughts rather than outputs.)
I do not think that would do what you seem to think it would do. If something optimizes one little chunk of the world really hard, ignoring everything else, that doesn’t mean the rest of the world is unchanged; by default there are lots of side effects. E.g. if something is building nanotech in a 1m cube, ignoring everything outside the cube, at the very least I’d expect that dump nuke levels of waste heat into its immediate surroundings.
I agree that this would be scary if the system is, for example, as smart as physically possible. What I’m imagining is:
(1) if you find a way to ensure that the system is only weakly superhuman (e.g., it performs vast amounts of low-level-Google-engineer-quality reasoning, only rare short controlled bursts of von-Neumann-quality reasoning, and nothing dramatically above the von-Neumann level), and
(2) if you get the system to only care about thinking about this cube of space, and
(3) if you also somehow get the system to want to build the particular machine you care about,
then you can plausibly save the world, and (importantly) you’re not likely to destroy the world if you fail, assuming you really are correctly confident in 1, 2, and 3.
I think you can also get more safety margin if the cube is in Antarctica (or on the Moon?), if you’ve tried to seal it off from the environment to some degree, and if you actively monitor for things like toxic waste products, etc.
Notably, the “only care about thinking about this cube of space” part is important for a lot of the other safety features to work, like:
It’s a lot harder to get guarantees about the system’s intelligence if it’s optimizing the larger world (since it might then improve itself, or build a smart successor in its environment—good luck closing off all possible loopholes for what kinds of physical systems an AGI might build that count as “smart successors”, while still leaving it able to build nanotech!).
Likewise, it’s a lot harder to get guarantees that the system stably is optimizing what you want it to optimize, or stably has any specific internal property, if it’s willing and able to modify itself.
Part of why we can hope to notice, anticipate, and guard against bad side-effects like “waste products” is that the waste products aren’t optimized to have any particular effect on the external environment, and aren’t optimized to evade our efforts to notice, anticipate, or respond to the danger. For that reason, “An AGI that only terminally cares about the state of a certain cube of space, but does spend time thinking about the larger world”, is vastly scarier than an AGI that just-doesn’t-think in those directions.
If the system does start going off the rails, we’re a lot more likely to be able to shut it down if it isn’t thinking about us or about itself.
This makes me think that the “only care about thinking about certain things” part may be relatively important in order for a lot of other safety requirements to be tractable. It feels more “(realistically) necessary” than “sufficient” to me; but I do personally have a hunch (which hopefully we wouldn’t have to actually rely on as a safety assumption!) that the ability to do things in this reference class would get us, like, 80+% of the way to saving the world? (Dunno whether Eliezer or anyone else at MIRI would agree.)
Thank you for the long reply. The 2017 document postulates an “acute risk period” in which people don’t know how to align, and then a “stable period” once alignment theory is mature.
So if I’m getting the gist of things, rather than focus outright on the creation of a human-friendly superhuman AI, MIRI decided to focus on developing a more general theory and practice of alignment; and then once alignment theory is sufficiently mature and correct, one can focus on applying that theory to the specific crucial case, of aligning superhuman AI with extrapolated human volition.
But what’s happened is that we’re racing towards superhuman AI while the general theory of alignment is still crude, and this is a failure for the strategy of prioritizing general theory of alignment over the specific task of CEV.
Is that vaguely what happened?
“Align” is a vague term. Let’s distinguish “strawberry alignment” (where we can safely and reliably use an AGI to execute a task like “Place, onto this particular plate here, two strawberries identical down to the cellular but not molecular level.”) from “CEV alignment” (where we can safely and reliably use an AGI to carry out a CEV-like procedure.)
Strawberry alignment seems vastly easier than CEV alignment to me, and I think it’s a similar task (in both difficulty and kind) to what we’ll need AGI to do in order to prevent humanity from killing itself with other AGIs.
The “acute risk period” is the period where we’re at risk of someone immediately destroying the world once they figure out how to build AGI (or once hardware scales to the required level, or whatever).
Figuring out how to do strawberry alignment isn’t sufficient for ending the acute risk period, since humanity then has to actually apply this knowledge and build and deploy an aligned AGI to execute some pivotal act. But I do think that figuring out strawberry alignment is the main obstacle; if we knew how to do that, I think humanity would have double-digit odds of surviving and flourishing.
The “stable period” is the period between “humanity successfully makes it the case that no one can destroy the world with AGI” and “humanity figures out how to ensure the long-term future is awesome”.
This stable period is very similar to the idea of a “long reflection” posited by Toby Ord and Will MacAskill, though the lengths of time they cite sound far too long to me, at least if we’re measuring in sidereal time. (With fast-running human whole-brain emulations, I think you could complete the entire “long reflection” in just a few sidereal years, without cutting any corners or taking any serious risks.)
“Human-friendly” and “superhuman” are both vague—strawberry-aligned task AGI is less robustly friendly, and less broadly capable, than CEV AGI. But strawberry-aligned AGI is still superhuman in at least some respects—heck, a pocket calculator is too—and it’s still friendly enough to do some impressive things without killing us.
Alignment is a matter of degree, and more ambitious tasks can be much harder to align.
Strawberry alignment is more “general” in the sense that we’re not trying to impart as many human-specific values into the AGI (though we still need to impart some).
But it’s less “general” in the sense that strawberry-grade alignment is likely to be much more brittle than CEV-grade alignment, and strawberry-grade alignment is much more dependent on us carefully picking exactly the right tasks and procedures to make the alignment work.
No. If we’d focused on CEV-grade alignment over strawberry-grade alignment, we’d be in even worse shape if anything.
The problem is that timelines look short, so it’s looking more difficult to figure out strawberry alignment in time to prevent human extinction. We should nonetheless make strawberry alignment humanity’s top priority, and put an enormous effort into it, because there isn’t a higher-probability path to good outcomes. (AFAICT, anyway. Having at least some people try to prove me wrong here obviously seems worthwhile too.)
CEV alignment is even harder than strawberry alignment (by a large margin), so short timelines are much more of a problem for the ‘rush straight to CEV alignment’ plan than for the ‘do strawberry alignment first, then CEV afterwards’ plan.
The “stable period” is supposed to be a period in which AGI already exists, but nothing like CEV has yet been implemented, and yet “no one can destroy the world with AGI”. How would that work? How do you prevent everyone in the whole wide world from developing unsafe AGI during the stable period?
Use strawberry alignment to melt all the computing clusters containing more than 4 GPUs. (Not actually the best thing to do with strawberry alignment, IMO, but anything you can do here is outside the Overton Window, so I picked something of which I could say “Oh but I wouldn’t actually do that” if pressed.)
I think there are multiple viable options, like the toy example EY uses:
It’s obviously a super core question; there’s no point aligning your AGI if someone else just builds unaligned AGI a few months later and kills everyone. The “alignment problem” humanity has as its urgent task is exactly the problem of aligning cognitive work that can be leveraged to prevent the proliferation of tech that destroys the world. Once you solve that, humanity can afford to take as much time as it needs to solve everything else.
OK, I disagree very much with that strategy. You’re basically saying, your aim is not to design ethical/friendly/aligned AI, you’re saying your aim is to design AI that can take over the world without killing anyone. Then once that is accomplished, you’ll settle down to figure out how that unlimited power would best be used.
To put it another way: Your optimistic scenario is one in which the organization that first achieves AGI uses it to take over the world, install a benevolent interim regime that monopolizes access to AGI without itself making a deadly mistake, and which then eventually figures out how to implement CEV (for example); and then it’s finally safe to have autonomous AGI.
I have a different optimistic scenario: We definitively figure out the theory of how to implement CEV before AGI even arises, and then spread that knowledge widely, so that whoever it is in the world that first achieves AGI, they will already know what they should do with it.
Both these scenarios are utopian in different ways. The first one says that flawed humans can directly wield superintelligence for a protracted period without screwing things up. The second one says that flawed humans can fully figure out how to safely wield superintelligence before it even arrives.
Meanwhile, in reality, we’ve already proceeded an unknown distance up the curve towards superintelligence, but none of the organizations leading the way has much of a plan for what happens, if their creations escape their control.
In this situation, I say that people whose aim is to create ethical/friendly/aligned superintelligence, should focus on solving that problem. Leave the techno-military strategizing to the national security elites of the world. It’s not a topic that you can avoid completely, but in the end it’s not your job to figure out how mere humans can safely and humanely wield superhuman power. It’s your job to design an autonomous superhuman power that is intrinsically safe and humane. To that end we have CEV, we have June Ku’s work, and more. We should be focusing there, while remaining engaged with the developments in mainstream AI, like language models. That’s my manifesto.
My goal is an awesome, eudaimonistic long-run future. To get there, I strongly predict that you need to build AGI that is fully aligned with human values. To get there, I strongly predict that you need to have decades of experience actually working with AGI, since early generations of systems will inevitably have bugs and limitations and it would be catastrophic to lock in the wrong future because we did a rush job.
(I’d also expect us to need the equivalent of subjective centuries of further progress on understanding stuff like “how human brains encode morality”, “how moral reasoning works”, etc.)
If it’s true that you need decades of working experience with AGI (and solutions to moral philosophy, psychology, etc.) to pull off CEV, then something clearly needs to happen to prevent humanity from destroying itself in those intervening decades.
I don’t like the characterization “your aim is not to design ethical/friendly/aligned AI”, because it’s picking an arbitrary cut-off for which parts of the plan count as my “aim”, and because it makes it sound like I’m trying to build unethical, unfriendly, unaligned AI instead. Rather, I think alignment is hard and we need a lot of time (including a lot of time with functioning AGIs) to have a hope of solving the maximal version of the problem. Which inherently requires humanity to do something about that dangerous “we can build AGI but not CEV-align it” time window.
I don’t think the best solution to that problem is for the field to throw up their hands and say “we’re scientists, it’s not our job to think about practicalities like that” and hope someone else takes care of it. We’re human beings, not science-bots; we should use our human intelligence to think about which course of action is likeliest to produce good outcomes, and do that.
How long are your AGI timelines? I could imagine endorsing a plan like that if I were confident AGI is 200+ years away; but in fact I think it’s very unlikely to even be 100 years away, and my probability is mostly on scenarios like “it’s 8 years away” or “it’s 25 years away”.
I do agree that we’re likelier to see better outcomes if alignment knowledge is widespread, rather than being concentrated at a few big orgs. (All else equal, anyway. E.g., you might not want to do this if it somehow shortens timelines a bunch.)
But the kind of alignment knowledge I think matters here is primarily strawberry-grade alignment. It’s good if people widely know about things like CEV, but I wouldn’t advise a researcher to spend their 2022 working on advancing abstract CEV theory instead of advancing strawberry-grade alignment, if they’re equally interested in both problems and capable of working on either.
Talking about “taking over the world” strikes me as inviting a worst-argument-in-the-world style of reasoning. All the past examples of “taking over the world” weren’t cases where there’s some action A such that:
if no one does A, then all humans die and the future’s entire value is lost.
by comparison, it doesn’t matter much to anyone who does A; everyone stands to personally gain or lose a lot based on whether A is done, but they accrue similar value regardless of which actor does A. (Because there are vastly more than enough resources in the universe for everyone. The notion that this is a zero-sum conflict to grab a scarce pot of gold is calibrated to a very different world than the “ASI exists” world.)
doing A doesn’t necessarily mean that your idiosyncratic values will play a larger role in shaping the long-term future than anyone else’s, and in fact you’re bought into a specific plan aimed at preventing this outcome. (Because CEV, no-pot-of-gold, etc.)
I do think there are serious risks and moral hazards associated with a transition to that state of affairs. (I think this regardless of whether it’s a government or a private actor or an intergovernmental collaboration or whatever that’s running the task AGI.)
But I think it’s better for humanity to try to tackle those risks and moral hazards, than for humanity to just give up and die? And I haven’t heard a plausible-sounding plan for what humanity ought to do instead of addressing AGI proliferation somehow.
The ‘rush straight to CEV’ plan is exactly the same, except without the “settling down to figure out” part. Rushing straight to CEV isn’t doing any less ‘grabbing the world’s steering wheel’; it’s just taking less time to figure out which direction to go, before setting off.
This is the other reason it’s misleading to push on “taking over the world” noncentral fallacies here. Neither the rush-to-CEV plan nor the strawberries-followed-by-CEV plan is very much like people’s central prototypes for what “taking over the world” looks like (derived from the history of warfare or from Hollywood movies or what-have-you).
I’m tempted to point out that “rush-to-CEV” is more like “taking over the world” in many ways than “strawberries-followed-by-CEV” is. (Especially if “strawberries-followed-by-CEV” includes a step where the task-AGI operators engage in real, protracted debate and scholarly inquiry with the rest of the world to attempt to reach some level of consensus about whether CEV is a good idea, which version of CEV is best, etc.)
But IMO it makes more sense to just not go down the road of arguing about connotations, given that our language and intuitions aren’t calibrated to this totally-novel situation.
There’s clearly some length of time such that the cost of waiting that long to implement CEV outweighs the benefits. I think those are mostly costs of losing negentropy in the universe at large, though (as stars burn their fuel and/or move away from us via expansion), not costs like ‘the AGI operators get corrupted or make some major irreversible misstep because they waited an extra five years too long’.
I don’t know why you think the corruption/misstep risk of “waiting for an extra three years before running CEV” (for example) is larger than the ‘we might implement CEV wrong’ risk of rushing to implement CEV after zero years of tinkering with working AGI systems.
It seems like the sensible thing to do in this situation is to hope for the best, but plan for realistic outcomes that fall short of “the best”:
Realistically, there’s a strong chance (I would say: overwhelmingly strong) that we won’t be able to fully solve CEV before AGI arrives. So since our options in that case will be “strawberry-grade alignment, or just roll over and die”, let’s start by working on strawberry-grade alignment. Once we solve that problem, sure, we can shift resources into CEV. If you’re optimistic about ‘rush to CEV’, then IMO you should be even more optimistic that we can nail down strawberry alignment fast, at which point we should have made a lot of headway toward CEV alignment without gambling the whole future on our getting alignment perfect immediately and on the first try.
Likewise, realistically, there’s a strong chance (I would say overwhelming) that there will be some multi-year period where humanity can build AGI, but isn’t yet able to maximally align it. It would be good if we don’t just roll over and die in those worlds; so while we might hope for there to be no such period, we should make plans that are robust to such a period occurring.
There’s nothing about the strawberry plan that requires waiting, if it’s not net-beneficial to do so. You can in fact execute a ‘no one else can destroy the world with AGI’ pivotal act, start working on CEV, and then surprise yourself with how fast CEV falls into place and just go implement that in relatively short order.
What strawberry-ish actions do is give humanity the option of waiting. I think we’ll desperately need this option, but even if you disagree, I don’t think you should consider it net-negative to have the option available in the first place.
I agree with this.
This is the place where I say: “I don’t think the best solution to that problem is for the field to throw up their hands and say ‘we’re scientists, it’s not our job to think about practicalities like that’ and hope someone else takes care of it. We’re human beings, not science-bots; we should use our human intelligence to think about which course of action is likeliest to produce good outcomes, and do that.”
We aren’t helpless victims of our social roles, who must look at Research Path A and Research Path B and go, “Hmm, there’s a strategic consideration that says Path A is much more likely to save humanity than Path B… but thinking about practical strategic considerations is the kind of thing people wearing military uniforms do, not the kind of thing people wearing lab coats do.” You don’t have to do the non-world-saving thing just because it feels more normal or expected-of-your-role. You can actually just do the thing that makes sense.
(Again, maybe you disagree with me about what makes sense to do here. But the debate should be had at the level of ‘which factual beliefs suggest that A versus B is likeliest to produce a flourishing long-run future?‘, not at the level of ‘is it improper for scientists to update on information that sounds more geopolitics-y than linear-algebra-y?’. The real world doesn’t care about literary genres and roles; it’s all just facts, and disregarding a big slice of the world’s facts will tend to produce worse decisions.)
Very short. Longer timelines are logically possible, but I wouldn’t count on them.
As for this notion that something like CEV might require decades of thought to be figured out, or even might require decades of trial and error with AGI—that’s just a guess. I may be monotonous by saying June Ku over and over again (there are others whose work I intend to study too), but metaethical.ai is an extremely promising schema. If a serious effort was made to fill out that schema, while also critically but constructively examining its assumptions from all directions, who knows how far we’d get, and how quickly?
Another argument for shorter CEV timelines, is that AI itself may help complete the theory of CEV alignment. Along with the traditional powers of computation—calculation, optimization, deduction, etc—language models, despite their highly uneven output, are giving us a glimpse of what it will be like, to have AI contributing even to discussions like this. That day isn’t far off at all.
So from my perspective, long CEV timelines don’t actually seem likely. The other thing that I have great doubts about, is the stability of any world order in which a handful of humans - even if it were the NSA or the UN Security Council—use tool AGI to prevent everyone else from developing unsafe AGI. Targeting just one thing like GPUs won’t work forever because you can do computation in other ways; there will be great temptations to use tool AGI to carry out interventions that have nothing to do with stopping unsafe AGI… Anyone in such a position becomes a kind of world government.
The problem of “world government” or “what the fate of the world should be”, is something that CEV is meant to solve comprehensively, by providing an accurate first-principles extrapolation of humanity’s true volition, etc. But here, the scenario is an AGI-powered world takeover where the problems of governance and normativity have not been figured out. I’m not at all opposed to thinking about such scenarios; the next chapter of human affairs may indeed be one in which autonomous superhuman AI does not yet exist, but there are human elites possessing tool AGI. I just think that’s a highly unstable situation; making it stable and safe for humans would be hard to do without having something like CEV figured out; and because of the input of AI itself, one shouldn’t expect figuring out CEV to take a long time. I propose that it will be done relatively quickly, or not at all.
I agree with this part. That’s why I’ve been saying ‘maybe we can do this in a few subjective decades or centuries’ rather than ‘maybe we can do this in a few subjective millennia.’ 🙂
But I’m mostly imagining AGI helping us get CEV theory faster. Which obviously requires a lot of prior alignment just to make use of the AGI safely, and to trust its outputs.
The idea is to keep ratcheting up alignment so we can safely make use of more capabilities—and then, in at least some cases, using those new capabilities to further improve and accelerate our next ratcheting-up of alignment.
… And that makes you feel optimistic about the rush-to-CEV option? ‘Unaligned AGIs or proto-AGIs generating plausble-sounding arguments about how to do CEV’ is not a scenario that makes me update toward humanity surviving.
I share your pessimism about any group that would feel inclined to give in to those temptations, when the entire future light cone is at stake.
The scenario where we narrowly avoid paperclips by the skin of our teeth, and now have a chance to breathe and think things through before taking any major action, is indeed a fragile one in some respects, where there are many ways to rapidly destroy all of the future’s value by overextending. (E.g., using more AGI capabilities than you can currently align, or locking in contemporary human values that shouldn’t be locked in, or hastily picking the wrong theory or implementation of ‘how to do moral progress’.)
I don’t think we necessarily disagree about anything except ‘how hard is CEV’? It sounds to me like we’d mostly have the same intuitions conditional on ‘CEV is very hard’; but I take this very much for granted, so I’m freely focusing my attention on ‘OK, how could we make things go well given that fact?’.
I disagree on the plausibility of a stop-the-world scheme. The way things are now is as safe or stable as they will ever be. I think it’s a better plan to use the rising tide of AI capabilities to flesh out CEV. In particular, since the details of CEV depend on not-yet-understood details of human cognition, one should think about how to use near-future AI power to extract those details from the available data regarding brains and behavior. But AI can contribute everywhere else in the development of CEV theory and practice too.
But I won’t try to change your thinking. In practice, MIRI’s work on simpler forms of alignment (and its work on logical induction, decision theory paradigms, etc) is surely relevant to a “CEV now” approach too. What I am wondering is, where is the de-facto center of gravity for a “CEV now” effort? I’ve said many times that June Ku’s blueprint is the best I’ve seen, but I don’t see anyone working to refine it. And there are other people whose work seems relevant and has a promising rigor, but I’m not sure how it best fits together.
edit: I do see a value to non-CEV alignment work, distinct from figuring out how to stop the world safely: and that is to reduce the risk arising from the general usage of advanced AI systems. So it is a contribution to AI safety.
CEV seems much much more difficult than strawberry alignment and I have written it off as a potential option for a baby’s first try at constructing superintelligence.
To be clear, I also expect that strawberry alignment is too hard for these babies and we’ll just die. But things can always be even more difficult, and with targeting CEV on a first try, it sure would be.
There’s zero room, there is negative room, to give away to luxury targets like CEV. They’re not even going to be able to do strawberry alignment, and if by some miracle we were able to do strawberry alignment and so humanity survived, that miracle would not suffice to get CEV right on the first try.
I highly doubt anything in this universe could be ‘intrinsically’ safe and humane. I’m not sure what such an object would even look like or how we could verify that.
Even purposefully designed safety systems for deep pocketed customers for very simple operations in a completely controlled environment, such as car manufacturer’s assembly lines with a robot arm assigned solely to tightening nuts, have safety systems that are not presumed to be 100% safe in any case. That’s why they have multiple layers of interlocks, panic buttons, etc.
And that’s for something millions of times simpler than a superhuman agi in ideal conditions.
Perhaps before even the strawberry, it would be interesting to consider the difficulty of the movable arm itself.
A useful distinction. Yet of the rare outcomes that follow current timelines without ending in ruin, I expect the most likely one falls into neither category. Instead it’s an AGI that behaves like a weird supersmart human that bootstrapped its humanity from language models with relatively little architecture support (for alignment), as a result of a theoretical miracle where things like that are the default outcome. Possibly from giving a language model autonomy/agency to debug its thinking while having notebooks and a working memory, tuning the model in the process. It’s not going to reliably do as it’s told, could be deceptive, yet possibly doesn’t turn everything into paperclips. Arguably it’s aligned, but only the way weird individual humans are aligned, which is noncentrally strawberry-aligned, and too-indirectly-to-use-the-term CEV-aligned.