(Coherence aside, when I now look at that number it does seem a bit too high, and I feel tempted to move it to 2027-2028, but I dunno, that kind of intuition is likely to change quickly from day to day.)
Ajeya Cotra
Karma: 2,390
Hm, yeah, I bet if I reflected more things would shift around, but I’m not sure the fact that there’s a shortish period where the per-year probability is very elevated followed by a longer period with lower per-year probability is actually a bad sign.
Roughly speaking, right now we’re in an AI boom where spending on compute for training big models is going up rapidly, and it’s fairly easy to actually increase spending quickly because the current levels are low. There’s some chance of transformative AI in the middle of this spending boom—and because resource inputs are going up a ton each year, the probability of TAI by date X would also be increasing pretty rapidly.
But the current spending boom is pretty unsustainable if it doesn’t lead to TAI. At some point in the 2040s or 50s, if we haven’t gotten transformative AI by then, we’ll have been spending 10s of billions training models, and it won’t be that easy to keep ramping up quickly from there. And then because the input growth will have slowed, the increase in probability from one year to the next will also slow. (That said, not sure how this works out exactly.)
Two-year update on my personal AI timelines
Where does the selection come from? Will the designers toss a really impressive AI for not getting reward on that one timestep? I think not.
I was talking about gradient descent here, not designers.
It doesn’t seem like it would have to prevent us from building computers if it has access to far more compute than we could access on Earth. It would just be powerful enough to easily defeat the kind of AIs we could train with the relatively meager computing resources we could extract from Earth. In general the AI is a superpower and humans are dramatically technologically behind, so it seems like it has many degrees of freedom and doesn’t have to be particularly watching for this.
Neutralizing computational capabilities doesn’t seem to involve total destruction of physical matter or human extinction though, especially for a very powerful being. Seems like it’d be basically just as easy to ensure we + future AIs we might train are no threat as it is to vaporize the Earth.
My answer is a little more prosaic than Raemon. I don’t feel at all confident that an AI that already had God-like abilities would choose to literally kill all humans to use their bodies’ atoms for its own ends; it seems totally plausible to me that—whether because of exotic things like “multiverse-wide super-rationality” or “acausal trade” or just “being nice”—the AI will leave Earth alone, since (as you say) it would be very cheap for it to do so.
The thing I’m referring to as “takeover” is the measures that an AI would take to make sure that humans can’t take back control—while it’s not fully secure and doesn’t have God-like abilities. Once a group of AIs have decided to try to get out of human control, they’re functionally at war with humanity. Humans could do things like physically destroy the datacenters they’re running on, and they would probably want to make sure they can’t do that.
Securing AI control and defending from human counter-moves seems likely to involve some violence—but it could be a scale of violence that’s “merely” in line with historical instances where a technologically more advanced group of humans colonized or took control of a less-advanced group of humans; most historical takeovers don’t involve literally killing every single member of the other group.
The key point is that it seems likely that AIs will secure the power to get to decide what happens with the future; I’m pretty unsure exactly how they use it, and especially if it involves physically destroying Earth / killing all humans for resources. These resources seem pretty meager compared to the rest of the universe.
I mean things like tricks to improve the sample efficiency of human feedback, doing more projects that are un-enhanced RLHF to learn things about how un-enhanced RLHF works, etc.
I’m pretty confused about how to think about the value of various ML alignment papers. But I think even if some piece of empirical ML work on alignment is really valuable for reducing x-risk, I wouldn’t expect its value to take the form of providing insight to readers like you or me. So you as a reader not getting much out of it is compatible with the work being super valuable, and we probably need to assess it on different terms.
The main channel of value that I see for doing work like “learning to summarize” and the critiques project and various interpretability projects is something like “identifying a tech tree that it seems helpful to get as far as possible along by the Singularity, and beginning to climb that tech tree.”
In the case of critiques—ultimately, it seems like having AIs red team each other and pointing out ways that another AI’s output could be dangerous seems like it will make a quantitative difference. If we had a really well-oiled debate setup, then we would catch issues we wouldn’t have caught with vanilla human feedback, meaning our models could get smarter before they pose an existential threat—and these smarter models can more effectively work on problems like alignment for us.[1]
It seems good to have that functionality developed as far as it can be developed in as many frontier labs as possible. The first steps of that look kind of boring, and don’t substantially change our view of the problem. But first steps are the foundation for later steps, and the baseline against which you compare later steps. (Also every step can seem boring in the sense of bringing no game-changing insights, while nonetheless helping a lot.)
When the main point of some piece of work is to get good at something that seems valuable to be really good at later, and to build tacit knowledge and various kinds of infrastructure for doing that thing, a paper about it is not going to feel that enlightening to someone who wants high-level insights that change their picture of the overall problem. (Kind of like someone writing a blog post about how they developed effective management and performance evaluation processes at their company isn’t going to provide much insight into the abstract theory of principal-agent problems. The value of that activity was in the company running better, not people learning things from the blog post about it.)
I’m still not sure how valuable I think this work is, because I don’t know how well it’s doing at efficiently climbing tech trees or at picking the right tech trees, but I think that’s how I’d think about evaluating it.
[1] Or do a “pivotal act,” though I think I probably don’t agree with some of the connotations of that term.
I was mainly talking about the current margin when I talked about how excited I am about the theoretical vs empirical work I see “going on” right now and how excited I tend to be about currently-active researchers who are doing theory vs empirical research. And I was talking about the future when I said that I expect empirical work to end up with the lion’s share of credit for AI risk reduction.
Eliezer, Bostrom, and co certainly made a big impact in raising the problem to people’s awareness and articulating some of its contours. It’s kind of a matter of semantics whether you want to call that “theoretical research” or “problem advocacy” / “cause prioritization” / “community building” / whatever, and no matter which bucket you put it in I agree it’ll probably end up with an outsized impact for x-risk-reduction, by bringing the problem to attention sooner than it would have otherwise been brought to attention and therefore probably allowing more work to happen on it before TAI is developed.
But just like how founding CEOs tend to end up with ~10% equity once their companies have grown large, I don’t think this historical problem-advocacy-slash-theoretical-research work alone will end up with a very large amount of total credit.
On the main thrust of my point, I’m significantly less excited about MIRI-sphere work that is much less like “articulating a problem and advocating for its importance” and much more like “attempting to solve a problem.” E.g. stuff like logical inductors, embedded agency, etc seem a lot less valuable to me than stuff like the orthogonality thesis and so on.
I think the retroactive editing of rewards (not just to punish explicitly bad action but to slightly improve evaluation of everything) is actually pretty default, though I understand if people disagree. It seems like an extremely natural thing to do that would make your AI more capable and make it more likely to pass most behavioral safety interventions.
In other words, even if the average episode length is short (e.g. 1 hour), I think the default outcome is to have the rewards for that episode be computed as far after the fact as possible, because that helps Alex improve at long-range planning (a skill Magma would try hard to get it to have). This can be done in a way that doesn’t compromise speed of training—you simply reward Alex immediately with your best guess reward, then keep editing it later as more information comes in. At all points in time you have a “good enough” reward ready to go, while also capturing the benefits of pushing your model to think in as long-term a way as possible.
Thanks, but I’m not working on that project! That project is led by Beth Barnes.
Hm, not sure I understand but I wasn’t trying to make super specific mechanistic claims here—I agree that what I said doesn’t reduce confusion about the specific internal mechanisms of how lying gets to be hard for most humans, but I wasn’t intending to claim that it was. I also should have said something like “evolutionary, cultural, and individual history” instead (I was using “evolution” as a shorthand to indicate it seems common among various cultures but of course that doesn’t mean don’t-lie genes are directly bred into us! Most human universals aren’t; we probably don’t have honor-the-dead and different-words-for-male-and-female genes).
I was just making the pretty basic point “AIs in general, and Alex in particular, are produced through a very different process from humans, so it seems like ‘humans find lying hard’ is pretty weak evidence that ‘AI will by default find lying hard.’”
I agree that asking “What specific neurological phenomena make it so most people find it hard to lie?” could serve as inspiration to do AI honesty research, and wasn’t intending to claim otherwise in that paragraph (though separately, I am somewhat pessimistic about this research direction).
I’m agnostic about whether the AI values reward terminally or values some other complicated mix of things. The claim I’m making is behavioral—a claim that the strategy of “try to figure out how to get the most reward” would be selected over other strategies like “always do the nice thing.”
The strategy could be compatible with a bunch of different psychological profiles. “Playing the training game” is a filter over models—lots of possible models could do it, the claim is just that we need to reason about the distribution of psychologies given that the psychologies that make it to the end of training most likely employ the strategy of playing the training game on the training distribution.
Why do I think this? Consider an AI that has high situational awareness, reasoning ability and creative planning ability (assumptions of my situation which don’t yet say anything about values). This AI has the ability to think about what kinds of actions would get the most reward (just like it has the ability to write a sonnet or solve a math problem or write some piece of software; it understands the task and has the requisite subskills). And once it has the ability, it is likely to be pushed in the direction of exercising that ability (since doing so would increase its reward).
This changes its psychology in whatever way most easily results in it doing more of the think-about-what-would-get-the-most-reward-and-do-it behavior. Terminally valuing reward and only reward would certainly do the trick, but a lot of other things would too (e.g. valuing paperclips in the very long run).
Geoffrey Irving, Jan Leike, Paul Christiano, Rohin Shah, and probably others were doing various kinds of empirical work a few years before Redwood (though I would guess Oliver doesn’t like that work and so wouldn’t consider it a counterexample to his view).
I agree that in an absolute sense there is very little empirical work that I’m excited about going on, but I think there’s even less theoretical work going on that I’m excited about, and when people who share my views on the nature of the problem work on empirical work I feel that it works better than when they do theoretical work.
The gradient pressure towards valuing reward terminally when you’ve already figured out reliable strategies for doing what humans want, seems very weak....in practice, it seems to me like these differences would basically only happen due to operator error, or cosmic rays, or other genuinely very rare events (as you describe in the “Security Holes” section).
Yeah, I disagree. With plain HFDT, it seems like there’s continuous pressure to improve things on the margin by being manipulative—telling human evaluators what they want to hear, playing to pervasive political and emotional and cognitive biases, minimizing and covering up evidence of slight suboptimalities to make performance on the task look better, etc. I think that in basically every complex training episode a model could do a little better by explicitly thinking about the reward and being a little-less-than-fully-forthright.
Note I was at −16 with one vote, and only 3 people have voted so far. So it’s a lot due to the karma-weight of the first disagreer.
I think updating negatively on the situation/action pair has functionally the same effect as changing the reward to be what you now think it should be—my understanding is that RL can itself be implemented as just updates on situation/action pairs, so you could have trained your whole model that way. Since the reason you updated negatively on that situation/action pair is because of something you noticed long after the action was complete, it is still pushing your models to care about the longer-run.
This posits that the model has learned to wirehead
I don’t think it posits that the model has learned to wirehead—directly being motivated to maximize reward or being motivated by anything causally downstream of reward (like “more copies of myself” or “[insert long-term future goal that requires me being around to steer the world toward that goal]”) would work.
The claim I’m making is that somehow you made a gradient update toward a model that is more likely to behave well according to your judgment after the edit—and two salient ways that update could be working on the inside is “the model learns to care a bit more about long-run reward after editing” and “the model learns to care a bit more about something downstream of long-run reward.”
A lot of updates like this seem to push the model toward caring a lot about one of those two things (or some combo) and away from caring about the immediate rewards you were citing earlier as a reason it may not want to take over.
I don’t see it that way, no. Today’s coding models can help automate some parts of the ML researcher workflow a little bit, and I think tomorrow’s coding models will automate more and more complex parts, and so on. I think this expansion could be pretty rapid, but I don’t think it’ll look like “not much going on until something snaps into place.”