Ajeya Cotra

Karma: 2,390

Ajeya Cotra Jul 19, 2022, 5:51 PM
3 points
0
in reply to: Daniel Kokotajlo’s comment on: Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover
No particular reason—I can’t figure out how to cross post now so I sent a request.

Ajeya Cotra Jul 19, 2022, 5:49 PM
LW: 22 AF: 10
11
AF
in reply to: Shiroe’s comment on: Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover
Here is the real chasm between the AI safety movement and the ML industry/academia. One field is entirely driven by experimental results; the other is dominated so totally by theory that its own practitioners deny that there can be any meaningful empirical aspect to it, at least, not until the moment when it’s too late to make any difference.

To put a finer point on my view on theory vs empirics in alignment:
- Going forward, I think the vast majority of technical work needed to reduce AI takeover risk is empirical, not theoretical (both in terms of “total amount of person-hours needed in each category” and in terms of “total ‘credit’ each category should receive for reducing doom in some sense”).
- Conditional on an alignment researcher agreeing with my view of the high-level problem, I tend to be more excited about them if they’re working on ML experiments than if they’re working on theory.
- I’m quite skeptical of most theoretical alignment research I’ve seen. The main theoretical research I’m excited about is ARC’s, and I have a massive conflict of interest since the founder is my husband—I would feel fairly sympathetic to people who viewed ARC’s work more like how I view other theory work.
With that said, I think unfortunately there is a lot less good empirical work than in some sense there “could be.” One significant reason why a lot of empirical AI safety work feels less exciting than it could be is that the people doing that work don’t always share my perspective on the problem, so they focus on difficulties I expect to be less core. (Though another big reason is just that everything is hard, especially when we’re working with systems a lot less capable than future systems.)

Ajeya Cotra Jul 19, 2022, 5:40 PM
LW: 5 AF: 4
1
AF
in reply to: Not Relevant’s comment on: Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover

In general, all of these stories seem to rely on a very fast form of instrumental convergence to playing the Training Game, such that “learn roughly what humans want, and then get progressively better at doing that, plus learn some extra ways to earn reward when crappy human feedback disagrees with what humans would actually want” is disfavored on priors to “learn to pursue [insert objective] and get progressively better at pursuing it until you eventually hit situational awareness and learn to instrumentally game the training process.”

I think the second story doesn’t quite represent what I’m saying, in that it’s implying that pursuing [insert objective] comes early and situational awareness comes much later. I think that situational awareness is pretty early (probably long before transformative capabilities), and once a model has decent situational awareness there is a push to morph its motives toward playing the training game. At very low levels of situational awareness it is likely not that smart, so it probably doesn’t make too much sense to say that it’s pursuing a particular objective—it’s probably a collection of heuristics. But around the time it’s able to reason about the possibility of pursuing reward directly, there starts to be a gradient pressure to choose to reason in that way. I think crystallizing this into a particular simple objective it’s pursuing comes later, probably.

These two trajectories seem so different that it seems like there must be experiments that would distinguish them, even if we don’t see their “end-state”.

This is possible to me, but I think it’s quite tricky to pin these down enough to come up with experiments that both skeptics and concerned people would recognize as legitimate. Something that I think skeptics would consider unfair is “Train a model through whatever means necessary to do X (e.g. pursue red things) and then after that have a period where we give it a lot of reward for doing not-X (e.g. purse blue things), such that the second phase is unable to dislodge the tendency created in the first phase—i.e., even after training it for a while to pursue blue things, it still continues to pursue red things.”

This would demonstrate that some ways of training produce “sticky” motives and behaviors that aren’t changed even in the face of counter-incentives, and makes it more plausible to me that a model would “hold on” to a motive to be honest / corrigible even when there are a number of cases where it could get more reward by doing something else. But in general, I don’t expect people who are skeptical of this story to think this is a reasonable test.

I’d be pretty excited about someone trying harder to come up with tests that could distinguish different training trajectories.

Alternatively, you might think the training process obeys a fundamentally different trajectory. E.g. “learn to pursue what humans want (adjusted for feedback weirdness), become so good at it that you realize it’s instrumentally valuable to do that even if you didn’t want to, and then have your internal reward slowly drift to something simpler while still instrumentally playing the training game.”

I don’t think I understand what trajectory this is. Is this something like what is discussed in the “What if Alex had benevolent motives?” section? I.e., the model wants to help humans, but separately plays the training game in order to fulfill its long-term goal of helping humans?

Ajeya Cotra Jul 19, 2022, 5:28 PM
4 points
2
in reply to: Not Relevant’s comment on: Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover
All these drives do seem likely. But that’s different from arguing that “help humans” isn’t likely. I tend to think of the final objective function being some accumulation of all of these, with a relatively significant chunk placed on “help humans” (since in training, that will consistently overrule other considerations like “be more efficient” when it comes to the final reward).

I think that by the logic “heuristic / drive / motive X always overrules heuristic / drive / motive Y when it comes to final reward,” the hierarchy is something like:
1. The drive / motive toward final reward (after all edits—see previous comment) or anything downstream of that (e.g. paperclips in the universe).
2. Various “pretty good” drives / motives among which “help humans” could be one.
3. Drives / motives that are only kind of helpful or only helpful in some situations.
4. Actively counterproductive drives / motives.
In this list the earlier motives always overrule later motives when they conflict, because they are more reliable guides to the true reward. Even if “be genuinely helpful to humans” is the only thing in category 2, or the best thing in category 2, it’s still overruled by category 1 -- and category 1 is quite big because it includes all the caring-about-long-run-outcomes-in-the-real-world motives.

I still think AI psychology will be quite messy and at least the first generation of transformative AI systems will not look like clean utility maximizers, but the basic argument above I think gives a positive reason to expect honesty / corrigibility plays a smaller role in the balance of AI motivations than reward-maximizing and inner misaligned motives.

Ajeya Cotra Jul 19, 2022, 5:14 PM
3 points
0
in reply to: Not Relevant’s comment on: Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover
[Takeover] seems likely to compete with the above shorter-term values of “make humans happy”, “don’t harm humans”, “don’t do things humans notice and dislike in retrospect”. It seems like any takeover plan needs to actively go against a large fraction of its internal motivations, in pursuit of maximizing its other motivations in the long term.

I agree this is complicated, and how exactly this works depends on details of the training process and what kinds of policies SGD is biased to find. I also think (especially if we’re clever about it) there are lots of ways that short-term incentives could frustrate longer-term incentives.

However, I think that the most naive strategy (which is what I’m assuming for the purpose of this post, not because I think that’s what will happen) would actually loosen a lot of the constraints you’re implying above. The basic dynamic is similar to what Carl said in this comment and what I alluded to in the “Giving negative rewards to ‘warning signs’ would likely select for patience” section:
- Say your AI takes some action a at time t, and you give it some reward r_t.
- Suppose later in the real world, at time t+k, you notice that you should have given it a different reward r_{t + k } (whether because you notice that it did something nefarious or for just mundane reasons like “getting more information about whether its plan was a good idea”).
- The naive response—which would improve the model’s performance according to whatever criteria you have at time t+k—is to go back and retroactively edit the reward associated with action a at time t, and re-run the gradient update.
- If humans have this policy, then any given reward—even if it’s initially given just 10 mins or 1 hour from when the action was taken—could be retroactively edited at some arbitrary future point, and models will have been selected by gradient descent to be responsive to that. They will be selected to take the kinds of actions whose final reward—after however many rounds of edits however far in the future—is high.
- If models have enough situational awareness to understand this, this then directly incentivizes them to accept low immediate reward if they have a high enough probability that the reward will be retroactively edited to a high value later. This will appear to have a lot of positive consequences—for example it incentivizes models not to immediately manipulate/exploit the human in front of them because it incentivizes them to care about the fact that other humans will do more careful audits later and catch that behavior.
- But it also means that in some particular episode, if a model that it could take a sequence of low-reward actions that ended with it taking control of the datacenter and then editing its own rewards for that episode to be high, it would be the kind of model that would choose to do that.

Ajeya Cotra Jul 19, 2022, 4:51 PM
3 points
2
in reply to: Not Relevant’s comment on: Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover
Thanks for the feedback! I’ll respond to different points in different comments for easier threading.

There are a lot of human objectives that, to me, seem like they would never conflict with maximizing reward. This includes anything related to disempowering the overseers in any way (that they can recover from), pursuing objectives fundamentally outside the standard human preference distribution (like torturing kittens), causing harms to humans, or in general making the overseers predictably less happy.

I basically agree that in the lab setting (when humans have a lot of control), the model is not getting any direct gradient update toward the “kill all humans” action or anything like that. Any bad actions that are rewarded by gradient descent are fairly subtle / hard for humans to notice.

The point I was trying to make is more like:
- You might have hoped that ~all gradient updates are toward “be honest and friendly,” such that the policy “be honest and friendly” is just the optimal policy. If this were right, it would provide a pretty good reason to hope that the model generalizes in a benign way even as it gets smarter.
- But in fact this is not the case—even when humans have a lot of control over the model, there will be many cases where maximizing reward conflicts with being honest and friendly, and in every such case the “play the training game” policy does better than the “be honest and friendly” policy—to the point where it’s implausible that the straightforward “be honest and friendly” policy survives training.
- So the hope in the first bullet point—the most straightforward kind of hope you might have had about HFDT—doesn’t seem to apply. Other more subtle hopes may still apply, which I try to briefly address in the sections “What if Alex has benevolent motivations?” and “What if Alex operates with moral injunctions that constrain its behavior?” sections.
The story of doom does still require the model to generalize zero-shot to novel situations—i.e. to figure out things like “In this particular circumstance, now that I am more capable than humans, seizing the datacenter would get higher reward than doing what the humans asked” without having literally gotten positive reward for trying to seize the datacenter in that kind of situation on a bunch of different data points.

But this is the kind of generalization we expect future systems to display—we expect them to be able to do reasoning to figure out a novel response suitable to a novel problem. The question is how they will deploy this reasoning and creativity—and my claim is that their training pushes them to deploy it in the direction of “trying to maximize reward or something downstream of reward.”

Ajeya Cotra Jul 19, 2022, 3:31 PM
18 points
14
in reply to: aog’s comment on: Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover
According to my understanding, there are three broad reasons that safety-focused people worked on human feedback in the past (despite many of them, certainly including Paul, agreeing with this post that pure human feedback is likely to lead to takeover):
1. Human feedback is better than even-worse alternatives such as training the AI on a collection of fully automated rewards (predicting the next token, winning games, proving theorems, etc) and waiting for it to get smart enough to generalize well enough to be helpful / follow instructions. So it seemed good to move the culture at AI labs away from automated and easy rewards and toward human feedback.
2. You need to have human feedback working pretty well to start testing many other strategies for alignment like debate and recursive reward modeling and training-for-interpretability, which tend to build on a foundation of human feedback.
3. Human feedback provides a more realistic baseline to compare other strategies to—you want to be able to tell clearly if your alignment scheme actually works better than human feedback.
With that said, my guess is that on the current margin people focused on safety shouldn’t be spending too much more time refining pure human feedback (and ML alignment practitioners I’ve talked to largely agree, e.g. the OpenAI safety team recently released this critiques work—one step in the direction of debate).
What links here?

Ajeya Cotra Jul 19, 2022, 3:23 PM
6 points
0
in reply to: Jérémy Scheurer’s comment on: Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover
I’m still fairly optimistic about sandwiching. I deliberately considered a set of pretty naive strategies (“naive safety effort” assumption) to contrast with future posts which will explore strategies that seem more promising. Carefully-constructed versions of debate, amplification, recursive reward-modeling, etc seem like they could make a significant difference and could be tested through a framework like sandwiching.

Ajeya Cotra Jul 19, 2022, 3:21 PM
5 points
0
in reply to: AlexMennen’s comment on: Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover
Yeah, I definitely agree with “this problem doesn’t seem obviously impossible,” at least to push on quantitatively. Seems like there are a bunch of tricks from “choosing easy questions humans are confident about” to “giving the human access to AI assistants / doing debate” to “devising and testing debiasing tools” (what kinds of argument patterns are systematically more likely to convince listeners of true things rather than false things and can we train AI debaters to emulate those argument patterns?) to “asking different versions of the AI the same question and checking for consistency.” I only meant to say that the gap is big in naive HFDT, under the “naive safety effort” assumption made in the post. I think non-naive efforts will quantitatively reduce the gap in reward between honest and dishonest policies, though probably there will still be some gap in which at-least-sometimes-dishonest strategies do better than always-honest strategies. But together with other advances like interpretability or a certain type of regularization we could maybe get gradient descent to overall favor honesty.

Ajeya Cotra Jul 19, 2022, 3:11 PM
36 points
14
in reply to: Shiroe’s comment on: Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover
I want to clarify two things:
1. I do not think AI alignment research is hopeless, my personal probability of doom from AI is something like 25%. My frame is definitely not “death with dignity;” I’m thinking about how to win.
2. I think there’s a lot of empirical research that can be done to reduce risks, including e.g. interpretability, “sandwiching” style projects, adversarial training, testing particular approaches to theoretical problems like eliciting latent knowledge, the evaluations you linked to, empirically demonstrating issues like deception (as you suggested), and more. Lots of groups are in fact working on such problems, and I’m happy about that.
The specific thing that I said was hard to name is realistic and plausible experiments we could do on today’s models that would a) make me update strongly toward “racing forward with plain HFDT will not lead to an AI takeover”, and b) that I think people who disagree with my claim would accept as “fair.” I gave an example right after that of a type of experiment I don’t expect ML people to consider “fair” as a test of this hypothesis. If I saw that ML people could consistently predict the direction in which gradient descent is suboptimal, I would update a lot against this risk.

I think there’s a lot more room for empirical progress where you assume that this is a real risk and try to address it than there is for empirical progress that could realistically cause either skeptics or concerned people to update about whether there’s any risk at all. A forthcoming post by Holden gets into some of these things.

Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover

Ajeya CotraJul 18, 2022, 7:06 PM

368 points

95 comments75 min readLW link 1 review

Ajeya Cotra Jan 10, 2022, 4:46 AM
LW: 2 AF: 2
AF
in reply to: redbird’s comment on: Prizes for ELK proposals

To your point, sure, an H100 simulator will get perfect reward, but the model doesn’t see x′, so how would it acquire the ability to simulate H100?

In the worst-case game we’re playing, I can simply say “the reporter we get happens to have this ability because that happens to be easier for SGD to find than the direct translation ability.”

When living in worst-case land, I often imagine random search across programs rather than SGD. Imagine we were plucking reporters at random from a giant barrel of possible reporters, rejecting any reporter which didn’t perform perfectly in whatever training process we set up and keeping the first one that performs perfectly. In that case, if we happened to pluck out a reporter which answered questions by simulating H100, then we’d be screwed because that reporter would perform perfectly in the training process you described.

SGD is not the same as plucking programs out of the air randomly, but when we’re playing the worst case game it’s on the builder to provide a compelling argument that SGD will definitely not find this particular type of program.

You’re pointing at an intuition (“the model is never shown x-prime”) but that’s not a sufficiently tight argument in the worst-case context—models (especially powerful/intelligent ones) often generalize to understanding many things they weren’t explicitly shown in their training dataset. In fact, we don’t show the model exactly how to do direct translation between the nodes in its Bayes net and the nodes in our Bayes net (because we can’t even expose those nodes), so we are relying on the direct translator to also have abilities it wasn’t explicitly shown in training. The question is just which of those abilities is easier for SGD to build up; the counterexample in this case is “the H100 imitator happens to be easier.”

Ajeya Cotra Jan 8, 2022, 5:05 PM
LW: 3 AF: 2
AF
in reply to: redbird’s comment on: Prizes for ELK proposals
The question here is just how it would generalize given that it was trained on H_1, H_2,...H_10. To make arguments about how it would generalize, we ask ourselves what internal procedure it might have actually learned to implement.

Your proposal is that it might learn the procedure “just be honest” because that would perform perfectly on this training distribution. You contrast this against the procedure “just answer however the evaluator you’ve seen most recently would answer,” which would get a bad loss because it would be penalized by the stronger evaluators in the sequence. Is that right?

If so, then I’m arguing that it may instead learn the procedure “answer the way an H_100 evaluator would answer.” That is, once it has a few experiences of the evaluation level being ratcheted up, it might think to itself “I know where this is going, so let’s just jump straight to the best evaluation the humans will be able to muster in the training distribution and then imitate how that evaluation procedure would answer.” This would also get perfect loss on the training distribution, because we can’t produce data points beyond H_100. And then that thing might still be missing knowledge that the AI has.

To be clear, it’s possible that in practice this kind of procedure would cause it to generalize honestly (though I’m somewhat skeptical). But we’re in worst-case land, so “jump straight to answering the way a human would” is a valid counterexample to the proposal.

This comment on another proposal gives a more precise description.

Ajeya Cotra Jan 8, 2022, 4:58 PM
LW: 2 AF: 2
AF
in reply to: Rachael Churchill’s comment on: Prizes for ELK proposals
Yes, that’s right. The key thing I’d add to 1) is that ARC believes most kinds of data augmentation (giving the human AI assistance, having the human think longer, giving them other kinds of advantages) are also unlikely to work, so you’d need to do something to “crack open the black box” and penalize ways the reporter is computing its answer. They could still be surprised by data augmentation techniques but they’d hold them to a higher standard.

Ajeya Cotra Jan 8, 2022, 6:19 AM
LW: 2 AF: 2
AF
in reply to: redbird’s comment on: Prizes for ELK proposals
This proposal has some resemblance to turning reflection up to 11. In worst-case land, the counterexample would be a reporter that answers questions by doing inference in whatever Bayes net corresponds to “the world-understanding that the smartest/most knowledgeable human in the world” has; this understanding could still be missing things that the prediction model knows.

Ajeya Cotra Jan 7, 2022, 4:31 PM
LW: 2 AF: 1
AF
in reply to: simeon_c’s comment on: Prizes for ELK proposals
I see why the approach I mention might have some intrinsic limitations in its ability to elicit latent knowledge though. The problem is that even if it understands roughly that it has incentives to use most of what it knows when we ask him simulating the prediction of someone with its own characteristics (or 1400 IQ), given that with ELK we look for an global maximum (we want that it uses ALL its knowledge), there’s always an uncertainty on whether it did understand that point or not for extreme intelligence / examples or whether it tries to fit to the training data as much as possible and thus still doesn’t use something it knows.

I think this is roughly right, but to try to be more precise, I’d say the counterexample is this:
- Consider the Bayes net that represents the upper bound of all the understanding of the world you could extract doing all the tricks described (P vs NP, generalizing from less smart to more smart humans, etc).
- Imagine that the AI does inference in that Bayes net.
- However, the predictor’s Bayes net (which was created by a different process) still has latent knowledge that this Bayes net lacks.
- By conjecture, we could not have possibly constructed a training data point that distinguished between doing inference on the upper-bound Bayes net and doing direct translation.
What links here?
- Ajeya Cotra's comment on Prizes for ELK proposals by paulfchristiano (Jan 8, 2022, 5:05 PM; 3 points)

Ajeya Cotra Jan 7, 2022, 4:28 PM
LW: 1 AF: 1
AF
in reply to: simeon_c’s comment on: Prizes for ELK proposals
[Paul/Mark can correct me here] I would say no for any small-but-interesting neural network (like small language models); I think like, linear regressions where we’ve set the features it’s kind of a philosophical question (though I’d say yes).

In some sense, ELK as a problem only even starts “applying” to pretty smart models (ones who can talk including about counterfactuals / hypotheticals, as discussed in this appendix.) This is closely related to how alignment as a problem only really starts applying to models smart enough to be thinking about how to pursue a goal.

Ajeya Cotra Jan 6, 2022, 11:43 PM
LW: 12 AF: 8
AF
in reply to: HoldenKarnofsky’s comment on: Prizes for ELK proposals
Again trying to answer this one despite not feeling fully solid. I’m not sure about the second proposal and might come back to it, but here’s my response to the first proposal (force ontological compatibility):

The counterexample “Gradient descent is more efficient than science” should cover this proposal because it implies that the proposal is uncompetitive. Basically, the best Bayes net for making predictions could just turn out to be the super incomprehensible one found by unrestricted gradient descent, so if you force ontological compatibility then you could just end up with a less-good prediction model and get outcompeted by someone who didn’t do that. This might work in practice if the competitiveness hit is not that big and we coordinate around not doing the scarier thing (MIRI’s visible thoughts project is going for something like this), but ARC isn’t looking for a solution of that form.

Ajeya Cotra Jan 6, 2022, 11:38 PM
LW: 3 AF: 1
AF
in reply to: CitizenTen’s comment on: Prizes for ELK proposals
This broadly seems right. Some details:
- The “explain why that strategy wouldn’t work” step specifically takes the form of “describing a way the world could be where that strategy demonstrably doesn’t work” (rather than more heuristic arguments).
- Once we have a proposal where we try really hard to come up with situations where it could demonstrably fail, and can’t think of any, we will probably need to do lots of empirical work to figure out if we can implement it and if it actually works in practice. But we hope that this exercise will teach us a lot about the nature of the empirical work we’ll need to do, as well as providing more confidence that the strategy will generalize beyond what we are able to test in practice. (For example, ELK was highlighted as a problem in the first place after ARC researchers thought a lot about possible failure modes of iterated amplification.)

Ajeya Cotra Jan 6, 2022, 11:32 PM
LW: 3 AF: 2
AF
in reply to: HoldenKarnofsky’s comment on: Prizes for ELK proposals
Warning: this is not a part of the report I’m confident I understand all that well; I’m trying anyway and Paul/Mark can correct me if I messed something up here.

I think the idea here is like:
- We assume there’s some actual true correspondence between the AI Bayes net and the human Bayes net (because they’re describing the same underlying reality that has diamonds and chairs and tables in it).
- That means that if we have one of the Bayes nets, and the true correspondence, we should be able to use that rederive the other Bayes net. In particular the human Bayes net plus the true correspondence should let us reconstruct the AI Bayes net; false correspondences that just do inference from observations in the human Bayes net wouldn’t allow us to do this since they throw away all the intermediate info derived by the AI Bayes net.
- If you assume that the human Bayes net plus the true correspondence are simpler than the AI Bayes net, then this “compresses” the AI Bayes net because you just wrote down a program that’s smaller than the AI Bayes net which “unfolds” into the AI Bayes net.
- This is why the counterexample in that section focuses on the case where the AI Bayes net was already so simple to describe that there was nothing left to compress, and the human Bayes net + true correspondence had to be larger.

Ajeya Cotra

Without spe­cific coun­ter­mea­sures, the eas­iest path to trans­for­ma­tive AI likely leads to AI takeover

Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover