I’m interested in doing in-depth dialogues to find cruxes. Message me if you are interested in doing this.
I do alignment research, mostly stuff that is vaguely agent foundations. Currently doing independent alignment research on ontology identification. Formerly on Vivek’s team at MIRI. Most of my writing before mid 2023 is not representative of my current views about alignment difficulty.
I think the scheme you’re describing caps the agent at moderate problem-solving capabilities. Not being able to notice past mistakes is a heck of a disability.
It’s not entirely clear to me that the math works out for AIs being helpful on net relative to humans just doing it, because of the supervision required, and the trust and misalignment issues.
But on this question (for AIs that are just capable of “prosaic and relatively unenlightened ML research”) it feels like shot-in-the-dark guesses. It’s very unclear to me what is and isn’t possible.
Thanks, I appreciate the draft. I see why it’s not plausible to get started on now, since much of it depends on having AGIs or proto-AGIs to play with.
I guess I shouldn’t respond too much in public until you’ve published the doc, but:
If I’m interpreting correctly, a number of the things you intend to try involve having a misaligned (but controlled) proto-AGI run experiments involving training (or otherwise messing with in some way) an AGI. I hope you have some empathy the internal screaming I have toward this category of things.
A bunch of the ideas do seem reasonable to want to try (given that you had AGIs to play with, and were very confident that doing so wouldn’t allow them to escape or otherwise gain influence). I am sympathetic to the various ideas that involve gaining understanding of how to influence goals better by training in various ways.
There are chunks of these ideas that definitely aren’t “prosaic and relatively unenlightened ML research”, and involve very-high-trust security stuff or non-trivial epistemic work.
I’d be a little more sympathetic to these kinda desperate last-minute things if I had no hope in literally just understanding how to build task-AGI properly, in a well understood way. We can do this now. I’m baffled that almost all of the EA-alignment-sphere has given up on even trying to do this. From talking to people this weekend this shift seems downstream of thinking that we can make AGIs do alignment work, without thinking this through in detail.
The total quantity of risk reduction is unclear, but seems substantial to me. I’d guess takeover risk goes from 50% to 5% if you do a very good job at executing on huge amounts of prosaic and relatively unenlightened ML research at the relevant time
Agree it’s unclear. I think the chance of most of the ideas being helpful depends on some variables that we don’t clearly know yet. I think 90% risk improvement can’t be right, because there’s a lot of correlation between each of the things working or failing. And a lot of the risk comes from imperfect execution of the control scheme, which adds on top.
One underlying intuition that I want to express: The world where we are making proto-AGIs run all these experiments is pure chaos. Politically and epistemically and with all the work we need to do. I think pushing toward this chaotic world is much worse than other worlds we could push for right now.
But if I thought control was likely to work very well and saw a much more plausible path to alignment among the “stuff to try”, I’d think it was a reasonable strategy.
I also think that getting the ML community to work on things effectively is probably substantially harder than getting AIs to work on things effectively
On some axes, but won’t there to be axes where AIs are more difficult than humans also? Sycophancy&slop being the most salient. Misalignment issues being another.
This work isn’t extremely easy to verify or scale up (such that I don’t think “throw a billion dollars at it” just works),
This makes sense now. But I think this line should make you worry about whether you can make controlled AIs do it.
I think if the model is scheming it can behave arbitrarily badly in concentrated ways (either in a small number of actions or in a short period of time), but you can make it behave well in the average case using online training.
I think we kind of agree here. The cruxes remain: I think that the metric for “behave well” won’t be good enough for “real” large research acceleration. And “average case” means very little when it allows room for deliberate-or-not mistakes sometimes when they can be plausibly got-away-with. [Edit: Or sabotage, escape, etc.]
Also, you need hardcore knowledge restrictions in order for the AI not to be able to tell the difference between I’m-doing-original-research vs humans-know-how-to-evaluate-this-work. Such restrictions are plausibly crippling for many kinds of research assistance.
FWIW, I don’t think “data-efficient long-horizon RL” (which is sample efficient in a online training sense) implies you can make faithful simulations.
I think there exists an extremely strong/unrealistic version of believing in “data-efficient long-horizon RL” that does allow this. I’m aware you don’t believe this version of the statement, I was just using it to illustrate one end of a spectrum. Do you think the spectrum I was illustrating doesn’t make sense?
Yep this is the third crux I think. Perhaps the most important.
To me it looks like you’re making a wild guess that “prosaic and relatively unenlightened ML research” is a very large fraction of the necessary work for solving alignment, without any justification that I know of?
For all the pathways to solving alignment that I am aware of, this is clearly false. I think if you know of a pathway that just involves mostly “prosaic and relatively unenlightened ML research”, you should write out this plan, why you expect it to work, and then ask OpenPhil throw a billion dollars toward every available ML-research-capable human to do this work right now. Surely it’d be better to get started already?
I’m not entirely sure where our upstream cruxes are. We definitely disagree about your conclusions. My best guess is the “core mistake” comment below, and the “faithful simulators” comment is another possibility.
Maybe another relevant thing that looks wrong to me: You will still get slop when you train an AI to look like it is epistemically virtuously updating its beliefs. You’ll get outputs that look very epistemically virtuous, but it takes time and expertise to rank them in a way that reflects actual epistemic virtue level, just like other kinds of slop.
I don’t see why you would have more trust in agents created this way.
(My parent comment was more of a semi-serious joke/tease than an argument, my other comments made actual arguments after I’d read more. Idk why this one was upvoted more, that’s silly).
these are also alignment failures we see in humans.
Many of them have close analogies in human behaviour. But you seem to be implying “and therefore those are non-issues”???
There are many groups of humans (or groups of humans), that if you set them on the task of solving alignment, will at some point decide to do something else. In fact, most groups of humans will probably fail like this.
How is this evidence in favour of your plan ultimately resulting in a solution to alignment???
but these systems empirically often move in reasonable and socially-beneficial directions over time
Is this the actual basis of your belief in your plan to ultimately get a difficult scientific problem solved?
and i expect we can make AI agents a lot more aligned than humans typically are
Ahh I see. Yeah this is crazy, why would you expect this? I think maybe you’re confusing yourself by using the word “aligned” here, can we taboo it? Human reflective instability looks like: they realize they don’t care about being a lawyer and go become a monk. Or they realize they don’t want to be a monk and go become a hippy (this one’s my dad). Or they have a mid-life crisis and do a bunch of stereotypical mid-life crisis things. Or they go crazy in more extreme ways.
We have a lot of experience with the space of human reflective instabilities. We’re pretty familiar with the ways that humans interact with tribes and are influenced by them, and sometimes break with them.
But the space of reflective-goal-weirdness is much larger and stranger than we have (human) experience with. There are a lot of degrees of freedom in goal specification that we can’t nail down easily through training. Also, AIs will be much newer, much more in progress, than humans are (not quite sure how to express this, another way to say it is to point to the quantity of robustness&normality training that evolution has subjected humans to).
Therefore I think it’s extremely, wildly wrong to expect “we can make AI agents a lot more [reflectively goal stable with predictable goals and safe failure-modes] than humans typically are”.
but, Claude sure as hell seems to
Why do you even consider this relevant evidence?
[Edit 25/02/25:
To expand on this last point, you’re saying:
If we have agents that sure as hell seem to care about the law and are not just pretending (they really will, in most cases, act like they care about the law) then that seems to be a good state to be in.
It seems like you’re doing the same dichotomy here, where you say it’s either pretending or it’s aligned. I know that they will act like they care about the law. We both see the same evidence, I’m not just ignoring it. I just think you’re interpreting this evidence poorly, perhaps by being insufficiently careful about “alignment” as meaning “reflectively goal stable with predictable goals and predictable instabilities” vs “acts like a law-abiding citizen at the moment”.
]
to the extent developers succeed in creating faithful simulators
There’s a crux I have with Ryan which is “whether future capabilities will allow data-efficient long-horizon RL fine-tuning that generalizes well”. As of last time we talked about it, Ryan says we probably will, I say we probably won’t.
If we have the kind of generalizing ML that we can use to make faithful simulations, then alignment is pretty much solved. We make exact human uploads, and that’s pretty much it. This is one end of the spectrum on this question.
There are weaker versions, which I think are what Ryan believes will be possible. In a slightly weaker case, you don’t get something anywhere close to a human simulation, but you do get a machine that pursues the metric that you fine-tuned it to pursue, even out of distribution (with a relatively small amount of data).
But I think the evidence is against this. Long horizon tasks are currently difficult to successfully train on, unless you have dense intermediate feedback. Capabilities progress in the last decade has come from leaning heavily on dense intermediate feedback.
I expect long-horizon RL to remain pretty low data efficiency (i.e. take a lot of data before it generalizes well OOD).
My guess is that your core mistake is here:
When I say agents are “not egregiously misaligned,” I mean they mostly perform their work earnestly – in the same way humans are mostly earnest and vaguely try to do their job. Maybe agents are a bit sycophantic, but not more than the humans whom they would replace. Therefore, if agents are consistently “not egregiously misaligned,” the situation is no worse than if humans performed their research instead.
Obviously, all agents having undergone training to look “not egregiously misaligned”, will not look egregiously misaligned. You seem to be assuming that there is mostly a dichotomy between “not egregiously misaligned” and “conniving to satisfy some other set of preferences”. But there are a lot of messy places in between these two positions, including “I’m not really sure what I want” or <goals-that-are-highly-dependent-on-the-environment-e.g.-status-seeking>.
All AIs you train will be somewhere in this in between messy place. What you are hoping for is that if you put a group of these together, they will “self-correct” and force/modify each other to keep pursuing to the same goals-you-trained-them-to-look-like-they-wanted?
Is this basically correct? If so, this won’t work just because this is absolute chaos and the goals-you-trained-them-to-look-like-they-wanted aren’t enough to steer this chaotic system where you want it to go.
are these agents going to do sloppy research?
I think there were a few times where you are somewhat misreading your critics when they say “slop”. It doesn’t mean “bad”. It means something closer to “very subtly bad in a way that is difficult to distinguish from quality work”. Where the second part is the important part.
E.g. I find it difficult to use LLMs to help me do math or code weird algorithms, because they are good enough at outputting something that looks right. It feels like it takes longer to detect and fix their mistakes than it does to do it from scratch myself.
(Some) acceleration doesn’t require being fully competitive with humans while deference does.
Agreed. The invention of calculators was useful for research, and the invention of more tools will also be helpful.
I think AIs that can autonomously do moderate duration ML tasks (e.g., 1 week tasks), but don’t really have any interesting new ideas could plausibly speed up safety work by 5-10x if they were cheap and fast enough.
Maybe some kinds of “safety work”, but real alignment involves a human obtaining a deep understanding of intelligence and agency. The path to this understanding probably isn’t made of >90% moderate duration ML tasks. (You need >90% to get 5-10x because of communication costs, it’s often necessary to understand details of experiment implementation to get insight from them. And costs from the AI making mistakes and not quite doing the experiments right).
(vague memory from the in person discussions we had last year, might be inaccurate):
jeremy!2023: If you’re expecting AI to be capable enough to “accelerate alignment research” significantly, it’ll need to be a full-blown agent that learns stuff. And that’ll be enough to create alignment problems because data-efficient long-horizon generalization is not something we can do.
joshc!2023: No way, all you need is AI with stereotyped skills. Imagine how fast we could do interp experiments if we had AIs that were good at writing code but dumb in other ways!
...
joshc!now:
Training AI agents so they can improve their beliefs (e.g. do research) as well as the best humans can.
Seems like the reasoning behind your conclusions has changed a lot since we talked, but the conclusions haven’t changed much?
If you were an AI: Negative reward, probably a bad belief updating process.
In that case, what does the conditional goal look like when you translate it into a preference relation over outcomes?
We can’t reduce the domain of the utility function without destroying some information. If we tried to change the domain variables from [g, h, shutdown] to [g, shutdown], we wouldn’t get the desired behaviour. Maybe you have a particular translation method in mind?
I don’t mess up the medical test because true information is instrumentally useful to me, given my goals.
Yep that’s what I meant. The goal u is constructed to make information about h instrumentally useful for achieving u, even if g is poorly specified. The agent can prefer h over ~h or vice versa, just as we prefer a particular outcome of a medical test. But because of the instrumental (information) value of the test, we don’t interfere with it.
I think the utility indifference genre of solutions (which try to avoid preferences between shutdown and not-shutdown) are unnatural and create other problems. My approach allows the agent to shutdown even if it would prefer to be in the non-shutdown world.
With regards to the agent believing that it’s impossible to influence the probability that its plan passes validation
This is a misinterpretation. The agent entirely has true beliefs. It knows it could manipulate the validation step. It just doesn’t want to, because of the conditional shape of its goal. This is a common behaviour among humans, for example you wouldn’t mess up a medical test to make it come out negative, because you need to know the result in order to know what to do afterwards.
I propose: the best planners must break the beta.
Because if a planner is going to be the best, it needs to be capable of finding unusual (better!) plans. If it’s capable of finding those, there’s ~no benefit of knowing the conventional wisdom about how to do it (climbing slang: beta).
Edit: or maybe: good planners don’t need beta?
I think you’re wrong to be psychoanalysing why people aren’t paying attention to your work. You’re overcomplicating it. Most people just think you’re wrong upon hearing a short summary, and don’t trust you enough to spend time learning the details. Whether your scenario is important or not, from your perspective it’ll usually look like people are bouncing off for bad reasons.
For example, I read the executive summary. For several shallow reasons,[1] the scenario seemed unlikely and unimportant. I didn’t expect there to be better arguments further on. So I stopped. Other people have different world models and will bounce off for different reasons.
Which isn’t to say it’s wrong (that’s just my current weakly held guess). My point is just that even if you’re correct, the way it looks a priori to most worldviews is sufficient to explain why people are bouncing off it and not engaging properly.
Perhaps I’ll encounter information in the future that indicates my bouncing off was a mistake, and I’ll go back.
There are a couple of layers of maybes, so the scenario doesn’t seem likely. I expect power to be more concentrated. I expect takeoff to be faster. I expect capabilities to have a high cap. I expect alignment to be hard for any goal. Something about maintaining a similar societal structure without various chaotic game-board-flips seems unlikely. The goals-instilled-in-our-replacements are pretty specific (institution-aligned), and pretty obviously misaligned from overall human flourishing. Sure humans are usually myopic, but we do sometimes consider the consequences and act against local incentives.
I don’t know whether these reasons are correct, or how well you’ve argued against them. They’re weakly held and weakly considered, so I wouldn’t have usually written them down. They are just here to make my point more concrete.
The description of how sequential choice can be defined is helpful, I was previously confused by how this was supposed to work. This matches what I meant by preferences over tuples of outcomes. Thanks!
We’d incorrectly rule out the possibility that the agent goes for (B+,B).
There’s two things we might want from the idea of incomplete preferences:
To predict the actions of agents.
Because complete agents behave dangerously sometimes, and we want to design better agents with different behaviour.
I think modelling an agent as having incomplete preferences is great for (1). Very useful. We make better predictions if we don’t rule out the possibility that the agent goes for B after choosing B+. I think we agree here.
For (2), the relevant quote is:
As a general point, you can always look at a decision ex post and back out different ways to rationalise it. The nontrivial task is here prediction, using features of the agent.
If we can always rationalise a decision ex post as being generated by a complete agent, then let’s just build that complete agent. Incompleteness isn’t helping us, because the behaviour could have been generated by complete preferences.
Perhaps I’m misusing the word “representable”? But what I meant was that any single sequence of actions generate by the agent could also have been generated by an outcome-utility maximizer (that has the same world model). This seems like the relevant definition, right?
That’s not right
Are you saying that my description (following) is incorrect?
[incomplete preferences w/ caprice] would be equivalent to 1. choosing the best policy by ranking them in the partial order of outcomes (randomizing over multiple maxima), then 2. implementing that policy without further consideration.
Or are you saying that it is correct, but you disagree that this implies that it is “behaviorally indistinguishable from an agent with complete preferences”? If this is the case, then I think we might disagree on the definition of “behaviorally indistinguishable”? I’m using it like: If you observe a single sequence of actions from this agent (and knowing the agent’s world model), can you construct a utility function over outcomes that could have produced that sequence.
Or consider another example. The agent trades A for B, then B for A, then declines to trade A for B+. That’s compatible with the Caprice rule, but not with complete preferences.
This is compatible with a resolute outcome-utility maximizer (for whom A is a maxima). There’s no rule that says an agent must take the shortest route to the same outcome (right?).
As Gustafsson notes, if an agent uses resolute choice to avoid the money pump for cyclic preferences, that agent has to choose against their strict preferences at some point.
...
There’s no such drawback for agents with incomplete preferences using resolute choice.
Sure, but why is that a drawback? It can’t be money pumped, right? Agents following resolute choice often choose against their local strict preferences in other decision problems. (E.g. Newcomb’s). And this is considered an argument in favour of resolute choice.
I think it’s important to note the OOD push that comes from online-accumulated knowledge and reasoning. Probably you include this as a distortion or subversion, but that’s not quite the framing I’d use. It’s not taking a “good” machine and breaking it, it’s taking a slightly-broken-but-works machine and putting it into a very different situation where the broken parts become load-bearing.
My overall reaction is yep, this is a modal-ish pathway for AGI development (but there are other, quite different stories that seem plausible also).
I like this reason to be unsatisfied with the EUM theory of agency.
One of the difficulties in theorising about agency is that all the theories are flexible enough to explain anything. Each theory is incomplete and vague in some way, so this makes the problem worse, but even when you make a detailed model of e.g. active inference, it ends up being pretty much formally equivalent to EUM.
I think the solution to this is to compare theories using engineering desiderata. Our goal is ultimately to build a safe AGI, so we want a theory that helps us reason about safety desiderata.
One of the really important safety desiderata is some kind of goal stability. When we build a powerful agent, we don’t want it to change its mind about what’s important. It should act to achieve known, predictable outcomes, even when it discovers facts and concepts we don’t know about.
So my criticism of this research direction is that I don’t think it’ll be a good framework for making goal-stable agents. You want a framework that naturally models internal conflict of goals, and in particular you want to model this as conflict between agents. Conflict and cooperation between bounded, not-quite-rational agents is messy and hard to predict. Multi-agent systems are complex and detail dependent. Therefore it seems difficult to show that the overall agent will be stable.
(A reasonable response would be “but no proposed vague theories of bounded agency have this goal stability property, maybe this coalitional approach will turn out to help us come up with a solution”, and that’s true and fair enough, but I think research directions like this seem more promising).