Seth Herd

Karma: 5,124

Message me here or at seth dot herd at gmail dot com.

I was a researcher in cognitive psychology and cognitive neuroscience for about two decades. I studied complex human thought using neural network models of brain function. Now I’m applying what I’ve learned to the study of AI alignment.

Research overview:

Alignment is the study of how to give AIs goals or values aligned with ours, so we’re not in competition with our own creations. Recent breakthroughs in AI like ChatGPT make it possible we’ll have smarter-than-human AIs soon. So we’d better get ready. If their goals don’t align well enough with ours, they’ll probably outsmart us and get their way — and treat us as we do ants or monkeys. See this excellent intro video for more.

There are good and deep reasons to think that aligning AI will be very hard. But I think we have promising solutions that bypass most of those difficulties, and could be relatively easy to use for the types of AGI we’re most likely to develop first.

That doesn’t mean I think building AGI is safe. Humans often screw up complex projects, particularly on the first try, and we won’t get many tries. If it were up to me I’d Shut It All Down, but I don’t see how we could get all of humanity to stop building AGI. So I focus on finding alignment solutions for the types of AGI people are building.

In brief I think we can probably build and align language model agents (or language model cognitive architectures) even when they’re more autonomous and competent than humans. We’d use a stacking suite of alignment methods that can mostly or entirely avoid using RL for alignment, and achieve corrigibility (human-in-the-loop error correction) by having a central goal of following instructions. This scenario leaves multiple humans in charge of ASIs, creating some dangerous dynamics, but those problems might be navigated, too.

Bio

I did computational cognitive neuroscience research from getting my PhD in 2006 until the end of 2022. I’ve worked on computational theories of vision, executive function, episodic memory, and decision-making, using neural network models of brain function to integrate data across levels of analysis from psychological down to molecular mechanisms of learning in neurons, and everything in between. I’ve focused on the interactions between different brain neural networks that are needed to explain complex thought. Here’s a list of my publications.

I was increasingly concerned with AGI applications of the research, and reluctant to publish my full theories lest they be used to accelerate AI progress. I’m incredibly excited to now be working directly on alignment, currently as a research fellow at the Astera Institute.

More on approach

The field of AGI alignment is “pre-paradigmatic.” So I spend a lot of my time thinking about what problems need to be solved, and how we should go about solving them. Solving the wrong problems seems like a waste of time we can’t afford.

When LLMs suddenly started looking intelligent and useful, I noted that applying cognitive neuroscience ideas to them might well enable them to reach AGI and soon ASI levels. Current LLMs are like humans with no episodic memory for their experiences, and very little executive function for planning and goal-directed self-control. Adding those cognitive systems to LLMs can make them into cognitive architectures with all of humans’ cognitive capacities—a “real” artificial general intelligence that will soon be able to outsmart humans.

My work since then has convinced me that we could probably also align such an AGI so that it stays aligned even if it grows much smarter than we are. Instead of trying to give it a definition of ethics it can’t misunderstand or re-interpret (value alignment mis-specification), we’ll do the obvious thing: design it to follow instructions. It’s counter-intuitive to imagine an intelligent entity that wants nothing more than to follow instructions, but there’s no logical reason this can’t be done. An instruction-following proto-AGI can be instructed to act as a helpful collaborator in keeping it aligned as it grows smarter.

I increasingly suspect we should be actively working to build such intelligences. It seems like our our best hope of survival, since I don’t see how we can convince the whole world to pause AGI efforts, and other routes to AGI seem much harder to align since they won’t “think” in English. Thus far, I haven’t been able to engage enough careful critique of my ideas to know if this is wishful thinking, so I haven’t embarked on actually helping develop language model cognitive architectures.

Even though these approaches are pretty straightforward, they’d have to be implemented carefully. Humans often get things wrong on their first try at a complex project. So my p(doom) estimate of our long-term survival as a species is in the 50% range, too complex to call. That’s despite having a pretty good mix of relevant knowledge and having spent a lot of time working through various scenarios. So I think anyone with a very high or very low estimate is overestimating their certainty.

Seth Herd 10 Dec 2024 18:12 UTC
10 points
7
on: [Valence series] 1. Introduction
This series explains why we like some things and not others, including ideas. It’s cutting edge psychological theory.

Seth Herd 10 Dec 2024 3:39 UTC
4 points
0
on: The first AGI may be a good engineer but bad strategist
I applaud floating ideas like this.
It’s useful in thinking about the nature of AGI. Ultimately, something that isn’t limited to messy biology and a small skull is going to outpace us pretty quickly in all domains. We might get lucky if an early one fucks up a takeover attempt and that makes us suddenly alert enough to avoid a second one; but that seems moderately unlikely.
I think you’re right that humans have something that could be termed a wisdom advantage, and maybe this is what you meant: we’ve been evolved for millions to billions of years (depending on where the relevant mechanisms started) to avoid things that might get us killed. That could be termed wisdom. AGI is not evolved but designed and trained, so it might have some nasty blind spots. Current AI certainly does.
We have a fine-tuned intuitive sense of danger that prevents us from doing things that could get us killed (at least those our intution can grapple with. The Darwin award for doing bungee jumping with a cable is an example of things intution doesn’t do well). AGI does not.
That could be substituted with careful logic; as you say, thinking longer and harder can substitute for a lot.
As for engineering, that’s partly based on math but not for the majority of the job. There’s a lot of logic of materials, chemistry, etc depending on what you’re engineering. It’s systematic thought, but also creative thought. I’m scoring this roughly a push between early AGI and humans. Currently they can do math very well but just like we do: by using an external tool. So it’s not really better integrated.
Some of our intuitions for danger also apply to abstract situations like engineering, so we’ve got that advantage.
Again, more thought can substitute for talent.
I don’t think this the wisdom advantage is true for the reasons you give. Your central argument for human advantage in wisdom:
The only way to evaluate long term wisdom is to let the agent make a decision, wait years, and see if the agent’s goals have advanced.
mostly applies to AGI as well. We don’t learn wisdom from age, we learn it from paying serious attention to the stories and lessons from those who have tried and succeeded or failed.
We do get better at it with age, but that’s only in small part from trying and failing or succeeding at particular strategies ourselves. We have a vast library of failures and successes available in others’ stories. We learn more and get better at using them by learning as we age. The AGI can do that too, faster in some regards, but missing some others.

Seth Herd 9 Dec 2024 22:42 UTC
7 points
−2
on: Seth Herd’s Shortform
The psychological perspective on AGI alignment:

I wish we had more psychologists working on alignment. Our first AGIs are probably going to have complex mix of multiple goals, values, and habits, like humans. I’m not a clinical psychologist, and I don’t know of other alignment researchers who are. I wish we didn’t need them, but I’m afraid we do.

Understanding ML and training will be important, but we’re going to have to consider how context-dependent motivations interact in a complex, evolving system.

It looks like the first AGIs are fairly likely to be agents or cognitive architectures based on foundation models.

Those foundation models are shoggoths with a thousand faces, like the multitudes we each contain, but even more varied. And the scaffolding we surround those systems with to turn them into agents will make them more complex at the same time as offering some leverage points for understanding and directing them.

Edit: This is mostly framed in relation to formal methods, but it’s meant to address the larger group working on foundation model alignment, too. For them, the move I’m suggesting is toward thinking more about systems as entities rather than just models: minds that learn continuously (and so can change how they interpret the world and their goals) as well as having goals from user prompts, system prompts, RLHF/fine-tuning, and simulacra/or psuedo-goals from predictive base model training.

These systems will still be subject to the types of goal misspecification and optimization risks that classical agent foundations theory addresses; but they’ll also have goals, values, and habits competing in a complex, context-dependent way, like humans. When they learn continuously, either by episodic memory like improved vector databases, or by selecting important information for periodic fine-tuning, they will have not just a context-dependent set of tendencies, but a whole trajectory through value/goal interpretation/alignment space.

“We’re doomed then! Psychology has barely learned a thing in its couple hundred years!”, you might well be thinking. I agree that psychology can’t say much, and can’t say anything with certainty. But I don’t think that means the enterprise is doomed. More on that later.

Regardless of the consequences, I’m afraid this is a simple fact about the alignment problem. We are not likely to convince the whole world to start over and do it right by taking a more systematic approach. A slowdown seems out of reach, let alone a full pause long enough for an alternate approach to catch up.

Applying formal methods to the alignment problem in systems based on deep networks looks mostly like a pipe dream to me. I wish it weren’t so, but I desire to believe the truth.

And psychology is not a completely hopeless subject. Human behavior is predictable in broad form even while it’s highly unpredictable in its detail. An ethical human will generally do ethical things, even though they have some unethical thoughts. Humans are mostly coherent; our many urges have to fight each other.

So the burbling chaos of a shoggoth at its core might not be a dealbreaker, if there are ways to ensure that the majority of the agent’s motivations are aligned, and those aligned motivations can keep the rest from doing much damage.

Accepting that this is the type of AGI we need to align may be difficult if you’ve pinned your hopes for alignment on formal methods. But foundation model agents have upsides for alignment that are about as dramatically good as their inscrutable complexity is bad.

Reasons for guarded optimism about aligning the type of AGI we’re probably getting first are the subject of most of my other posts. Accepting that we need to think about alignment even when it’s messy is a first step toward succeeding if we can.

Seth Herd 9 Dec 2024 15:25 UTC
12 points
1
in reply to: habryka’s comment on: Habryka’s Shortform Feed
There’s another important effect here: a laggy time course of public opinion. I saw more popular press articles about EA than I ever have, linking SBF to them, but with a large lag after the events. So the early surveys showing a small effect happened before public conversation really bounced around the idea that SBFs crimes were motivated by EA utilitarian logic. The first time many people would remember hearing about EA would be from those later articles and discussions.

The effect probably amplified considerably over time as that hypothesis bounced through public discourse.

The original point stands but this is making the effect look much larger in this case.

Seth Herd 6 Dec 2024 23:38 UTC
2 points
0
in reply to: ChristianKl’s comment on: Why muscle tension can be unsexy
Excessive counter-tension would slow movement, yes. But muscles are usually maintaining some small level of tension. There’s not really a fully untensed state.

That said, I’m not actually sure that more counter-tension does speed up reaction times within normal ranges. There is such a thing as anticipatory muscle tension when an action is prepared, but that may be a by-product, not a functional way of speeding up reaction times. And I’m not sure it happens when no specific action is being prepared.

So I don’t know. If I’d thought a little harder about this, I’d have framed it as a hypothesis. Sorry!

Seth Herd 6 Dec 2024 23:30 UTC
2 points
0
in reply to: Jonas Hallgren’s comment on: Model Integrity: MAI on Value Alignment
I think we’re talking about two different things here. You’re talking about how to have agents interact well with each other, and how to make their principles of interaction legible to humans. I’m talking about how to make sure those agents don’t take over the world and kill us all, if/when they become smarter than we are in every important way.

Seth Herd 6 Dec 2024 23:25 UTC
2 points
0
in reply to: Jan_Kulveit’s comment on: Hierarchical Agency: A Missing Piece in AI Alignment
Ah. Now I understand why you’re going this direction.

I think a single human mind is modeled very poorly as a composite of multiple agents.

This notion is far more popular with computer scientists than with neuroscientists. We’ve known about it since Minsky and think about it; it just doesn’t seem to mostly be the case.

Sure you can model it that way, but it’s not doing much useful work.

I expect the same of our first AGIs as foundation model agents. They will have separate components, but those will not be well-modeled as agents. And they will have different capabilities and different tendencies, but neither of those are particularly agent-y either.

I guess the devil is in the details, and you might come up with a really useful analysis using the metaphor of subagents. But it seems like an inefficient direction.

Seth Herd 6 Dec 2024 20:21 UTC
8 points
5
on: AI Timelines
Knowing how much time we’ve got is important to using it well. It’s worth this sort of careful analysis.

I found most of this to be wasted effort based on too much of an outside view. The human brain gives neither an upper nor lower bound on the computation needed to achieve transformative AGI. Inside views that include gears-level models of how our first AGIs will function seem much more valuable; thus Daniel Kokatijlo’s predictions seem far better informed than the others here.

Outside views like “things take longer than they could, often a lot longer” are valuable, but if we look at predicting when other engineering feats would first be accomplished, good predictions would’ve taken both expert engineers and usually, those who could predict when funding and enthusiasm for the project would become available.

In any case, more careful timeline predictions like this would improve our odds.

Seth Herd 6 Dec 2024 20:11 UTC
4 points
0
on: My views on “doom”
Sharing well-informed, carefully-reasoned scenarios of how things might go right or wrong helps figure out how to steer the future.

Seth Herd 6 Dec 2024 20:09 UTC
6 points
2
on: Thoughts on “AI is easy to control” by Pope & Belrose
This post engages substantively and clearly with IMO the first or second most important thing we could be accomplishing on LW: making better estimates of how difficult alignment will be.

It analyzes how people who know a good deal about alignment theory could say something like “AI is easy to control” in good faith—and why that’s wrong, in both senses.

What Belrose and Pope were mostly saying, without being explicit about it, is that current AI is easy to control, then extrapolating from there by basically assuming we won’t make any changes to AI in the future that might dramatically change this situation.

This post addresses this and more subtle points, clarifying the discussion.

Seth Herd 6 Dec 2024 20:01 UTC
5 points
0
in reply to: Sinclair Chen’s comment on: Sinclair Chen’s Shortform
I think you are probably attending more often to sensory experiences, and thereby both creating and remembering more detailed representations of physical reality.

You are probably doing less abstract thought, since the number of seconds in a day hasn’t changed.

Which do you want to spend more time on? And which sorts? It’s a pretty personal question. I like to try to make my abstract thought productive (relative to my values), freeing up some time to enjoy sensory experiences.

I’m not sure there’s a difference in representational density in doing sensory experience vs. abstract thought. Maybe there is. One factor in making it seem like you’re having more sensory experience is how much you can remember after a set amount of time; another is whether each moment seems more intense by having strong emotional experience attached to it.

Or maybe you mean something different by more subjective experience.

Seth Herd 6 Dec 2024 19:56 UTC
5 points
1
in reply to: Sinclair Chen’s comment on: Sinclair Chen’s Shortform
It depends entirely on what you mean by consciousness. The term is used for several distinct things. If my mom had lost her sense of individuality but was still having a vivid experience of life, I’d keep valuing her. If she was no longer having a subjective experience (which would pretty much require being unconscious since her brain is producing an experience as part of how it works to do stuff), I would no longer value her but consider her already gone.

Seth Herd 6 Dec 2024 19:50 UTC
4 points
2
on: Careless talk on US-China AI competition? (and criticism of CAIS coverage)
This post was important in retrospect.

If only we’d all read it and listened.

The central point was “don’t fan the flames of an AI race with China by carelessly suggesting it’s already happening or inevitable”.

The central example is that saying something like “China has made numerous attempts to bypass restrictions on chip imports” is inflammatory when the truth is something more like “some companies in China have been getting chips in ways that bypass restrictions on chip imports, but we don’t think the Chinese government probably had anything to do with it”.

Seth Herd 6 Dec 2024 2:51 UTC
2 points
0
in reply to: rosehadshar’s comment on: Should there be just one western AGI project?
Maybe it’s overconfident. I’m not even sure if I hope I’m right. A Trump Executive Branch, or anything close, in charge of AGI seems even worse than Sam Altman or similar setting themselves up as god-emperor.
The central premises here are
- slow enough takeoff
  - likely but not certain
- Sufficient intelligence in government
  - Politicians aren’t necessarily that smart or forward-looking
  - National security professionals are.
- Visible takeoff- the public is made aware
  - This does seem more questionable.
  - But OpenAI is currently full of leaks; keeping human-level AGI secret long enough for it to take over the world before the national security apparatus knows seems really hard.
Outside of all that, could there be some sort of comedy of errors or massive collective and individual idiocy that prevents the government from doing its job in a very obvious (in retrospect at least) case?
Yeah, it’s possible. History, people, and organizations are complex and weird.

Seth Herd 6 Dec 2024 2:45 UTC
4 points
0
in reply to: AnthonyC’s comment on: Should there be just one western AGI project?
Thank you!

I don’t have time; I’m terribly slow at writing posts, and I’m behind on my actual paid work on alignment.

If you find this compelling and want to turn it into a top-level post, I’d appreciate it!

The fact that this got some disagreement votes means it’s probably more debatable than I was initially thinking.

Seth Herd 6 Dec 2024 2:40 UTC
2 points
0
on: Model Integrity: MAI on Value Alignment
Now I’ve actually skimmed the article. I was not wrong in my initial guess.

The central claim of the article:

This distinction becomes crucial as AI systems become more powerful—many would prefer [sic—not?] a compliant assistant, but a co-founder with integrity.

Is comically poorly chosen.

Actual individuals I know have frequently thought they’d like a co-founder with integrity, only to change their mind as that co-founders’ integrity causes them to fight for control of the shared project.

This is an excellent metaphor for the alignment problem, and the problem with value alignment as a goal (see my other comment). If you get it just a little bit wrong, you’ll regret it. A lot. Even if your “co-founder with integrity” merely seizes control of your shared project peacefully, rather than doing what a superintelligence would, and doing it by any means necessary.

To put it another way: this proposal shows the problem with thinking about “aligning” language models and limited language model agent: if you don’t think about the next step, creating a fully competent, autonomous entity, your alignment efforts are dangerous, not useful.

What people really want is an assistant/cofounder that is highly competent and as autonomous as you tell them to be—until you tell them to stop. Integrity means pursuing your values even when your friends beg you to stop. That is exactly what we do not want from AGI.

Seth Herd 6 Dec 2024 2:24 UTC
2 points
0
in reply to: Jonas Hallgren’s comment on: Model Integrity: MAI on Value Alignment
I’m sure that heuristic-like policies will be useful for multi-agent coordination.

Those heuristic policies can be given, and probably should, as instructions.

The problem is that if you set a capable agent in motion following heuristics or any other central motivation, you’ll probably discover that you got your definitions at least a little wrong—and those differences grow as they’re extrapolated further as the agent learns.

This is the classic problem with alignment. Half-steps toward making autonomous agents “do what we want” are very likely to lead toward disaster. Keeping a human in the loop by making instruction-following the central, top-level goal, and everything else we might want as a contingent subgoal.

This is the classic concept of corribibility. Being receptive to and following new instructions is the top-level goal that enables it.

Seth Herd 6 Dec 2024 2:17 UTC
2 points
0
in reply to: David Johnston’s comment on: Model Integrity: MAI on Value Alignment
The first. Value alignment is much harder. But it will be vastly easier with smarter-than-human help. So there are two difficult steps, and it’s clear which one should be tackled first.

The difficulty with value alignment is both in figuring out what we actually want, and then figuring out how to make those values stable in mind that changes as it learns new things.

Seth Herd 5 Dec 2024 21:25 UTC
3 points
0
in reply to: TristanTrim’s comment on: Seth Herd’s Shortform
I certainly don’t expect people to read a bunch of stuff before engaging! I’m really pleased that you’ve read so much of my stuff. I’ll get back to these conversations soon hopefully, I’ve had to focus on new posts.

I think your feelings about math are shared by a lot of the alignment community. I like the way you’ve expressed those intuitions.

I think math might be the best tool to solve alignment if we had unlimited time—but it looks like we very much do not.

Seth Herd 5 Dec 2024 20:46 UTC
8 points
0
on: Model Integrity: MAI on Value Alignment
I’m just going to leave my generic comment on value alignment: working on it is largely a waste of time, because Instruction-following AGI is easier and more likely than value aligned AGI.

Instruction-following includes issuing new instructions to correct mistaken interpretations of previous instructions (corrigibility). That alone makes it vastly safer than taking a swing at value alignment and hoping you got it right on the first try—that agent will true to pursue its values, whether you got the right ones in there or not.

Even if instruction-following wasn’t a lot safer, (which I’m pretty sure it is, at least in the short term), the people in charge of actually creating and deploying AGIs would use that logic as an excuse to make AGI follow their instructions and therefore their values, rather than the average values of all of humanity (whatever that would mean when you extrapolate it into an unknown future).

And this approach doesn’t prevent going for value alignment in the long term, it just allows superintelligent help in getting it right on the first try. See Intent alignment as a stepping-stone to value alignment.

I have yet to see any real counterarguments to this claim (other than that we should pursue value alignment because it’s better for all of humanity and would prevent misuse of AGI/ASI, which I agree with—as soon as it’s even modestly safe).

If this take is mistaken, I want to know.