johnswentworth

Karma: 50,811

johnswentworth 10 Jan 2025 21:58 UTC
LW: 31 AF: 15
16
AF
in reply to: Buck’s comment on: johnswentworth’s Shortform
A few problems with this frame.
First: you’re making reasonably-pessimistic assumptions about the AI, but very optimistic assumptions about the humans/organization. Sure, someone could look for the problem by using AIs to do research on other subject that we already know a lot about. But that’s a very expensive and complicated project—a whole field, and all the subtle hints about it, need to be removed from the training data, and then a whole new model trained! I doubt that a major lab is going to seriously take steps much cheaper and easier than that, let alone something that complicated.
One could reasonably respond “well, at least we’ve factored apart the hard technical bottleneck from the part which can be solved by smart human users or good org structure”. Which is reasonable to some extent, but also… if a product requires a user to get 100 complicated and confusing steps all correct in order for the product to work, then that’s usually best thought of as a product design problem, not a user problem. Making the plan at least somewhat robust to people behaving realistically less-than-perfectly is itself part of the problem.
Second: looking for the problem by testing on other fields itself has subtle failure modes, i.e. various ways to Not Measure What You Think You Are Measuring. A couple off-the-cuff examples:
- A lab attempting this strategy brings in some string theory experts to evaluate their attempts to rederive string theory with AI assistance. But maybe (as I’ve heard claimed many times) string theory is itself an empty echo-chamber, and some form of sycophancy or telling people what they want to hear is the only way this AI-assisted attempt gets a good evaluation from the string theorists.
- It turns out that fields-we-don’t-understand mostly form a natural category distinct from fields-we-do-understand, or that we don’t understand alignment precisely because our existing tools which generalize across many other fields don’t work so well on alignment. Either of those would be a (not-improbable-on-priors) specific reason to expect that our experience attempting to rederive some other field does not generalize well to alignment.
And to be clear, I don’t think of these as nitpicks, or as things which could go wrong separately from all the things originally listed. They’re just the same central kinds of failure modes showing up again, and I expect them to generalize to other hacky attempts to tackle the problem.
Third: it doesn’t really matter whether the model is trying to make it hard for us to notice the problem. What matters is (a) how likely we are to notice the problem “by default”, and (b) whether the AI makes us more or less likely to notice the problem, regardless of whether it’s trying to do so. The first story at top-of-thread is a good central example here:
- Perhaps the path to superintelligence looks like applying lots of search/optimization over shallow heuristics. Then we potentially die to things which aren’t smart enough to be intentionally deceptive, but nonetheless have been selected-upon to have a lot of deceptive behaviors (via e.g. lots of RL on human feedback).
Generalizing that story to attempts to outsource alignment work to earlier AI: perhaps the path to moderately-capable intelligence looks like applying lots of search/optimization over shallow heuristics. If the selection pressure is sufficient, that system may well learn to e.g. be sycophantic in exactly the situations where it won’t be caught… though it would be “learning” a bunch of shallow heuristics with that de-facto behavior, rather than intentionally “trying” to be sycophantic in exactly those situations. Then the sycophantic-on-hard-to-verify-domains AI tells the developers that of course their favorite ideas for aligning the next generation of AI will work great, and it all goes downhill from there.

johnswentworth 10 Jan 2025 19:27 UTC
LW: 20 AF: 10
10
AF
in reply to: Buck’s comment on: johnswentworth’s Shortform
scheming is the main plausible source of catastrophic risk from the first AIs that either pose substantial misalignment risk or that are extremely useful...
Seems quite wrong. The main plausible source of catastrophic risk from the first AIs that either pose substantial misalignment risk or that are extremely useful is that they cause more powerful AIs to be built which will eventually be catastrophic, but which have problems that are not easily iterable-upon (either because problems are hidden, or things move quickly, or …).
And causing more powerful AIs to be built which will eventually be catastrophic is not something which requires a great deal of intelligent planning; humanity is already racing in that direction on its own, and it would take a great deal of intelligent planning to avert it. This story, for example:
- People try to do the whole “outsource alignment research to early AGI” thing, but the human overseers are themselves sufficiently incompetent at alignment of superintelligences that the early AGI produces a plan which looks great to the overseers (as it was trained to do), and that plan totally fails to align more-powerful next-gen AGI at all. And at that point, they’re already on the more-powerful next gen, so it’s too late.
This story sounds clearly extremely plausible (do you disagree with that?), involves exactly the sort of AI you’re talking about (“the first AIs that either pose substantial misalignment risk or that are extremely useful”), but the catastropic risk does not come from that AI scheming. It comes from people being dumb by default, the AI making them think it’s ok (without particularly strategizing to do so), and then people barreling ahead until it’s too late.
These other problems all seem like they require the models to be way smarter in order for them to be a big problem.
Also seems false? Some of the relevant stories:
- As mentioned above, the “outsource alignment to AGI” failure-story was about exactly the level of AI you’re talking about.
- In worlds where hard takeoff naturally occurs, it naturally occurs when AI is just past human level in general capabilities (and in particular AI R&D), which I expect is also roughly the same level you’re talking about (do you disagree with that?).
- The story about an o1-style AI does not involve far possibilities and would very plausibly kick in at-or-before the first AIs that either pose substantial misalignment risk or that are extremely useful.
A few of the other stories also seem debatable depending on trajectory of different capabilities, but at the very least those three seem clearly potentially relevant even for the first highly dangerous or useful AIs.

johnswentworth 10 Jan 2025 15:10 UTC
LW: 6 AF: 4
2
AF
in reply to: evhub’s comment on: Fabien’s Shortform
Yeah, I’m aware of that model. I personally generally expect the “science on model organisms”-style path to contribute basically zero value to aligning advanced AI, because (a) the “model organisms” in question are terrible models, in the sense that findings on them will predictably not generalize to even moderately different/stronger systems (like e.g. this story), and (b) in practice IIUC that sort of work is almost exclusively focused on the prototypical failure story of strategic deception and scheming, which is a very narrow slice of the AI extinction probability mass.

johnswentworth 10 Jan 2025 15:09 UTC
30 points
4
in reply to: johnswentworth’s comment on: johnswentworth’s Shortform
Also (separate comment because I expect this one to be more divisive): I think the scheming story has been disproportionately memetically successful largely because it’s relatively easy to imagine hacky ways of preventing an AI from intentionally scheming. And that’s mostly a bad thing; it’s a form of streetlighting.

johnswentworth 10 Jan 2025 15:08 UTC
LW: 116 AF: 41
24
AF
on: johnswentworth’s Shortform
I think a very common problem in alignment research today is that people focus almost exclusively on a specific story about strategic deception/scheming, and that story is a very narrow slice of the AI extinction probability mass. At some point I should probably write a proper post on this, but for now here are few off-the-cuff example AI extinction stories which don’t look like the prototypical scheming story. (These are copied from a Facebook thread.)
- Perhaps the path to superintelligence looks like applying lots of search/optimization over shallow heuristics. Then we potentially die to things which aren’t smart enough to be intentionally deceptive, but nonetheless have been selected-upon to have a lot of deceptive behaviors (via e.g. lots of RL on human feedback).
- The “Getting What We Measure” scenario from Paul’s old “What Failure Looks Like” post.
- The “fusion power generator scenario”.
- Perhaps someone trains a STEM-AGI, which can’t think about humans much at all. In the course of its work, that AGI reasons that an oxygen-rich atmosphere is very inconvenient for manufacturing, and aims to get rid of it. It doesn’t think about humans at all, but the human operators can’t understand most of the AI’s plans anyway, so the plan goes through. As an added bonus, nobody can figure out why the atmosphere is losing oxygen until it’s far too late, because the world is complicated and becomes more so with a bunch of AIs running around and no one AI has a big-picture understanding of anything either (much like today’s humans have no big-picture understanding of the whole human economy/society).
- People try to do the whole “outsource alignment research to early AGI” thing, but the human overseers are themselves sufficiently incompetent at alignment of superintelligences that the early AGI produces a plan which looks great to the overseers (as it was trained to do), and that plan totally fails to align more-powerful next-gen AGI at all. And at that point, they’re already on the more-powerful next gen, so it’s too late.
- The classic overnight hard takeoff: a system becomes capable of self-improving at all but doesn’t seem very alarmingly good at it, somebody leaves it running overnight, exponentials kick in, and there is no morning.
- (At least some) AGIs act much like a colonizing civilization. Plenty of humans ally with it, trade with it, try to get it to fight their outgroup, etc, and the AGIs locally respect the agreements with the humans and cooperate with their allies, but the end result is humanity gradually losing all control and eventually dying out.
- Perhaps early AGI involves lots of moderately-intelligent subagents. The AI as a whole mostly seems pretty aligned most of the time, but at some point a particular subagent starts self-improving, goes supercritical, and takes over the rest of the system overnight. (Think cancer, but more agentic.)
- Perhaps the path to superintelligence looks like scaling up o1-style runtime reasoning to the point where we’re using an LLM to simulate a whole society. But the effects of a whole society (or parts of a society) on the world are relatively decoupled from the things-individual-people-say-taken-at-face-value. For instance, lots of people talk a lot about reducing poverty, yet have basically-no effect on poverty. So developers attempt to rely on chain-of-thought transparency, and shoot themselves in the foot.

johnswentworth 8 Jan 2025 16:31 UTC
5 points
−2
in reply to: Thane Ruthenis’s comment on: What Indicators Should We Watch to Disambiguate AGI Timelines?
- Tricks that work on smaller scales often don’t generalize to larger scales.
- Tricks that work on larger scales often don’t work on smaller scales (due to bigger ML models having various novel emergent properties).
My understanding is that these two claims are mostly false in practice. In particular, there have been a few studies (like e.g. this) which try to run yesterday’s algorithms with today’s scale, and today’s algorithms with yesterday’s scale, in order to attribute progress to scale vs algorithmic improvements. I haven’t gone through those studies in very careful detail, but my understanding is that they pretty consistently find today’s algorithms outperform yesterday’s algorithms even when scaled down, and yesterday’s algorithms underperform today’s even when scaled up. So unless I’ve badly misunderstood those studies, the mental model in which different tricks work best on different scales is basically just false, at least at the range of different scales the field has gone through in the past ~decade.
That said, there are cases where I could imagine Ilya’s claim making sense, e.g. if the “experiments” he’s talking about are experiments in using the net rather than training the net. Certainly one can do qualitatively different things with GPT4 than GPT2, so if one is testing e.g. a scaffolding setup or a net’s ability to play a particular game, then one needs to use the larger net. Perhaps that’s what Ilya had in mind?

johnswentworth 7 Jan 2025 18:39 UTC
20 points
3
in reply to: Alexander Gietelink Oldenziel’s comment on: Alexander Gietelink Oldenziel’s Shortform
I don’t remember the details, but IIRC ZIP is mostly based on Lempel-Ziv, and it’s fairly straightforward to modify Lempel-Ziv to allow for efficient local decoding.
My guess would be that the large majority of the compression achieved by ZIP on NN weights is because the NN weights are mostly-roughly-standard-normal, and IEEE floats are not very efficient for standard normal variables. So ZIP achieves high compression for “kinda boring reasons”, in the sense that we already knew all about that compressibillity but just don’t leverage it in day-to-day operations because our float arithmetic hardware uses IEEE.

johnswentworth 6 Jan 2025 1:00 UTC
10 points
−3
on: Is “hidden complexity of wishes problem” solved?
Short answer: no.
Longer answer: we need to distinguish between two things people might have in mind when they say that LLMs “solve the hidden complexity of wishes problem”.
First, one might imagine that LLMs “solve the hidden complexity of wishes problem” because they’re able to answer natural-language questions about humans’ wishes much the same way a human would. Alas, that’s a misunderstanding of the problem. If the ability to answer natural-language questions about humans’ wishes in human-like ways were all we needed in order to solve the “hidden complexity of wishes” problem, then a plain old human would be a solution to the problem; one could just ask the human. Part of the problem is that humans themselves understand their own wishes so poorly that their own natural-language responses to questions are not a safe optimization target either.
Second, one might imagine LLMs “solve the hidden complexity of wishes problem” because when we ask an LLM to solve a problem, it solves the problem in a human-like way. It’s not about the LLM’s knowledge of humans’ (answers to questions about their) wishes, but rather about LLMs solving problems and optimizing in ways which mimic human problem-solving and optimization. And that does handle the hidden complexity problem… but only insofar as we continue to use LLMs in exactly the same way. If we start e.g. scaling up o1-style methods, or doing HCH, or put the LLM in some other scaffolding so we’re not directly asking it to solve a problem and then using the human-like solutions it generates… then we’re (potentially) back to having a hidden complexity problem. For each of those different methods of using the LLM to solve problems, we have to separately consider whether the human-mimicry properties of the LLM generalize to that method enough to handle the hidden complexity issue.
(Toy example: suppose we use LLMs to mimic a very very large organization. Like most real-world organizations, information and constraints end up fairly siloed/modularized, so some parts of the system are optimizing for e.g. “put out the fire” and don’t know that grandma’s in the house at all. And then maybe that part of the system chooses a nice efficient fire-extinguishing approach which kills grandma, like e.g. collapsing the house and then smothering it.)
And crucially: if AI is ever to solve problems too hard for humans (which is one of its main value propositions), we’re definitely going to need to do something with LLMs besides use them to solve problems in human-like ways.

johnswentworth 5 Jan 2025 12:22 UTC
3 points
0
in reply to: brambleboy’s comment on: Values Are Real Like Harry Potter
If you can still have values without reward signals that tell you about them, then doesn’t that mean your values are defined by more than just what the “screen” shows? That even if you could see and understand every part of someone’s reward system, you still wouldn’t know everything about their values?
No.
An analogy: suppose I run a small messaging app, and all the users’ messages are stored in a database. The messages are also cached in a faster-but-less-stable system. One day the database gets wiped for some reason, so I use the cache to repopulate the database.
In this example, even though I use the cache to repopulate the database in this one weird case, it is still correct to say that the database is generally the source of ground truth for user messages in the system; the weird case is in fact weird. (Indeed, that’s exactly how software engineers would normally talk about it.)
Spelling out the analogy: in a human brain in ordinary operation, our values (I claim) ground out in the reward stream, analogous to the database. There’s still a bunch of “caching” of values, and in weird cases like the one you suggest, one might “repopulate” the reward stream from the “cached” values elsewhere in the system. But it’s still correct to say that the reward stream is generally the source of ground truth for values in the system; the weird case is in fact weird.

johnswentworth 2 Jan 2025 23:08 UTC
6 points
4
in reply to: brambleboy’s comment on: Values Are Real Like Harry Potter
Good question.
First and most important: if you know beforehand that you’re at risk of entering such a state, then you should (according to your current values) probably put mechanisms in place to pressure your future self to restore your old reward stream. (This is not to say that fully preserving the reward stream is always the right thing to do, but the question of when one shouldn’t conserve one’s reward stream is a separate one which we can factor apart from the question at hand.)
… and AFAICT, it happens that the human brain already works in a way which would make that happen to some extent by default. In particular, most of our day-to-day planning draws on cached value-estimates which would still remain, at least for a time, even if the underlying rewards suddenly zeroed out.
… and it also happens that other humans, like e.g. your friends, would probably prefer (according to their values) for you to have roughly-ordinary reward signals rather than zeros. So that would also push in a similar direction.
And again, you might decide to edit the rewards away from the original baseline afterwards. But that’s a separate question.
On the other hand, consider a mind which was never human in the first place, never had any values or rewards, and is given the same ability to modify its rewards as in your hypothetical. Then—I claim—that mind has no particular reason to favor any rewards at all. (Although we humans might prefer that it choose some particular rewards!)
Your question touched on several different things, so let me know if that missed the parts you were most interested in.

johnswentworth 2 Jan 2025 15:06 UTC
11 points
8
in reply to: Lucius Bushnaq’s comment on: The Plan − 2024 Update
Interpretability is not currently trying to look at AIs to determine whether they will kill us. That’s way too advanced for where we’re at.
Right, and that’s a problem. There’s this big qualitative gap between the kinds of questions interp is even trying to address today, and the kinds of questions it needs to address. It’s the gap between talking about stuff inside the net, and talking about stuff in the environment (which the stuff inside the net represents).
And I think the focus on LLMs is largely to blame for that gap seeming “way too advanced for where we’re at”. I expect it’s much easier to cross if we focus on image models instead.
(And to be clear, even after crossing the internals/environment gap, there will still be a long ways to go before we’re ready to ask about e.g. whether an AI will kill us. But the internals/environment gap is the main qualitative barrier I know of; after that it should be more a matter of iteration and ordinary science.)

johnswentworth 2 Jan 2025 12:28 UTC
5 points
0
in reply to: Jonas Hallgren’s comment on: The Plan − 2024 Update
Great question.
First, I do think of systems biology as one of the main fields where the tools we’re developing should apply.
But I would not say that fields like artificial life or computational biology have done much to cross the theory-practice gap, mainly because they have little theory at all. Artificial life, at least what I’ve seen of it, is mostly just running various cool simulations without any generalizable theory at all. When there is “theory”, it’s usually e.g. someone vibing about the free energy principle, but it’s just vibing without any gears behind it. Levin’s work is very cool, but doesn’t seem to involve any theory beyond kinda vibing about communication and coordination. (Though of course all this could just be my own ignorance talking, please do let me know if there’s some substantive theory I haven’t heard of.)
Uri Alon’s book is probably the best example I know of actual theory in biology, but most of the specifics are too narrow for our purposes. Some of it is a useful example to keep in mind.

johnswentworth 2 Jan 2025 12:19 UTC
8 points
3
in reply to: Neel Nanda’s comment on: The Plan − 2024 Update
The preprocessing itself is one of the main important things we need to understand (I would even argue it’s the main important thing), if our interpretability methods are ever going to tell us about how the stuff-inside-the-net relates to the stuff-in-the-environment (which is what we actually care about).

johnswentworth 1 Jan 2025 14:31 UTC
13 points
2
in reply to: Lucius Bushnaq’s comment on: The Plan − 2024 Update
But why do we care more about statistical relationships between physical humans and dogs than statistical relationships between the word “human” and the word “dog” as characters on your screen?
An overly-cute but not completely wrong answer: because I care about whether AI kills all the physical humans, not whether something somewhere writes the string “kill all the humans”. My terminal-ish values are mostly over the physical stuff.
I think the point you’re trying to make is roughly “well, it’s all pretty entangled with the physical stuff anyway, so why favor one medium or another? Instrumentally, either suffices.”. And the point I’m trying to make in response is “it matters a lot how complicated the relationship is between the medium and the physical stuff, because terminally it’s the physical we care about, so instrumentally stuff that’s more simply related to the physical stuff is a lot more useful to understand.”.

johnswentworth 1 Jan 2025 11:34 UTC
8 points
0
in reply to: Erik Jenner’s comment on: The Plan − 2024 Update
I don’t know how much the difficulty of crossing the theory-practice gap has deviated from your expectations since then.
It’s been pretty on-par.
But I would indeed be worried that a lot of the difficulty is going to be in getting any good results for deep learning, and that finding additional theoretical/conceptual results in other settings doesn’t constitute much progress on that.
Amusingly, I tend to worry more about the opposite failure mode: findings on today’s nets won’t generalize to tomorrow’s nets (even without another transformers-level paradigm shift), and therefore leveraging evidence from other places is the only way to do work which will actually be relevant.
(More accurately, I worry that the relevance or use-cases of findings on today’s nets won’t generalize to tomorrow’s nets. Central example: if we go from a GPT-style LLM to a much bigger o1/o3-style model which is effectively simulating a whole society talking to each other, then the relationship between the tokens and the real-world effects of the system changes a lot. So even if work on the GPT-style models tells us something about the o1/o3-style models, its relevance is potentially very different.)
I assume that was some other type of experiment involving image generators? (and the notion of “working well” there isn’t directly comparable to what you tried now?)
Yeah, that was on a little MNIST net. And the degree of success I saw in that earlier experiment was actually about on par with what we saw in our more recent experiments, our bar was just quite a lot higher this time around. This time we were aiming for things like e.g. “move one person’s head” rather than “move any stuff in any natural way at all”.

johnswentworth 1 Jan 2025 11:24 UTC
10 points
1
in reply to: Lucius Bushnaq’s comment on: The Plan − 2024 Update
Interpretability on an LLM might, for example, tell me a great deal about the statistical relationships between the word “human” and the word “dog” in various contexts. And the “trivial” sense in which this tells me about physical stuff is that the texts in question are embedded in the world—they’re characters on my screen, for instance.
The problem is that I don’t care that much about the characters on my screen in-and-of themselves. I mostly care about the characters on my screen insofar as they tell me about other things, like e.g. physical humans and dogs.
So, say I’m doing interpretability work on an LLM, and I find some statistical pattern between the word “human” and the word “dog”. (Flag: this is oversimplified compared to actual interp.) What does that pattern tell me about physical humans and physical dogs, the things I actually care about? How does that pattern even relate to physical humans and physical dogs? Well shit, that’s a whole very complicated question in its own right.
On the other hand, if I’m doing interp work on an image generator… I’m forced to start lower-level, so by the time I’m working with things like humans and dogs I’ve already understood a whole lot of stuff about the lower-level patterns which constitute humans and dogs (which is itself probably useful, that’s exactly the sort of thing I want to learn about). Then I find some relationship between the human parts of the image and the dog parts of the image, and insofar as the generator was trained on real-world images, that much more directly tells me about physical humans and dogs and how they relate statistically (like e.g. where they’re likely to be located relative to each other).

johnswentworth 1 Jan 2025 1:25 UTC
25 points
16
in reply to: Lucius Bushnaq’s comment on: The Plan − 2024 Update
The thing I ultimately care about is patterns in our physical world, like trees or humans or painted rocks. I am interested in patterns in speech/text (like e.g. bigram distributions) basically only insofar as they tell something useful about patterns in the physical world. I am also interested in patterns in pixels only insofar as they tell something useful about patterns in the physical world. But it’s a lot easier to go from “pattern in pixels” to “pattern in physical world” than from “pattern in tokens” to “pattern in physical world”. (Excluding the trivial sense in which tokens are embedded in the physical world and therefore any pattern in tokens is a pattern in the physical world; that’s not what we’re talking about here.)
That’s the sense in which pixels are “closer to the metal”, and why I care about that property.
Does that make sense?

johnswentworth 31 Dec 2024 14:39 UTC
14 points
2
in reply to: JuliaHP’s comment on: The Field of AI Alignment: A Postmortem, and What To Do About It
I currently think broad technical knowledge is the main requisite, and I think self-study can suffice for the large majority of that in principle. The main failure mode I see would-be autodidacts run into is motivation, but if you can stay motivated then there’s plenty of study materials.
For practice solving novel problems, just picking some interesting problems (preferably not AI) and working on them for a while is a fine way to practice.

The Plan − 2024 Update

johnswentworth31 Dec 2024 13:29 UTC

113 points

27 comments4 min readLW link

johnswentworth 30 Dec 2024 22:19 UTC
13 points
4
in reply to: aysja’s comment on: The Field of AI Alignment: A Postmortem, and What To Do About It
I’m not convinced that the “hard parts” of alignment are difficult in the standardly difficult, g-requiring way that e.g., a physics post-doc might possess.
To be clear, I wasn’t talking about physics postdocs mainly because of raw g. Raw g is a necessary element, and physics postdocs are pretty heavily loaded on it, but I was talking about physics postdocs mostly because of the large volume of applied math tools they have.
The usual way that someone sees footholds on the hard parts of alignment is to have a broad enough technical background that they can see some analogy to something they know about, and try borrowing tools that work on that other thing. Thus the importance of a large volume of technical knowledge.