I’m interested in doing in-depth dialogues to find cruxes. Message me if you are interested in doing this.
I do alignment research, mostly stuff that is vaguely agent foundations. Currently doing independent alignment research on ontology identification. Formerly on Vivek’s team at MIRI. Most of my writing before mid 2023 is not representative of my current views about alignment difficulty.
Dialogues are more difficult to create (if done well between people with different beliefs), and are less pleasant to read, but are often higher value for reaching true beliefs as a group.
Dialogues seem under-incentivised relative to comments, given the amount of effort involved. Maybe they would get more karma if we could vote on individual replies, so it’s more like a comment chain?
This could also help with skimming a dialogue because you can skip to the best parts, to see whether it’s worth reading the whole thing.
The ideal situation understanding-wise is that we understand AI at an algorithmic level. We can say stuff like: there are X,Y,Z components of the algorithm, and X passes (e.g.) beliefs to Y in format b, and Z can be viewed as a function that takes information in format w and links it with… etc. And infrabayes might be the theory you use to explain what some of the internal datastructures mean. Heuristic arguments might be how some subcomponent of the algorithm works. Most theoretical AI work (both from the alignment community and in normal AI and ML theory) potentially has relevance, but it’s not super clear which bits are most likely to be directly useful.
This seems like the ultimate goal of interp research (and it’s a good goal). Or, I think the current story for heuristic arguments is using them to “explain” a trained neural network by breaking it down into something more like an X,Y,Z components explanation.
At this point, we can analyse the overall AI algorithm, and understand what happens when it updates its beliefs radically, or understand how its goals are stored and whether they ever change. And we can try to work out whether the particular structure will change itself in bad-to-us ways if it could self-modify. This is where it looks much more theoretical, like theoretical analysis of algorithms.
(The above is the “understood” end of the axis. The “not-understood” end looks like making an AI with pure evolution, with no understanding of how it works. There are many levels of partial understanding in between).
This kind of understanding is a prerequisite for the scheme in my post. This scheme could be implemented by modifying a well-understood AI.
Also what is its relation to natural language?
Not sure what you’re getting at here.
Fair enough, good points. I guess I classify these LLM agents as “something-like-an-LLM that is genuinely creative”, at least to some extent.
Although I don’t think the first example is great, seems more like a capability/observation-bandwidth issue.
I’m not sure how this is different from the solution I describe in the latter half of the post.
Great comment, agreed. There was some suggestion of (3), and maybe there was too much. I think there are times when expectations about the plan are equivalent to literal desires about how the task should be done. For making coffee, I expect that it won’t create much noise. But also, I actually want the coffee-making to not be particularly noisy, and if it’s the case that the first plan for making coffee also creates a lot of noise as a side effect, this is a situation where something in the goal specification has gone horribly wrong (and there should be some institutional response).
Yeah I think I remember Stuart talking about agents that request clarification whenever they are uncertain about how a concept generalizes. That is vaguely similar. I can’t remember whether he proposed any way to make that reflectively stable though.
From the perspective of this post, wouldn’t natural language work a bit as a redundancy specifier in that case and so LLMs are more alignable than RL agents?
LLMs in their current form don’t really cause Edge Instantiation problems. Plausibly this is because they internally implement many kinds of regularization toward “normality” (and also kinda quantilize by default). So maybe yeah, I think I agree with your statement in the sense that I think you intended it, as it refers to current technology. But it’s not clear to me that this remains true if we made something-like-an-LLM that is genuinely creative (in the sense of being capable of finding genuinely-out-of-the-box plans that achieve a particular outcome). It depends on how exactly it implements its regularization/redundency/quantilization and whether that implementation works for the particular OOD tasks we use it for.
Ultimately I don’t think LLM-ish vs RL-ish won’t be the main alignment-relevant axis. RL trained agents will also understand natural language, and contain natural-language-relevant algorithms. Better to focus on understood vs not-understood.
Yeah I agree there are similarities. I think a benefit of my approach, that I should have emphasized more, is that it’s reflectively stable (and theoretically simple and therefore easy to analyze). In your description of an AI that wants to seek clarification, it isn’t clear that it won’t self-modify (but it’s hard to tell).
There’s a general problem that people will want AGIs to find clever out-of-the-box solutions to problems, and there’s no principled distinction between “finding a clever out-of-the-box solution to a problem” and “Goodharting the problem specification”.
But there is a principled distinction. The distinction is whether the plan exploits differences between the goal specification and our actual goal. This is a structural difference, and we can detect using information about our actual goal.
So systems that systematically block the second thing are inevitably gonna systematically block the first thing, and I claim that your proposal here is no exception.
My proposal is usually an exception to this, because it takes advantage of the structural difference between the two cases. The trick is that the validation set only contains things that we actually want. If it were to contain extra constraints beyond what we actually want, then yeah that creates an alignment tax.
The Alice and Bob example isn’t a good argument against the independence axiom. The combined agent can be represented using a fact-conditional utility function. Include the event “get job offer” in the outcome space, so that the combined utility function is a function of that fact.
E.g.
Bob {A: 0, B: 0.5, C: 1}
Alice {A: 0.3, B: 0, C: 0}
Should merge to become
AliceBob {Ao: 0, Bo: 0.5, Co: 1, A¬o: 0, B¬o: 0, C¬o: 0.3}, where o=”get job offer”.
This is a far more natural way to combine agents. We can avoid the ontologically weird mixing of probabilities and preference implied by having preference () and also . Like… what does a geometrically rational agent actually care about, and why does it’s preferences change depending on its own beliefs and priors? A fact-conditional utility function is ontologically cleaner. Agents care about events in the world (potentially in different ways across branches of possibility, but it’s still fundamentally caring about events).
This removes all the appeal of geometric rationality for me. The remaining intuitive appeal comes from humans having preferences that are logarithmic in most resources, which is more simply represented as one utility function rather than as a geometric average of many.
Excited to attend, the 2023 conference was great!
Can we submit talks?
Yeah I can see how Scott’s quote can be interpreted that way. I think the people listed would usually be more careful with their words. But also, Scott isn’t necessarily claiming what you say he is. Everyone agrees that when you prompt a base model to act agentically, it can kinda do so. This can happen during RLHF. Properties of this behaviour will be absorbed from pretraining data, including moral systems. I don’t know how Scott is imagining this, but it needn’t be an inner homunculi that has consistent goals.
I think the thread below with Daniel and Evan and Ryan is good clarification of what people historically believed (which doesn’t put 0 probability on ‘inner homunculi’, but also didn’t consider it close to being the most likely way that scheming consequentialist agents could be created, which is what Alex is referring to[1]). E.g. Ajeya is clear at the beginning of this post that the training setup she’s considering isn’t the same as pretraining on a myopic prediction objective.
When he says ‘I think there’s a ton of wasted/ungrounded work around “avoiding schemers”, talking about that as though we have strong reasons to expect such entities.’
but his takes were probably a little more predictably unwelcome in this venue
I hope he doesn’t feel his takes are unwelcome here. I think they’re empirically very welcome. His posts seem to have a roughly similar level of controversy and popularity as e.g. so8res. I’m pretty sad that he largely stopped engaging with lesswrong.
There’s definitely value to being (rudely?) shaken out of lazy habits of thinking [...] and I think Alex has a knack for (at least sometimes correctly) calling out others’ confusion or equivocation.
Yeah I agree, that’s why I like to read Alex’s takes.
Really appreciate dialogues like this. This kind of engagement across worldviews should happen far more, and I’d love to do more of it myself.[1]
Some aspects were slightly disappointing:
Alex keeps putting (inaccurate) words in the mouths of people he disagrees with, without citation. E.g.
‘we still haven’t seen consistent-across-contexts agency from pretrained systems, a possibility seriously grappled with by eg The Parable of Predict-O-Matic).’
That post was describing a very different kind of AI than generative language models. In particular, it is explicitly designed to minimize long run prediction error.[2] In fact, the surrounding posts in the sequence discuss myopia and suggest myopic algorithms might be more fundamental/incentivised by default.
‘I think this is a better possible story than the “SGD selects for simplicity → inner-goal structure” but I also want to note that the reason you give above is not the same as the historical supports offered for the homunculus.’
“I think there’s a ton of wasted/ungrounded work around “avoiding schemers”, talking about that as though we have strong reasons to expect such entities. Off the top of my head: Eliezer and Nate calling this the “obvious result” of what you get from running ML processes. Ajeya writing about schemers, Evan writing about deceptive alignment (quite recently!), Habryka and Rob B saying similar-seeming things on Twitter” and “Again, I’m only critiquing the within-forward-pass version”
I think you’re saying here that all these people were predicting consistent-across-contexts inner homunculi from pretraining near-term LLMs? I think this is a pretty extreme strawman. In particular, most of their risk models (iirc) involve people explicitly training for outcome-achieving behaviour.
‘And so people wasted a lot of time, I claim, worrying about that whole “how can I specify ‘get my mother out of the building’ to the outcome pump” thing’
People spent time thinking about how to mitigate reward hacking? Yes. But that’s a very reasonable problem to work on, with strong empirical feedback loops. Can you give any examples of people wasting time trying to specify ‘get my mother out of the building’? I can’t remember any. How would that even work?
“And the usual result of LLMs (including Claude) is still to not act in an autonomous, agentic fashion. Even Claude doesn’t try to break out of its “cage” in normal usage, or to incite users to stop Anthropic from releasing Claude 4.0 in the future (and thereby decreasing the usage of current-Claude). ”
Who predicted this? You’re making up bad predictions. Eliezer in particular has been pretty clear that he doesn’t expect evidence of this form.
Alex seemed to occasionally enjoy throwing out insults sideways toward third parties.
E.g. “the LW community has largely written fanfiction alignment research”. I think communication between the various factions would go better if statements like this were written without deliberate intention to insult. It could have just been “the LW community has been largely working from bad assumptions”.
But I’m really glad this was published, I learned something about both Oliver and Alex’s models, and I’d think it was very positive even if there were more insults :)
If anyone is interested?
Quote from the post: “Predict-O-Matic will be objective. It is a machine of prediction, is it not? Its every cog and wheel is set to that task. So, the answer is simple: it will make whichever answer minimizes projected predictive error. There will be no exact ties; the statistics are always messy enough to see to that. And, if there are, it will choose alphabetically.”
Relevant quote from Evan in that post:
“Question: Yeah, so would you say that, GPT-3 is on the extreme end of world modeling. As far as what it’s learned in this training process?
What is GPT-3 actually doing? Who knows? Could it be the case for GPT-3 that as we train larger and more powerful language models, doing pre-training will eventually result in a deceptively aligned model? I think that’s possible. For specifically GPT-3 right now, I would argue that it looks like it’s just doing world modeling. It doesn’t seem like it has the situational awareness necessary to be deceptive. And, if I had to bet, I would guess that future language model pre-training will also look like that and won’t be deceptive. But that’s just a guess, and not a super confident one.
The biggest reason to think that pre-trained language models won’t be deceptive is just that their objective is extremely simple—just predict the world. That means that there’s less of a tricky path where stochastic gradient descent (SGD) has to spend a bunch of resources making their proxies just right, since it might just be able to very easily give it the very simple proxy of prediction. But that’s not fully clear—prediction can still be quite complex.
Also, this all potentially changes if you start doing fine-tuning, like RLHF (reinforcement learning from human feedback). Then what you’re trying to get it to do might be quite complex—something like “maximize human approval.” If it has to learn a goal like that, learning the right proxies becomes a lot harder.”
Tsvi has many underrated posts. This one was rated correctly.
I didn’t previously have a crisp conceptual handle for the category that Tsvi calls Playful Thinking. Initially it seemed a slightly unnatural category. Now it’s such a natural category that perhaps it should be called “Thinking”, and other kinds should be the ones with a modifier (e.g. maybe Directed Thinking?).
Tsvi gives many theoretical justifications for engaging in Playful Thinking. I want to talk about one because it was only briefly mentioned in the post:
Your sense of fun decorrelates you from brain worms / egregores / systems of deference, avoiding the dangers of those.
For me, engaging in intellectual play is an antidote to political mindkilledness. It’s not perfect. It doesn’t work for very long. But it does help.
When I switch from intellectual play to a politically charged topic, there’s a brief period where I’m just.. better at thinking about it. Perhaps it increases open-mindedness. But that’s not it. It’s more like increased ability to run down object-level thoughts without higher-level interference. A very valuable state of mind.
But this isn’t why I play. I play because it’s fun. And because it’s natural? It’s in our nature.
It’s easy to throw this away under pressure, and I’ve sometimes done so. This post is a good reminder of why I shouldn’t.
This post deserves to be remembered as a LessWrong classic.
It directly tries to solve a difficult and important cluster of problems (whether it succeeds is yet to be seen).
It uses a new diagrammatic method of manipulating sets of independence relations.
It’s a technical result! These feel like they’re getting rarer on LessWrong and should be encouraged.
There are several problems that are fundamentally about attaching very different world models together and transferring information from one to the other.
Ontology identification involves taking a goal defined in an old ontology[1] and accurately translating it into a new ontology.
High-level models and low-level models need to interact in a bounded agent. I.e. learning a high-level fact should influence your knowledge about low-level facts and vice versa.
Value identification is the problem of translating values from a human to an AI. This is much like ontology identification, with the added difficulty that we don’t get as much detailed access or control over the human world model.
Interpretability is about finding recognisable concepts and algorithms in trained neural networks.
In general, we can solve these problems using shared variables and shared sub-structures that are present in both models.
We can stitch together very different world models along shared variables. E.g. if you have two models of molecular dynamics, one faster and simpler than the other. You want to simulate in the fast one, then switch to the slow one when particular interactions happen. To transfer the state from one to the other you identify variables present in both models (probably atom locations, velocities, some others), then just copy these values to the other model. Under-specified variables must be inferred from priors.
If you want to transfer a new concept from WM1 to a less knowledgeable WM2, you can do so by identifying the lower-level concepts that both WMs share, then constructing an “explanation” out of those concepts. An “explanation” would look like a WM fragment purely built out of variables and structures already in WM2.
An explanation is also a pointer. If you want to point at a very specific concept in someone else’s WM, one way to do so is to explain that concept (in terms of lower level ideas that you are confident are shared).
Natural latents are a step toward solving all of these problems, via describing a subset of variables/structures that we should expect to find across all WMs (and more importantly, some of the conditions required for them to be present).
A natural latent should be extremely useful for any WM that contains variables which share redundant information. I think we can expect this to be common when highly redundant observations are compressed.
For example: If the ~same observation happens more than once, then any learner that generalizes well is going to notice this. It must somehow store the duplicated information (as well as each of the places where it is duplicated). That shared information is a natural latent. The result in this post suggests that this summary information should be isomorphic between agents, under the right conditions.[2]
As far as I know, it’s an open question which properties of environments&agents imply lots of natural latents.
The current state of this work has some limitations:
Both learners need access to the same or very similar low level observables X.
Both learners must have learned the same beliefs, otherwise their latents may be very different (although this is kinda fixed with an additional constraint on the latent).
John and David seem to have run into difficulties building useful applications of this theory.
John’s posts aren’t clear on how to identify and separate out the X variables from a general stream of data (although this seems fine for now).
With lots of compute, approximate models can be dropped in exchange for detailed models. One might drop the concept of “tree” in exchange for a complete categorization of types of trees.
On the one hand, this is still tracking the same latent information. The theorems still work. But on the other hand, it isn’t necessarily storing it in an easy-to-access way. This is fine for communication, but less fine for interpretability or manual joining of WMs.
Perhaps there is some assumption we can make that guarantees all levels of abstraction will remain stored. Or perhaps we should expect interpretability of a WM to often involve some inferential work on the part of that WM.
If this line of research goes well, I hope that we will have theorems that say something like:
“For some bounded agent design, given observations of some complicated well-understood data-generating structure Z, and enough compute / attentional resources, the agent will learn a model of Z which contains parts x,y,w (each with predicable levels of approximate isomorphism to parts of the real Z). Upon observing more data, we can expect some parts (x,y) of this structure to remain unchanged.”
Think of an ontology the choice of variables in a particular Bayes net, for our current purposes.
I’m leaning on the algorithmic definition of natural latents here.
I’m curious whether the recent trend toward bi-level optimization via chain-of-thought was any update for you? I would have thought this would have updated people (partially?) back toward actually-evolution-was-a-decent-analogy.
There’s this paragraph, which seems right-ish to me:
In order to experience a sharp left turn that arose due to the same mechanistic reasons as the sharp left turn of human evolution, an AI developer would have to:
Deliberately create a (very obvious[2]) inner optimizer, whose inner loss function includes no mention of human values / objectives.[3]
Grant that inner optimizer ~billions of times greater optimization power than the outer optimizer.[4]
Let the inner optimizer run freely without any supervision, limits or interventions from the outer optimizer.[5]
Extremely long chains-of-thought on hard problems is pretty much meeting these conditions, right?
I think we currently do not have good gears level models of lots of the important questions of AI/cognition/alignment, and I think the way to get there is by treating it as a software/physicalist/engineering problem, not presupposing an already higher level agentic/psychological/functionalist framing.
Here’s two ways that a high-level model can be wrong:
It isn’t detailed enough, but once you learn the detail it adds up to basically the same picture. E.g. Newtonian physics, ideal gas laws. When you get a more detailed model, you learn more about which edge-cases will break it. But the model basically still works, and is valuable for working out the more detailed model.
It’s built out of confused concepts. E.g. free will, consciousness (probably), many ways of thinking about personal identity, four humors model. We’re basically better off without this kind of model and should start from scratch.
It sounds like you’re saying high-level agency-as-outcome-directed is wrong in the second way? If so, I disagree, it looks much more like the first way. I don’t think I understand your beliefs well enough to argue about this, maybe there’s something I should read?
I have a discomfort that I want to try to gesture at:
Are you ultimately wanting to build a piece of software that solves a problem so difficult that it needs to modify itself? My impression from the post is that you are thinking about this level of capability in a distant way, and mostly focusing on much earlier and easier regimes. I think it’s probably very easy to work on legible low-level capabilities without making any progress on the regime that matters.
To me it looks important for researchers to have this ultimate goal constantly in their mind, because there are many pathways off-track. Does it look different to you?
Ultimately, this is a governance problem, not a technical problem. The choice to choose illegible capabilities is a political one.
I think this is a bad place to rely on governance, given the fuzziness of this boundary and the huge incentive toward capability over legibility. Am I right in thinking that you’re making a large-ish gamble here on the way the tech tree shakes out (such that it’s easy to see a legible-illegible boundary, and the legible approaches are competitive-ish) and also the way governance shakes out (such that governments decide that e.g. assigning detailed blame for failures is extremely important and worth delaying capabilities)?
I’m glad you’re doing ambitious things, and I’m generally a fan of trying to understand problems from scratch in the hope that they dissolve or become easier to solve.
Literally compute and man-power. I can’t afford the kind of cluster needed to even begin a pretraining research agenda, or to hire a new research team to work on this. I am less bottlenecked on the theoretical side atm, because I need to run into a lot of bottlenecks from actual grounded experiments first.
Why would this be a project that requires large scale experiments? Looks like something that a random PhD student with two GPUs could maybe make progress on. Might be a good problem to make a prize for even?
because stabler optimization tends to be more powerful / influential / able-to-skillfully-and-forcefully-steer-the-future
I personally doubt that this is true, which is maybe the crux here.
Would you like to do a dialogue about this? To me it seems clearly true in exactly the same way that having more time to pursue a goal makes it more likely you will achieve that goal.
It’s possible another crux is related to the danger of Goodharting, which I think you are exaggerating the danger of. When an agent actually understand what it wants, and/or understands the limits of its understanding, then Goodhart is easy to mitigate, and it should try hard to achieve its goals (i.e. optimize a metric).
Hmm good point. Looking at your dialogues has changed my mind, they have higher karma than the ones I was looking at.
You might also be unusual on some axis that makes arguments easier. It takes me a lot of time to go over peoples words and work out what beliefs are consistent with them. And the inverse, translating model to words, also takes a while.