what if I want to train a new model and run inference on it?

The API can also have built-in functions for training.

What if I want to experiment with a new scaffold?

Scaffolds can normally be built around APIs? I thought scaffolds was just all about what prompts you send to the model and what you do with the model outputs.

I do agree that this might be rough for some types of research. I imagine the arguments are pretty similar here as the arguments about how much research can be done without access to dangerous model weights.

This post is a founding pillar of my current understanding of Buddhism, insight meditation and awakening. I blieve this post (and, by extension, the whole sequence) creates a material reductive framework that solves—at least in broad strokes—a problem so important that it has founded at least one major world religion, the mechanics of which have been a mystery for at least two millennia. This post has been instrumental in improving my understanding of my own experiences with insight cycles.

Will this post be relevant 12 months from now? If this post is correct and human beings are alive in 1,000 years, this post will still be relevant just due to the Lindy effect. This post has been relevant for at least 2,000 years. We can expect it to be relevant for at least another 2,000.

Was this post invalidated by further work or other criticisms that came up? Not yet, and perhaps not ever.

Data point: I’m millennial (born 1992) and have a pretty strong aversion to phone calls, which is motivated mainly by the fact that I prefer most communication to be non-real-time so that I can take time to think about what to say without creating an awkward silence. And when I do engage in real-time communication, visual cues make it much less unpleasant, so phone calls are particularly bad in that I have to respond to someone in real time without either of us seeing the other’s face/​body language.

If I had to take a wild guess at why this seems to be generational, I’d suggest that older generations spent much of their lives with phone calls being the only way to quickly contact people far away, and prefer them due to familiarity. Perhaps if they’d grown up with email/​texting being options, they’d be more likely to prefer them instead.

This post was a useful source of intuition when I was reading about singular learning theory the other week (in order to pitch it to an algebraic geometer of my acquaintance along with gifting her a copy of If Anyone Builds It), but I feel like it “buries the lede” for why SLT is cool. (I’m way more excited about “this generalizes minimum description length to neural networks!” than “we could do developmental interpretability maybe.” De gustibus?)

That is, “flatness” in the loss landscape is about how many nearby-in-parameterspace models achieve similar loss, and you can get that by error-correction, not just by using fewer parameters (such that it takes fewer bits of evidence to find that setting)? Cool!

It seems that using SLT one could give a generally correct treatment of MDL. However, until such results are established

It looks like the author contributed to achieving this in October 2025′s “Compressibility Measures Complexity: Minimum Description Length Meets Singular Learning Theory”?

Maybe the appropriate mathematical object to represent trust might be related to those used to represent uncertainty in complex systems, such as wave functions associated with probabilities. After all, you can trust someone to precisely the extent to which you can constrain your own uncertainty about whether they will do things you wouldn’t want. These things , while a kind of scalar, certainly contain lots of information in the form of their distribution throughout space, as well as being complex numbers.

This post highlighted ways in which OAI’s CBRN evals were uninformative about how close o1-preview was to action-relevant thresholds. I think it’s valuable for increasing public knowledge about model capabilities/​risk profiles and keeping developers accountable.

I particularly appreciate the post breaking down the logic of rule-out evals (“is this test clearly easier than [real world threat model]” and “does the model clearly fail this test”). This frame still seems useful for assessing system cards in 2025.

I’m hopeful the details can be fleshed out in late crunch-time.

You don’t want to be figuring out ethics in crunch time, you want to be figuring out how to pass off (or buy time to pass off) the question to something-like-CEV or a long reflection.

(Self-review.) I was having fun with a rhetorical device in this one, which didn’t land for all readers; I guess that’s how it goes sometimes.

To try to explain what I was trying to do here in plainer words: I feel like a lot of people who read this website but don’t read textbooks walk away with an intuitive picture of deep learning as being like evolving an animal to do your bidding, which is scary because evolution is not controllable.

That was strikingly not the intuitive picture I got from reading standard academic tutorial material on the topic in late 2023 and early 2024. As a lifetime Less Wrong reader reading Simon Prince’s Understanding Deep Learning (2023), one gets the sense that Prince isn’t really thinking about “AI” as people on this website understand it, even if the term gets used.

It’s a computational statistics book. Prince is writing about a class of techniques for fitting statistical models to data. The fact that statistics happens to have some impressive applications doesn’t make it about summoning a little animal.

The difference in views seemed worth writing about. I was inspired by some Tweets by Charles Foster:

At some point I switched from seeing neural networks as arcane devices to seeing them as moldable variants of “boring” building blocks from signal processing, feedback control, associative learning, & functional programming. Like some kind of function approximation plastic/​epoxy

Tbh “neural network” is maybe too suggestive of a term. Like it anchors people on vague intuitions about emergence/​agency rather than on mechanistic thinking. Call them “nonlinear coupling networks” or “plastic basis functions” or go back to “parallel distributed processors”

Sorry, I worry that I’m still not succeeding at conveying the intuition: probably some people reading this review comment are shaking their head, disappointed that I seem to be trotting out the AI skeptic’s ignorant “It’s just math; math can’t hurt you” canard. So to be clear (and I think I was clear enough about this in the post; see, e.g., the final paragraph), I absolutely agree that math can kill you, obviously. I’m just saying that after I read the math, the summoning-a-little-animal mental image didn’t seem faithful to the math: you should be thinking about how the model’s outputs interpolate the training data, not how the little animal’s behavior unpredictably fails to reflect some putative utility function that “outer” training failed to instill.

… I’m still not communicating the thing, am I? You know what? Forget it. Don’t read this review and don’t read this post. Read Prince 2023 or Bishop and Bishop 2024. Read textbooks!

As of 2025, I still find this post (and comments) helpful for understanding different things people mean by “CoT unfaithfulness” and why these things still largely aren’t reasons to give up on CoT monitoring for current models.

Thanks for the response! That makes a lot of sense.

My intuition is that making the model less deep would make it non-linearly easier to interpret, since the last layers of the model is the most interpretable (since it’s closest to human-understable tokens and embeddings), while the earlier layers are interpretable (since they’re farther away). Also if layers end up doing duplicate work that could make interpreting the deduplicated version easier. I’m not sure though.

For “thinking dot by dot”, I think serial computation can only benefit from roughly as many filler tokens as you have layers, since each layer can only attend to previous layers. For example, if you had a two layer model, position i does some calculations in layer 1, position i+1 can attend to the layer 1 results from position i in layer 2, but at position i+2 there’s no layer 3 to attend to the layer 2 outputs from position i+1.

Note how the only difference between position i+1 and i+2 is an additional ”.” input and the positional encoding.

The dot-by-dot paper uses a highly parallel algorithm, so I think it’s more like adding dynamic attention heads /​ increasing the width of the network, while CoT is more like increasing the number of layers.

If I understand the architecture right (I might not), standard models are already effectively doing neuralese CoT, but bounded by the depth of the network (and also with the caveat that the model has fewer layers remaining to use the results after each step).

I think the title question remains important and neglected. I encourage people to revisit this question as we get closer to AIs that (are able to) exhibit scary/​egregiously misaligned behavior.

Specifically, it might be good to have answers to questions like:

• What do we expect to be the best kind of evidence for convincing developers to shutdown/​undeploy?

  • Maybe there is more compelling evidence than “your AI writing code to escape”

  • I’m reminded of hearing about how the public reception of the OAI/​Apollo scheming paper was surprising relative to some people’s expectations (people are more freaked out by weird CoT, less freaked out by actual scheme-y behavior).

• What are good plans for AI labs when they have such evidence?

• What are good plans for AIS researchers/​advocacy people when they have such evidence?

• Should you pursue safety research agenda X if shutdown/​undeploy is hard?

  • To spell out the thinking here: you might conceptualize safety research as having broadly two paths to impact:

  • 1- “Training against”: give training signal we can optimize over to reduce egregious misalignment

  • 2- “Shutdown”: give evidence to convince AI developers to slow down or un-deploy

  • One implication of thinking that 2 is very hard is that your research should either focus aggressively on 1 or risk being not that useful.
    
    

I hope I will return to this when I have time to read it properly and think about it properly, but for now I’ll just drop in two things at the meta-level: (1) I don’t know how comprehensible I’d have found something more in your usual concise style, but the above certainly seems nice and clear so it seems like you probably made a good choice. (2) I’m glad to hear that I’m perfectly tactful but now I’m worried about a different issue, namely that maybe I never say anything unless I have something mean^H^H^H^Hcritical to say, which I’m aware is the exact opposite of what generations of parents have been teaching their children to do :-). (I definitely do lean in that direction, and I’m somewhat prepared to defend it in that offering hopefully-informative criticism is arguably more useful than offering compliments, but it’s still probably suboptimal.)

“we don’t need to worry about Goodhart and misgeneralization of human values at extreme levels of optimization.”

That isn’t a conclusion I draw, though. I think you don’t know how to parse what I’m saying as different from that rather extreme conclusion—which I don’t at all agree with? I feel concerned by that. I think you haven’t been tracking my beliefs accurately if you think I’d come to this conclusion.

FWIW I agree with you on a), I don’t know what you mean by b), and I agree with c) partially—meaningfully different, sure.

Anyways, when I talk about “imperfect” values, I’m talking about a specific concept which I probably should have clarified. The model you still need to download to get my view is that Alignment Allows “Non-Robust” Decision-Influences and Doesn’t Require Robust Grading (the culmination of two previous essays).

Suppose that the humans do have diminishing returns of utility functions. Unfortunately, existing combination of instincts and moral intuitions do not prompt the majority of humans to help the poor, especially those who are far from potential helpers’ set of friends^[1], with nearly anything. And those who do so are unlikely to stay in power or were unlikely to receive fortunes or occupy relevant positions.

1. ^

   Friends are also likely to be in the same class as the potential helpers.

did you run it? what did you found out?

I think this post does a good job of arguing that preventing collusion is an important open problem for control /​ untrusted monitoring in particular and suggesting strategies for this.

I weakly wish this post were broken down into smaller chunks which separately explain “a basic design for untrusted monitoring” and collusion-specific analysis.

It’s perhaps important to note that this post conservatively assumes that an untrusted monitor will consistently try to collude with the policy when ”...the untrusted model you’re using as a monitor is very similar to the model you’re using as the policy, e.g. they might be different fine-tunes of models that have already been pre- and post- trained. In principle, you could use different models”.

• It’d be nice to conceptually and empirically investigate the extent to which using relevantly similar models lead to collusion

• Using different/​more decorrelated models, or “mixed deployments”, could be a useful and practical way to prevent collusion

This post might be worth revisiting for experiment ideas when models seem capable enough to exhibit interesting collusion behavior.

This assumes a bit too much rigor in common language. When most people wonder “what’s guaranteed in life”, they really mean “what kinds of events and causal links are so common in my context that I should give high probability to their occurence”.

The common answer “death and taxes” shows this well. One of them is far more likely than the other.

I think the high-level point in this post is correct and important, though not super action-guiding.

I find it interesting to notice in 2025 that this post failed to mention the “obvious” social failure mode of extreme power concentration/​power grabs/​coups. Seems like a missed opportunity for this particular reflection.

This post discusses an important point: it is impossible to be simultaneously perfectly priorist (“updateless”) and learn. Learning requires eventually “passing to” something like a posterior, which is inconsistent with forever maintaining “entanglement” with a counterfactual world. This is somewhat similar to the problem of traps (irreversible transitions): being prudent about risking traps requires relying on your prior, which prevents you from learning every conceivable opportunity.

My own position on this cluster of questions is that you should be priorist/​(infra-)Bayesian about physics but postist/​learner/​frequentist about logic. This idea is formally embodied in the no-regret criterion for Formal Computational Realism. I believe that this no-regret condition implies something like the OP’s “Eventual Learning”, but formally demonstrating it is future work.