Buck

Karma: 14,953

CEO at Redwood Research.

AI safety is a highly collaborative field—almost all the points I make were either explained to me by someone else, or developed in conversation with other people. I’m saying this here because it would feel repetitive to say “these ideas were developed in collaboration with various people” in all my comments, but I want to have it on the record that the ideas I present were almost entirely not developed by me in isolation.

Please contact me via email (bshlegeris@gmail.com) instead of messaging me on LessWrong.

If we are ever arguing on LessWrong and you feel like it’s kind of heated and would go better if we just talked about it verbally, please feel free to contact me and I’ll probably be willing to call to discuss briefly.

Buck 29 Sep 2025 1:27 UTC
6 points
0
in reply to: Chris Lakin’s comment on: Reasons to sell frontier lab equity to donate now rather than later
For some donation opportunities, e.g. political donations, that would be a crime.

Buck 28 Sep 2025 22:46 UTC
24 points
2
in reply to: Eliezer Yudkowsky’s comment on: A non-review of “If Anyone Builds It, Everyone Dies”
Where do you think you’ve spelled this argument out best? I’m aware of a lot of places where you’ve made the argument in passing, but I don’t know of anywhere where you say it in depth.
(My response last time (which also wasn’t really in depth; I should maybe try to articulate my position better sometime...) was this.)

Buck 28 Sep 2025 22:06 UTC
6 points
6
in reply to: evhub’s comment on: Tomás B.’s Shortform
The way I’d think about this is:
Currently, intellectual labor from machine learning researchers costs a lot of compute. A $1M/year ML researcher costs the same as having 30 or so H100s. At the point where you have AGI, you can probably run the equivalent of one ML researcher with substantially less hardware than that. (I’m amortizing, presumably you’ll be running your models on multiple chips doing inference on multiple requests simultaneously.) This means that some ways to convert intellectual labor into compute efficiency will be cost-effective when they weren’t previously. So I expect that ML will become substantially more labor-intensive and have much more finicky special casing.

Buck 28 Sep 2025 19:07 UTC
4 points
0
on: Prompt optimization can enable AI control research
Thanks for doing this. This result is helpful. We at Redwood will definitely try out these methods.

Buck 28 Sep 2025 14:01 UTC
6 points
0
in reply to: Rob Bensinger’s comment on: A Reply to MacAskill on “If Anyone Builds It, Everyone Dies”
You don’t feel like “I think the risk of misaligned AI takeover is enormously important.” suffices?

Buck 28 Sep 2025 2:05 UTC
11 points
3
in reply to: Rob Bensinger’s comment on: A Reply to MacAskill on “If Anyone Builds It, Everyone Dies”
I think you’ve substantially misunderstood what Will is talking about. He’s not making a recommendation that people rush through things. He’s noting what he believes (and I mostly agree) to be huge weaknesses in the book’s argument.
Similarly, he’s not saying labs have alignment in the bag. He’s just noting holes in the book’s arguments that extreme catastrophic misalignment is overwhelmingly likely.
All of this together makes me extremely confused if his real view is basically just “I agree with most of MIRI’s policy proposals but I think we shouldn’t rush to enact a halt or slowdown tomorrow”.
I assume that he disagrees with MIRI’s headline policy proposal of banning AI research, in the senses that he thinks it’s a poor choice of policy recommendation given tractability and the concern that this proposal might cause bad things to happen (like uneven bans on AI research). I don’t know what he thinks of whether it would be good to magically institute the MIRI policy proposal; I think it’s fundamentally unclear what hypothetical you’re even supposed to consider in order to answer that question.
I summarized my view on MIRI’s policy suggestions as “poor”, but I definitely think it will be extremely valuable to have the option to slow down AI development in the future.

Buck 28 Sep 2025 0:09 UTC
18 points
12
on: A Reply to MacAskill on “If Anyone Builds It, Everyone Dies”
I think your summary of Will’s argument is extremely misleading and unfair, for similar reasons to Claude.
Will thinks government compute monitoring is a bad idea
Why do you think Will thinks this? I think that government compute monitoring is an example of the kind of proposal that Will is worried that the book distracts from.

Buck 27 Sep 2025 22:24 UTC
23 points
4
in reply to: Eric Neyman’s comment on: Reasons to sell frontier lab equity to donate now rather than later
I think that the consideration you raise is important, but here’s something that came to mind while reading your comment:
An example: last year I put a decent fraction of my wealth in a DAF. At the time, I hadn’t heard any warnings not to do that. Today, I think that it would have been better if I had not put that money in the DAF, because I think the best donation opportunities are not 501(c)(3)s.
There’s an interpretation of this experience which has the opposite conclusion than the one you draw. Specifically, you didn’t try to make a donation, you tried to punt the choice down the road. If you had been more focused on maximizing impact using donations last year, that might have forced you to learn more about the situation and you might have noticed that political donations were a good opportunity.

Buck 23 Sep 2025 18:27 UTC
4 points
0
in reply to: Rohin Shah’s comment on: The title is reasonable
Amplified oversight can make substantial progress without studying actual schemers
(Though note that it’s unclear whether this progress will mitigate scheming risk.)

Buck 23 Sep 2025 18:11 UTC
39 points
24
on: We Support “If Anyone Builds It, Everyone Dies”
- I think the book’s thesis is basically right — that if anyone builds superintelligent AI in the next decade or two, it’ll have a terrifyingly high (15%+) chance of causing everyone to die in short order
I think this is an absurd statement of the book’s thesis: the book is plainly saying something much stronger than that. How did you pick 15% rather than for example 80% or 95%?
I am basically on the fence about the statement you have made. For reasons described here, I think P(human extinction|AI takeover) is like 30%, and I separately think P(AI takeover) is like 45%.
- I think the world where the book becomes an extremely popular bestseller is much better on expectation than the world where it doesn’t
I think this is probably right, but it’s unclear.
- I generally respect MIRI’s work and consider it underreported and underrated
It depends on what you mean by “work”. I think their work making AI risk arguments at the level of detail represented by the book is massively underreported and underrated (and Eliezer’s work on making a rationalist/AI safety community is very underrated). I think that their technical work is overrated by LessWrong readers.

Buck 23 Sep 2025 14:52 UTC
2 points
0
in reply to: Alexei’s comment on: Buck’s Shortform
I do think that the post had a worse argumentative structure than posts I normally write (hence all the confusions in the comments). But that was totally on me, not the AI. I’m interested in whether you think your problem was with the argumentative structure or the prose.

Buck 23 Sep 2025 14:49 UTC
10 points
0
in reply to: Hastings’s comment on: Buck’s Shortform
I didn’t intend this to be a prank! I just wanted to write faster and better.

Buck 23 Sep 2025 4:33 UTC
7 points
0
in reply to: jamjam’s comment on: Buck’s Shortform
I agree that I had had all the ideas, but I hadn’t previously been able to get AIs to even do the “grunt work” of turning it into prose with anything like that level of quality!

Buck 23 Sep 2025 2:28 UTC
30 points
0
on: Buck’s Shortform
Fun fact: My post Christian homeschoolers in the year 3000 was substantially written by Claude Opus 4.1.
You can see the chat here. I prompted Claude with a detailed outline, a previous draft that followed a very different structure, and a copy of “The case for ensuring powerful AIs are controlled” for reference about my writing style. The outline I gave Claude is in the Outline tab, and the old draft I provided is in the Old draft tab, of this doc.
As you can see, I did a bunch of back and forth with Claude to edit it. Then I copied to a Google doc and edited substantially on my own to get to the final product.
I was shocked by how good Claude was at this.
I believe this is in compliance with the LLM writing assistance policy.

Buck 22 Sep 2025 17:18 UTC
38 points
29
in reply to: Steven Byrnes’s comment on: This is a review of the reviews
Yeah, I agree. The audience for this book isn’t LessWrong, but lots of people seem to be acting as if pushing back on LessWrong is a defection that will hurt the book’s prospects.

Buck 21 Sep 2025 19:44 UTC
108 points
43
on: Buck’s Shortform
Zach Robinson, relevant because he’s on the Anthropic LTBT and for other reasons, tweets:
“If Anyone Builds It, Everyone Dies” by @ESYudkowsky and @So8res is getting a lot of attention this week. As someone who leads an org working to reduce existential risks, I’m grateful they’re pushing AI safety mainstream. But I think they’re wrong about doom being inevitable. 🧵
Don’t get me wrong—I take AI existential risk seriously. But presenting doom as a foregone conclusion isn’t helpful for solving the problem.
In 2022, superforecasters and AI researchers estimated the probability of existential catastrophic risk from AI by 2100 at around 0.4%-3%. A recent study found no correlation between near-term accuracy and long-term forecasts. TL;DR: predicting the future is really hard.
That doesn’t mean we should throw the existential risk baby out with the “Everyone Dies” bathwater. Most of us wouldn’t be willing to risk a 3% chance (or even a 0.3% chance!) of the people we love dying.
But accepting uncertainty matters for navigating this complex challenge thoughtfully.
Accepting uncertainty matters for two big reasons.
First, it leaves room for AI’s transformative benefits. Tech has doubled life expectancy, slashed extreme poverty, and eliminated diseases over the past two centuries. AI could accelerate these trends dramatically.
But Yudkowsky and Soares dismiss these possibilities as “beautiful dreams.” If we’re certain AI will kill us all, then all potential benefits get rounded to zero.
Second, focusing exclusively on extinction scenarios blinds us to other serious AI risks: authoritarian power grabs, democratic disruption through misinformation, mass surveillance, economic displacement, new forms of inequity. These deserve attention too.
People inspired by @EffectvAltruism have made real progress on AI safety while also mapping varied futures. I’m personally encouraged by recent work from @willmacaskill and others at @forethought_org on putting society on a path toward flourishing with transformative AI.
This includes questions about AI consciousness and welfare—how should we treat AI systems that might themselves suffer or flourish? These questions sound abstract but may become practical as AI systems become more sophisticated.
The policy debate surrounding AI development has become a false choice: stop all AI development (to prevent doom) vs. speed it up (for rewards).
I grew up participating in debate, so I know the importance of confidence. But there are two types: epistemic (based on evidence) and social (based on delivery). Despite being expressed in self-assured language, the evidence for imminent existential risk is far from airtight.
My take is this: We should take AI risk seriously—with all its uncertainties—and work hard to bend development toward better outcomes.
On the object level, I think Zach is massively underrating AI takeover risk, and I think that his reference to the benefits of AI misses the point.
On the meta level, I think Zach’s opinions are relevant (and IMO concerning) for people who are relying on Zach to ensure that Anthropic makes good choices about AI risks. I don’t think the perspective articulated in these tweets is consistent with him doing a good job there (though maybe this was just poor phrasing on his part, and his opinions are more reasonable than this).
What links here?
- More Reactions to If Anyone Builds It, Everyone Dies by Zvi (23 Sep 2025 16:00 UTC; 33 points)

Buck 21 Sep 2025 19:34 UTC
22 points
4
in reply to: Rohin Shah’s comment on: The title is reasonable
I engage on LessWrong because:
- It does actually help me sharpen my intuitions and arguments. When I’m trying to understand a complicated topic, I find it really helpful to spend a bunch of time talking about it with people. It’s a cheap and easy way of getting some spaced repetition.
- I think that despite the pretty bad epistemic problems on LessWrong, it’s still the best place to talk about these issues, and so I feel invested in improving discussion of them. (I’m less pessimistic than Rohin.)
  - There are a bunch of extremely unreasonable MIRI partisans on LessWrong (as well as some other unreasonable groups), but I think that’s a minority of people who I engage with; a lot of them just vote and don’t comment.
  - I think that my and Redwood’s engagement on LessWrong has had meaningful effects on how thoughtful LWers think about AI risk.
- I feel really triggered by people here being wrong about stuff, so I spend somewhat more time on it than I endorse.
  - This is partially because I identify strongly with the rationalist community and it hurts me to see the rationalists being unreasonable or wrong.
I do think that on the margin, I wish I felt more intuitively relatively motivated to work on my writing projects that are aimed at other audiences. For example, this weekend I’ve been arguing on LessWrong substantially as procrastination for writing a piece about AI control aimed at computer security experts, particularly those at AI companies. I think that post will be really valuable because I think that a lot of people in that audience are pretty persuadable, and I think it’s a really important point. But it’s less motivating because the feedback loops are longer and the audience doesn’t include many of my friends and my broader social community.

Buck 21 Sep 2025 18:23 UTC
11 points
15
in reply to: Jeremy Gillen’s comment on: Jeremy Gillen’s Shortform
I feel like your title for this short-form post is unreasonably aggressive, given what you’re saying here.
I found your articulation of the structure of the book’s argument helpful and clarifying.
I’m planning to write something more about this at some point: I think a key issue here is that we aren’t making the kind of arguments where “local validity” is a reliable concept. No-one is trying to make proofs, they’re trying to make defeasible heuristic arguments. Suppose the book makes an argument of the form “Because of argument A, I believe conclusion X. You might have thought that B is a counterargument to A. But actually, because of argument C, B doesn’t work.” If Will thinks that argument C doesn’t work, I think it’s fine for him to summarize this as: “they make an argument mostly around A, and which I don’t think suffices to establish X”.

Buck 21 Sep 2025 16:31 UTC
2 points
2
in reply to: Jeremy Gillen’s comment on: Jeremy Gillen’s Shortform
Whatever. I don’t think that’s a very important difference, and I don’t think it’s fair to call Will’s argument a straw man based on it. I think a very small proportion of readers would confidently interpret the book’s argument the way you did.
You’re claiming that the book’s argument is only trying to apply to an extremely narrow definition of AI training. You describe it as SGD on a training set; I assume you intend to refer to things like RL on diverse environments, like how R1 was trained. If that’s what the argument is about, it’s really important for the authors to explain how that argument connects to the broader notion of training used in practice, and I don’t remember this happening. I don’t remember them talking carefully about the still broader question of “what happens when you do get to examine results and intermediate results and make adjustments based on observations?”

Buck 21 Sep 2025 16:16 UTC
30 points
18
in reply to: Buck’s comment on: Jeremy Gillen’s Shortform
I don’t think that the MIRI book would hold up if you analyzed it with this level of persnicketiness–they were absolutely not precise at the level of distinguishing between the whole development process and single training runs. (Which is arguably fine–they were trying to write a popular book, not trying to persuade super high-context readers of anything!) So this complaint strikes me as somewhat of an isolated demand for rigor.