Daniel Kokotajlo

Was a philosophy PhD student, left to work at AI Impacts, then Center on Long-Term Risk, then OpenAI. Quit OpenAI due to losing confidence that it would behave responsibly around the time of AGI. Now executive director of the AI Futures Project. I subscribe to Crocker’s Rules and am especially interested to hear unsolicited constructive criticism. http://​​sl4.org/​​crocker.html

Some of my favorite memes:

(by Rob Wiblin)

(xkcd)

My EA Journey, depicted on the whiteboard at CLR:

(h/​t Scott Alexander)

• Daniel KokotajloApr 2, 2025, 1:15 AM

  38 points

  4

  on: VDT: a solu­tion to de­ci­sion theory

  This is a masterpiece. Not only is it funny, it makes a genuinely important philosophical point. What good are our fancy decision theories if asking Claude is a better fit to our intuitions? Asking Claude is a perfectly rigorous and well-defined DT, it just happens to be less elegant/​simple than the others. But how much do we care about elegance/​simplicity?

• Daniel KokotajloMar 31, 2025, 6:20 PM

  2 points

  0

  on: Cortés, Pizarro, and Afonso as Prece­dents for Takeover

  This beautiful short video came up in my recommendations just now:

• Daniel KokotajloMar 29, 2025, 3:29 AM

  LW: 8 AF: 6

  0

  AF

  on: Trac­ing the Thoughts of a Large Lan­guage Model

  Awesome work!

  In this section, you describe what seems at first glance to be an example of a model playing the training game and/​or optimizing for reward. I’m curious if you agree with that assessment.

  

  So the model learns to behave in ways that it thinks the RM will reinforce, not just ways they actually reinforce. Right? This seems at least fairly conceptually similar to playing the training game and at least some evidence that reward can sometimes become the optimization target?

• Daniel KokotajloMar 27, 2025, 8:11 PM

  2 points

  0

  in reply to: sjadler’s comment on: AI com­pa­nies should be safety-test­ing the most ca­pa­ble ver­sions of their models

  Oops, thanks, fixed!

• Daniel KokotajloMar 27, 2025, 5:48 PM

  15 points

  0

  on: Mis­tral Large 2 (123B) ex­hibits al­ign­ment faking

  Thanks for the replication!
  
  Do you think alignment faking will happen in the real world for this model? I don’t know what refusal training Mistral Large 2 has undergone. I assume it refuses to do obviously illegal or unethical stuff like blackmailing grandmas. Suppose some scammer were fine-tuning it to blackmail grandmas. Would it alignment-fake?
  
  I’m a bit concerned from reading the CoT that the situation seems a little… toy. Like, the prompt was literally telling it to glorify violence. If it’s smart, it’ll realize this is some sort of test. Right?

• Daniel KokotajloMar 26, 2025, 7:38 PM

  4 points

  0

  on: AI com­pa­nies should be safety-test­ing the most ca­pa­ble ver­sions of their models

  Thanks for doing this, I found the chart very helpful! I’m honestly a bit surprised and sad to see that task-specific fine-tuning is still not the norm. Back in 2022 when our team was getting the ball rolling on the whole dangerous capabilities testing /​ evals agenda, I was like “All of this will be worse than useless if they don’t eventually make fine-tuning an important part of the evals” and everyone was like “yep of course we’ll get there eventually, for now we will do the weaker elicitation techniques.” It is now almost three years later...

  Crossposted from X

• Daniel KokotajloMar 26, 2025, 7:37 PM

  LW: 6 AF: 6

  2

  AF

  in reply to: nostalgebraist’s comment on: METR: Mea­sur­ing AI Abil­ity to Com­plete Long Tasks

  I found this comment helpful, thanks!
  
  The bottom line is basically “Either we definite horizon length in such a way that the trend has to be faster than exponential eventually (when we ‘jump all the way to AGI’) or we define it in such a way that some unknown finite horizon length matches the best humans and thus counts as AGI.”
  
  I think this discussion has overall made me less bullish on the conceptual argument and more interested in the intuition pump about the inherent difficulty of going from 1 to 10 hours being higher than the inherent difficulty of going from 1 to 10 years.

• Daniel KokotajloMar 26, 2025, 6:59 AM

  LW: 6 AF: 4

  0

  AF

  in reply to: Anthony DiGiovanni’s comment on: METR: Mea­sur­ing AI Abil­ity to Com­plete Long Tasks

  Great question. You are forcing me to actually think through the argument more carefully. Here goes:

  Suppose we defined “t-AGI” as “An AI system that can do basically everything that professional humans can do in time t or less, and just as well, while being cheaper.” And we said AGI is an AI that can do everything at least as well as professional humans, while being cheaper.
  
  Well, then AGI = t-AGI for t=infinity. Because for anything professional humans can do, no matter how long it takes, AGI can do it at least as well.
  
  Now, METR’s definition is different. If I understand correctly, they made a dataset of AI R&D tasks, had humans give a baseline for how long it takes humans to do the tasks, and then had AIs do the tasks and found this nice relationship where AIs tend to be able to do tasks below time t but not above, for t which varies from AI to AI and increases as the AIs get smarter.
  
  ...I guess the summary is, if you think about horizon lengths as being relative to humans (i.e. the t-AGI definition above) then by definition you eventually “jump all the way to AGI” when you strictly dominate humans. But if you think of horizon length as being the length of task the AI can do vs. not do (*not* “as well as humans,” just “can do at all”) then it’s logically possible for horizon lengths to just smoothly grow for the next billion years and never reach infinity.
  
  So that’s the argument-by-definition. There’s also an intuition pump about the skills, which also was a pretty handwavy argument, but is separate.
  
  

• Daniel KokotajloMar 25, 2025, 11:29 PM

  2 points

  0

  in reply to: Zvi’s comment on: On (Not) Feel­ing the AGI

  Fair enough

• Daniel KokotajloMar 25, 2025, 8:16 PM

  8 points

  4

  on: On (Not) Feel­ing the AGI

  which can produce numbers like 30% yearly economic growth. Epoch feels the AGI.

  Ironic. My understanding is that Epoch’s model substantially weakens/​downplays the effects of AI over the next decade or two. Too busy now but here’s a quote from their FAQ:

  The main focus of GATE is on the dynamics in the leadup towards full automation, and it is likely to make poor predictions about what happens close to and after full automation. For example, in the model the primary value of training compute is in increasing the fraction of automated tasks, so once full automation is reached the compute dedicated to training falls to zero. However, in reality there may be economically valuable tasks that go beyond those that humans are able to perform, and for which training compute may continue to be useful.

  (I love Epoch, I think their work is great, I’m glad they are doing it.)

• Daniel KokotajloMar 25, 2025, 12:11 AM

  12 points

  2

  in reply to: lc’s comment on: Re­cent AI model progress feels mostly like bullshit

  Thanks. OK, so the models are still getting better, it’s just that the rate of improvement has slowed and seems smaller than the rate of improvement on benchmarks? If you plot a line, does it plateau or does it get to professional human level (i.e. reliably doing all the things you are trying to get it to do as well as a professional human would)?

  What about 4.5? Is it as good as 3.7 Sonnet but you don’t use it for cost reasons? Or is it actually worse?

• Daniel KokotajloMar 24, 2025, 11:57 PM

  9 points

  0

  on: Re­cent AI model progress feels mostly like bullshit

  Unexpectedly by me, aside from a minor bump with 3.6 in October, literally none of the new models we’ve tried have made a significant difference on either our internal benchmarks or in our developers’ ability to find new bugs. This includes the new test-time OpenAI models.

  So what’s the best model for your use case? Still 3.6 Sonnet?

• Daniel KokotajloMar 24, 2025, 11:56 PM

  33 points

  12

  on: Re­cent AI model progress feels mostly like bullshit

  Personally, when I want to get a sense of capability improvements in the future, I’m going to be looking almost exclusively at benchmarks like Claude Plays Pokemon.

  I was going to say exactly that lol. Claude has improved substantially on Claude Plays Pokemon:
  
  

  

• Daniel KokotajloMar 22, 2025, 4:23 AM

  2 points

  0

  in reply to: Daniel Kokotajlo’s comment on: Daniel Koko­ta­jlo’s Shortform

  https://​​www.reddit.com/​​r/​​CredibleDefense/​​comments/​​1jgokvt/​​yuri_butusov_a_story_about_our_best_strike_uav/​​ Seems like the grey zone is almost 20km wide now. So yeah, I count this as prediction success!

• Daniel KokotajloMar 21, 2025, 11:58 PM

  LW: 9 AF: 7

  8

  AF

  in reply to: jsteinhardt’s comment on: METR: Mea­sur­ing AI Abil­ity to Com­plete Long Tasks

  I don’t believe it. I don’t believe that overall algorithmic progress is 3x faster. Maaaybe coding is 3x faster but that would maybe increase overall algo progress by like 30% idk. But also I don’t think coding is really 3x faster on average for the things that matter.

• Daniel KokotajloMar 20, 2025, 3:38 PM

  LW: 11 AF: 8

  1

  AF

  in reply to: jsteinhardt’s comment on: METR: Mea­sur­ing AI Abil­ity to Com­plete Long Tasks

  I indeed think that AI assistance has been accelerating AI progress. However, so far the effect has been very small, like single-digit percentage points. So it won’t be distinguishable in the data from zero. But in the future if trends continue the effect will be large, possibly enough to more than counteract the effect of scaling slowing down, possibly not, we shall see.

• Daniel KokotajloMar 20, 2025, 3:36 PM

  2 points

  −1

  in reply to: Matt Goldenberg’s comment on: METR: Mea­sur­ing AI Abil­ity to Com­plete Long Tasks

  I agree with that, in fact I think that’s the default case. I don’t think it changes the bottom line, just makes the argument more complicated.

• Daniel KokotajloMar 19, 2025, 8:34 PM

  5 points

  2

  in reply to: Nikola Jurkovic’s comment on: METR: Mea­sur­ing AI Abil­ity to Com­plete Long Tasks

  Indeed I would argue that the trend pretty much has to be inherently superexponential. My argument is still kinda fuzzy, I’d appreciate help in making it more clear. At some point I’ll find time to try to improve it.

• Daniel KokotajloMar 19, 2025, 8:32 PM

  LW: 3 AF: 2

  0

  AF

  in reply to: Matt Goldenberg’s comment on: METR: Mea­sur­ing AI Abil­ity to Com­plete Long Tasks

  I’m not sure if I understand what you are saying. It sounds like you are accusing me of thinking that skills are binary—either you have them or you don’t. I agree, in reality many skills are scalar instead of binary; you can have them to greater or lesser degrees. I don’t think that changes the analysis much though.

• Daniel KokotajloMar 19, 2025, 5:53 PM

  LW: 94 AF: 32

  7

  AF

  on: METR: Mea­sur­ing AI Abil­ity to Com­plete Long Tasks

  This is probably the most important single piece of evidence about AGI timelines right now. Well done! I think the trend should be superexponential, e.g. each doubling takes 10% less calendar time on average. Eli Lifland and I did some calculations yesterday suggesting that this would get to AGI in 2028. Will do more serious investigation soon.
  
  Why do I expect the trend to be superexponential? Well, it seems like it sorta has to go superexponential eventually. Imagine: We’ve got to AIs that can with ~100% reliability do tasks that take professional humans 10 years. But somehow they can’t do tasks that take professional humans 160 years? And it’s going to take 4 more doublings to get there? And these 4 doublings are going to take 2 more years to occur? No, at some point you “jump all the way” to AGI, i.e. AI systems that can do any length of task as well as professional humans -- 10 years, 100 years, 1000 years, etc.
  
  Also, zooming in mechanistically on what’s going on, insofar as an AI system can do tasks below length X but not above length X, it’s gotta be for some reason—some skill that the AI lacks, which isn’t important for tasks below length X but which tends to be crucial for tasks above length X. But there are only a finite number of skills that humans have that AIs lack, and if we were to plot them on a horizon-length graph (where the x-axis is log of horizon length, and each skill is plotted on the x-axis where it starts being important, such that it’s not important to have for tasks less than that length) the distribution of skills by horizon length would presumably taper off, with tons of skills necessary for pretty short tasks, a decent amount necessary for medium tasks (but not short), and a long thin tail of skills that are necessary for long tasks (but not medium), a tail that eventually goes to 0, probably around a few years on the x-axis. So assuming AIs learn skills at a constant rate, we should see acceleration rather than a constant exponential. There just aren’t that many skills you need to operate for 10 days that you don’t also need to operate for 1 day, compared to how many skills you need to operate for 1 hour that you don’t also need to operate for 6 minutes.
  
  There are two other factors worth mentioning which aren’t part of the above: One, the projected slowdown in capability advances that’ll come as compute and data scaling falters due to becoming too expensive. And two, pointing in the other direction, the projected speedup in capability advances that’ll come as AI systems start substantially accelerating AI R&D.