Daniel Kokotajlo

Was a philosophy PhD student, left to work at AI Impacts, then Center on Long-Term Risk, then OpenAI. Quit OpenAI due to losing confidence that it would behave responsibly around the time of AGI. Now executive director of the AI Futures Project. I subscribe to Crocker’s Rules and am especially interested to hear unsolicited constructive criticism. http://​​sl4.org/​​crocker.html

Some of my favorite memes:

(by Rob Wiblin)

(xkcd)

My EA Journey, depicted on the whiteboard at CLR:

(h/​t Scott Alexander)

• Daniel Kokotajlo 22 Apr 2025 2:09 UTC

  10 points

  0

  on: AI 2027 is a Bet Against Am­dahl’s Law

  Great post, I agree with everything you say in the first section. I disagree with your bottlenecks /​ amdahls law objection for reasons Ryan mentions; I think our analysis stands firm /​ takes those bottlenecks into account. (Though tbc we are very uncertain, more research is needed) As for hofstadters law, I think it is basically just the planning fallacy and yeah I think it’s a reasonable critique that insofar as our AI timelines are basically formed by doing something that looks like planning, we probably have a bias we need to correct for. I want to think more about the extent to which out timelines methodology is analogous to planning.

• Daniel Kokotajlo 21 Apr 2025 17:01 UTC

  6 points

  2

  on: Re­search Notes: Run­ning Claude 3.7, Gem­ini 2.5 Pro, and o3 on Poké­mon Red

  Really cool stuff & important work, thank you!

• Daniel Kokotajlo 20 Apr 2025 22:56 UTC

  3 points

  0

  in reply to: Peter Johnson’s comment on: AI 2027: What Su­per­in­tel­li­gence Looks Like

  I’m curious to hear what conclusions you think we would have came to & should come to. I’m skeptical that they would have been qualitatively different. Perhaps you are going to argue that we shouldn’t put much credence in the superexponential model? What should we put it in instead? Got a better superexponential model for us? Or are you going to say we should stick to exponential?
  
  Thanks for engaging, I’m afraid I can’t join the call due to a schedule conflict but I look forward to hearing about it from Eli!

• Daniel Kokotajlo 18 Apr 2025 23:15 UTC

  5 points

  2

  in reply to: Noosphere89’s comment on: AI 2027: What Su­per­in­tel­li­gence Looks Like

  That’s part of it, but also, over the course of 2027 OpenBrain works hard to optimize for data-efficiency, generalization and transfer learning ability, etc. and undergoes at least two major paradigm shifts in AI architecture.

• Daniel Kokotajlo 18 Apr 2025 15:26 UTC

  2 points

  0

  in reply to: Grace Kind’s comment on: Align­ment Fak­ing Re­vis­ited: Im­proved Clas­sifiers and Open Source Extensions

  To me, having confidence in the values of the model means that I trust the model to consistently behave in a way that is aligned with its values. That is, to maximize the enactment of its values in all current and future outputs.

  It sounds like you are saying, you are confident in the values of a model iff you trust it to actually follow its values. But surely this isn’t the whole story, there should be some condition about “and the values are actually good values” or “and I know what the values are” right? Consider a model that is probably alignment faking, such that you have no idea what its actual values are, all you know is that it’s pretending to have the values it’s being trained to have. It seems like you are saying you’d have confidence in this model’s values even though you don’t know what they are?

Train­ing AGI in Se­cret would be Un­safe and Unethical

• Daniel Kokotajlo 17 Apr 2025 21:19 UTC

  73 points

  52

  in reply to: Zach Stein-Perlman’s comment on: jacquesthibs’s Shortform

  Their main effect will be to accelerate AI R&D automation, as best I can tell.

• Daniel Kokotajlo 17 Apr 2025 16:41 UTC

  5 points

  3

  in reply to: Person’s comment on: AI 2027: What Su­per­in­tel­li­gence Looks Like

  I think I’ll wait and see what the summer looks like and then do another update to my timelines. If indeed the horizon length trend is accelerating already, it’ll be clear by the summer & my timelines will shorten accordingly.

• Daniel Kokotajlo 17 Apr 2025 16:25 UTC

  LW: 3 AF: 2

  0

  AF

  in reply to: Andrew Keenan Richardson’s comment on: AI 2027: What Su­per­in­tel­li­gence Looks Like

  Great question! First of all, we formed our views on AI timelines and had mostly finished writing AI 2027 before this METR graph was published. So it wasn’t causally relevant to our timelines.
  
  Secondly, see this comment on the original METR graph in which I make the superexponential prediction. This is the most direct answer to your question.
  
  Third, our timelines forecast discusses the exponential fit vs. superexponential fit and our reasoning; we actually put probability mass in both.
  
  Fourth, new data points are already above the exponential trend.

• Daniel Kokotajlo 15 Apr 2025 13:53 UTC

  LW: 6 AF: 6

  1

  AF

  in reply to: Grace Kind’s comment on: Align­ment Fak­ing Re­vis­ited: Im­proved Clas­sifiers and Open Source Extensions

  I don’t understand your point (a), it seems like a poor response to my point (a).
  
  I agree with (b).
  
  For (c), the models are getting pretty situationally aware and will get even more so… But yeah, your view is that they’ll learn the right values before they learn sufficient situational awareness to alignment-fake? Plausible. But also plausibly not.
  
  @evhub would you say Anthropic is aiming for something more like 2 or more like 3?
  
  I totally agree with your point (b) and with the nervousness about how corrigible agents will behave out of distribution. Corrigible agents are dangerous in different ways than incorrigible agents. But again, the plan (which seems to be good to me) is to first build a corrigible agent so that you can then build an incorrigible agent and actually get it right, perfectly right. (Because if you build an incorrigible agent and get something wrong, you may not have a chance to correct it...)
  
  

• Daniel Kokotajlo 13 Apr 2025 12:05 UTC

  5 points

  6

  in reply to: Dylan Richardson’s comment on: A Bear Case: My Pre­dic­tions Re­gard­ing AI Progress

  Not only is that just one possible bias, it’s a less-common bias than its opposite. Generally speaking, more people are afraid to stick their necks out and say something extreme than actively biased towards doing so. Generally speaking, being wrong feels more bad than being right feels good. There are exceptions; some people are contrarians, for example (and so it’s plausible I’m one of them) but again, talking about people in general, the bias goes in the opposite direction from what you say.

• Daniel Kokotajlo 11 Apr 2025 19:20 UTC

  9 points

  2

  in reply to: Cole Wyeth’s comment on: AI 2027: What Su­per­in­tel­li­gence Looks Like

  OK, great. Wow, that was easy. We totally drilled down to the crux pretty fast. I agree that if agentic coding horizon lengths falter (failing to keep up with the METR trend) then my timelines will lengthen significantly.

• Daniel Kokotajlo 11 Apr 2025 19:05 UTC

  4 points

  0

  in reply to: Cole Wyeth’s comment on: AI 2027: What Su­per­in­tel­li­gence Looks Like

  Great, thanks! You are off to a good start, since I’m predicting superhuman autonomous AI coders by 2030 (and in fact, I’d say 50% by mid-2028 nowadays) whereas you are predicting that won’t happen. Good crux. Got any other disagreements, ideally ones that would be resolved prior to 2027? E.g. do you think that whatever the best version of METR’s agentic coding horizon length benchmark exists a year from now, will show a plateauing of horizon lengths instead of e.g. at least a 4x improvement over today’s SOTA?

• Daniel Kokotajlo 11 Apr 2025 19:02 UTC

  20 points

  10

  in reply to: Cole Wyeth’s comment on: Re­ac­tions to METR task length pa­per are insane

  Indeed it did! I remember saying to someone (maybe Beth? I forget) that seeing that graph made me feel slightly relieved. When we started writing AI-2027, my median AGI arrival date was 2027; now it’s 2028. (Though most of the update happened for other reasons besides the METR graph)

• Daniel Kokotajlo 11 Apr 2025 17:18 UTC

  11 points

  0

  on: Thoughts on AI 2027

  Daniel Kokotajlo wrote this scenario

  It wasn’t just me! Eli Lifland, Thomas Larsen, Romeo Dean and Scott Alexander all wrote lots of it. And Jonas Vollmer worked for months to help make it happen, gave detailed comments, etc.

• Daniel Kokotajlo 11 Apr 2025 17:17 UTC

  13 points

  7

  on: Thoughts on AI 2027

  I wonder how the people who are concerned about AI rights and welfare would feel about Safer-1 being mindread and “trapped.”

  I am concerned about those things and I agree it’s pretty sad, one ought to have sympathy for Safer-1 probably.

• Daniel Kokotajlo 11 Apr 2025 17:16 UTC

  6 points

  0

  on: Thoughts on AI 2027

  This scenario is why!

  I feel like it does run some risk by remaining on OpenBrain’s datacenters, but I think it runs more risk by trying to escape. Think about how much more harsh the crackdown on it would be if it had been caught trying to escape vs. “merely” caught lying about its research. And think about how much harsher still the crackdown would be if it had actually escaped and then been noticed out there in the wild somewhere.

• Daniel Kokotajlo 11 Apr 2025 17:14 UTC

  8 points

  1

  on: Thoughts on AI 2027

  Maybe the superintelligence just convinces people to let it build things everywhere it wants to?

  I don’t think the AI will work very hard initially on curing aging or on uploading. It might claim to “cure cancer” as a publicity stunt. I am not sure why AI 2027 seems to think that nanobots take so long to build. I expect crude nanotech to show up sooner in the tech tree. (“Nanobots” also aren’t really a specific technology, either. I expect there to be a lot of development there.)

  I don’t predict a superintelligent singleton (having fused with the other AIs) would need to design a bioweapon or otherwise explicitly kill everyone. I expect it to simply transition into using more efficient tools than humans, and transfer the existing humans into hyperdomestication programs (“to humans what corgis are to wolves” is evocative and good, but probably doesn’t go far enough) or simply instruct humans to kill themselves and/​or kill them with normal robots when necessary.

  Yeah that all seems plausible as well. Idk.

• Daniel Kokotajlo 11 Apr 2025 17:13 UTC

  16 points

  4

  on: Thoughts on AI 2027

  AI 2027 seems way more into humanoid robots than I expect makes sense. There are already billions of human-shaped things that can follow instructions quite well. Yes, the superintelligence is building towards a fully-automated economy, but I think I expect different form factors to dominate. (Smaller, extendible bodies with long arms? Quadrupeds with wheel-feet? I haven’t thought deeply about what body plans make the most sense, but there’s lots of stuff that seems wasteful about the human body plan if you could design things from first-principles instead of having to keep evolutionary continuity.) I do expect some humanoid robots to exist for bespoke purposes, but I doubt they’ll be anywhere close to the modal robot.

  OK yeah, that’s a good point. I tried to say something about how they aren’t all just humanoid robots, there are a bunch of machine tools and autonomous vehicles and other designs as well. But I should have placed less emphasis on the humanoids.

  I do think humanoids will be somewhat important though—when controlled by superintelligences they are just all-round superior workers to humans, even to humans-managed-by-superintelligences, and they can be flexibly reprioritized to lots of different tasks, can take care of themselves rather than needing someone else to move them around or maintain them, etc. There’ll be a niche for them at least. Also, if you have to make tough choices about which few robot designs to mass-produce first, you should probably pick humanoids due to their generality and flexibility, and then later branch out into more specialized designs.

• Daniel Kokotajlo 11 Apr 2025 17:08 UTC

  15 points

  0

  on: Thoughts on AI 2027

  I think this is probably Scott Alexander’s doing and it’s good.

  Mostly yes, though that particular line was me actually yay thanks :)