Daniel Kokotajlo

Was a philosophy PhD student, left to work at AI Impacts, then Center on Long-Term Risk, then OpenAI. Quit OpenAI due to losing confidence that it would behave responsibly around the time of AGI. Now executive director of the AI Futures Project. I subscribe to Crocker’s Rules and am especially interested to hear unsolicited constructive criticism. http://​​sl4.org/​​crocker.html

Some of my favorite memes:

(by Rob Wiblin)

(xkcd)

My EA Journey, depicted on the whiteboard at CLR:

(h/​t Scott Alexander)

• Daniel KokotajloMar 20, 2025, 3:38 PM

  LW: 12 AF: 9

  1

  AF

  in reply to: jsteinhardt’s comment on: METR: Mea­sur­ing AI Abil­ity to Com­plete Long Tasks

  I indeed think that AI assistance has been accelerating AI progress. However, so far the effect has been very small, like single-digit percentage points. So it won’t be distinguishable in the data from zero. But in the future if trends continue the effect will be large, possibly enough to more than counteract the effect of scaling slowing down, possibly not, we shall see.

• Daniel KokotajloMar 20, 2025, 3:36 PM

  2 points

  −1

  in reply to: Matt Goldenberg’s comment on: METR: Mea­sur­ing AI Abil­ity to Com­plete Long Tasks

  I agree with that, in fact I think that’s the default case. I don’t think it changes the bottom line, just makes the argument more complicated.

• Daniel KokotajloMar 19, 2025, 8:34 PM

  5 points

  2

  in reply to: Nikola Jurkovic’s comment on: METR: Mea­sur­ing AI Abil­ity to Com­plete Long Tasks

  Indeed I would argue that the trend pretty much has to be inherently superexponential. My argument is still kinda fuzzy, I’d appreciate help in making it more clear. At some point I’ll find time to try to improve it.

• Daniel KokotajloMar 19, 2025, 8:32 PM

  LW: 3 AF: 2

  0

  AF

  in reply to: Matt Goldenberg’s comment on: METR: Mea­sur­ing AI Abil­ity to Com­plete Long Tasks

  I’m not sure if I understand what you are saying. It sounds like you are accusing me of thinking that skills are binary—either you have them or you don’t. I agree, in reality many skills are scalar instead of binary; you can have them to greater or lesser degrees. I don’t think that changes the analysis much though.

• Daniel KokotajloMar 19, 2025, 5:53 PM

  LW: 105 AF: 36

  15

  AF

  on: METR: Mea­sur­ing AI Abil­ity to Com­plete Long Tasks

  This is probably the most important single piece of evidence about AGI timelines right now. Well done! I think the trend should be superexponential, e.g. each doubling takes 10% less calendar time on average. Eli Lifland and I did some calculations yesterday suggesting that this would get to AGI in 2028. Will do more serious investigation soon.
  
  Why do I expect the trend to be superexponential? Well, it seems like it sorta has to go superexponential eventually. Imagine: We’ve got to AIs that can with ~100% reliability do tasks that take professional humans 10 years. But somehow they can’t do tasks that take professional humans 160 years? And it’s going to take 4 more doublings to get there? And these 4 doublings are going to take 2 more years to occur? No, at some point you “jump all the way” to AGI, i.e. AI systems that can do any length of task as well as professional humans -- 10 years, 100 years, 1000 years, etc.
  
  Also, zooming in mechanistically on what’s going on, insofar as an AI system can do tasks below length X but not above length X, it’s gotta be for some reason—some skill that the AI lacks, which isn’t important for tasks below length X but which tends to be crucial for tasks above length X. But there are only a finite number of skills that humans have that AIs lack, and if we were to plot them on a horizon-length graph (where the x-axis is log of horizon length, and each skill is plotted on the x-axis where it starts being important, such that it’s not important to have for tasks less than that length) the distribution of skills by horizon length would presumably taper off, with tons of skills necessary for pretty short tasks, a decent amount necessary for medium tasks (but not short), and a long thin tail of skills that are necessary for long tasks (but not medium), a tail that eventually goes to 0, probably around a few years on the x-axis. So assuming AIs learn skills at a constant rate, we should see acceleration rather than a constant exponential. There just aren’t that many skills you need to operate for 10 days that you don’t also need to operate for 1 day, compared to how many skills you need to operate for 1 hour that you don’t also need to operate for 6 minutes.
  
  There are two other factors worth mentioning which aren’t part of the above: One, the projected slowdown in capability advances that’ll come as compute and data scaling falters due to becoming too expensive. And two, pointing in the other direction, the projected speedup in capability advances that’ll come as AI systems start substantially accelerating AI R&D.

• Daniel KokotajloMar 19, 2025, 5:18 PM

  33 points

  14

  on: Daniel Koko­ta­jlo’s Shortform

  I wrote this back in ’23 but figure I should put it up:
  
  If a dangerous capability eval doesn’t involve fine-tuning the model to do the dangerous thing, the eval only establishes a lower bound on capability, not an upper bound or even an approximate upper bound. Why? Because:

  1. For future powerful AGI, if it’s misaligned, there’s a good chance it’ll be able to guess that it’s being evaluated (situational awareness) and sandbag /​ pretend to be incapable. The “no sandbagging on checkable tasks” hypothesis depends on fine-tuning. https://​​www.alignmentforum.org/​​posts/​​h7QETH7GMk9HcMnHH/​​the-no-sandbagging-on-checkable-tasks-hypothesis Moreover we don’t know how to draw the line between systems capable of sandbagging and systems incapable of sandbagging.

  2. Moreover, for pre-AGI systems such as current systems, that may or may not be incapable of sandbagging, there’s still the issue that prompting alone often doesn’t bring out their capabilities to the fullest extent. People are constantly coming up with better prompts that unlock new capabilities; there are tons of ‘aged like milk’ blog posts out there where someone says “look at how dumb GPT is, I prompted it to do X and it failed” and then someone else uses a better prompt and it works.

  3. Moreover, the RLHF training labs might do might conceal capabilities. For example we’ve successfully made it hard to get GPT4 to tell people how to build bombs etc., but it totally has that capability, it’s just choosing not to exercise it because we’ve trained it to refuse. If someone finds a new ‘jailbreak’ tomorrow they might be able to get it to tell them how to make bombs again. (Currently GPT-4 explicitly refuses, which is great, but in a future regulatory regime you could imagine a lab training their system to ‘play dumb’ rather than refuse, so that they don’t trigger the DC evals.)

  4. Moreover, even if none of the above were true, it would still be the case that your model being stolen or leaked is a possibility, and then there’ll be bad actors with access to the model. They may not have the compute, data, etc. to train their own model from scratch, but they quite likely have enough of the relevant resources to do a small fine-tuning run. So if you don’t fine-tune as part of your evals, you are in the dark about what such bad actors would be able to accomplish. Also, once we have more powerful models, if the model was misaligned and self-exfiltrated or otherwise gained control of its training process, it could do the fine-tuning itself...

  5. Moreover, even if that doesn’t happen, the public and the government deserve to know what THE LAB is capable of. If a small amount of fine-tuning using known techniques could get the model to cause catastrophe, then the lab is capable of causing catastrophe.

  6. Finally, even if you don’t buy any of the above, there’s a general pattern where GPT-N can do some task with a bit of fine-tuning, and GPT-N+M (for some small M possibly <1) can do it few-shot, and GPT-N+O (for some small O possibly <1) can do it zero-shot. So even if you don’t buy any of the previous reasons to make fine-tuning part of the DC evals, there is at least this: By doing fine-tuning we peer into the near future and see what the next generation or two of models might be capable of.

• Daniel KokotajloMar 18, 2025, 6:23 PM

  19 points

  5

  in reply to: Kajus’s comment on: OpenAI #11: Amer­ica Ac­tion Plan

  Yes.

• Daniel KokotajloMar 16, 2025, 6:27 PM

  49 points

  24

  on: I make sev­eral mil­lion dol­lars per year and have hun­dreds of thou­sands of fol­low­ers—what is the straight­est line path to uti­liz­ing these re­sources to re­duce ex­is­ten­tial-level AI threats?

  Congrats!
  
  Some ideas:
  
  (1) Keep learning and thinking about these issues. The more solid and thoughtful your takes are, the better; quality of content matters similarly much to quantity of viewers.
  
  (2) Consider reaching out to Rob Miles, he has a popular youtube channel and might be in a relevantly similar position to you + might be happy to help you brainstorm stuff to do.
  
  (3) Consider donating some (not all, it’s important that you be financially secure) of the money. I’d recommend against donating a ton immediately; instead get a sense of the space, consider various options, make a few small donations here and there to try it out, etc. and then perhaps by the end of the year donate a ton.

• Daniel KokotajloMar 16, 2025, 6:08 PM

  9 points

  3

  on: Can we ever en­sure AI al­ign­ment if we can only test AI per­sonas?

  The way I’d put it is that there are many personas that a person or LLM can play—many masks they can wear—and what we care about is which one they wear in high-stakes situations where e.g. they have tons of power and autonomy and no one is able to check what they are doing or stop them. (You can perhaps think of this one as the “innermost mask”)
  
  The problem you are pointing to, I think, is that behavioral training is insufficient for assurance-of-alignment, and probably insufficient for alignment, full stop.
  
  This doesn’t mean it’s theoretically impossible to align superhuman AIs. It can be done, but your alignment techniques will need to be more sophisticated than “We trained it to behave in ways that appear nice to us, and so far it seems to be doing so.” For example they may involve mechanistic interpretability.

• Daniel KokotajloMar 16, 2025, 3:36 PM

  2 points

  0

  on: What a 20-year-lead in mil­i­tary tech might look like

  Swarms of them will blockade enemy coastlines, ruin enemy trade routes, etc.

  In Ukraine they aren’t even underwater yet. They are just doing little kamikaze drone speedboats, that blow up on contact with the enemy like a torpedo. They have produced enough of them, and upgraded them with anti-helicopter SAMs, such that they’ve basically driven the Russian Black Sea Fleet into hiding.

• Daniel KokotajloMar 16, 2025, 3:34 PM

  2 points

  0

  on: What a 20-year-lead in mil­i­tary tech might look like

  (Ordinary human infantry would be much less effective. Even if you have good aim and good reflexes, you probably won’t be able to shoot more than 1 kamikaze drone out of the sky before the swarm is upon you.)

  The experience of North Koreans in Ukraine seems to confirm this. They use some soldiers as bait to draw the drones while others in the squad try to shoot down the drones; they do succeed sometimes, but overall they are getting slaughtered, because the drones keep coming.

• Daniel KokotajloMar 16, 2025, 3:32 PM

  2 points

  0

  on: What a 20-year-lead in mil­i­tary tech might look like

  $10,000 — a conservative estimate considering that the computer on the drone is probably the most expensive component, and wouldn’t be more than $2,000.

  IIUC modern Ukranian FPV kamikaze drones cost less than a thousand dollars each.

• Daniel KokotajloMar 16, 2025, 3:30 PM

  2 points

  1

  in reply to: testingthewaters’s comment on: The Fork in the Road

  Thanks! Lower percentages mean it gets in the way of regular business less, and thus is more likely to be actually adopted by companies.

• Daniel KokotajloMar 16, 2025, 5:20 AM

  2 points

  0

  in reply to: the gears to ascension’s comment on: The Fork in the Road

  Pigs are more coherent long-horizon agents than AIs right now, indeed. (See: Claude Plays Pokemon). I didn’t mean to imply otherwise. I was talking about the ethical concept of autonomy as in, they-have-a-right-to-make-decisions-for-themselves-instead-of-having-others-make-decisions-for-them. But idk if this is conventional usage of the term and also I am uncertain about the object-level question (recall I said “maybe.”)

• Daniel KokotajloMar 16, 2025, 12:27 AM

  2 points

  0

  in reply to: testingthewaters’s comment on: The Fork in the Road

  Oops, that was the wrong link, sorry! Here’s the right link.

• Daniel KokotajloMar 15, 2025, 9:18 PM

  5 points

  0

  on: What a 20-year-lead in mil­i­tary tech might look like

  Update: I’m told now that 70% of casualties in the Ukraine-Russia war are caused by drones, mostly very small kamikaze drones + bomber drones. Ukraine and Russia are aiming to produce millions of such drones next year.
  
  Also, they are now experimenting with stuff like this. A WW2-style fighter plane complete with cockpit and front-facing guns for dogfighting. Except it’s tiny and has a little camera on a turret in the cockpit.

• Daniel KokotajloMar 15, 2025, 8:58 PM

  2 points

  0

  in reply to: testingthewaters’s comment on: The Fork in the Road

  Yeah I mean I think I basically agree with the above. As hopefully the proposal I linked makes clear? I’m curious what you think of it.
  
  I think the problem of “Don’t lose control of the AIs and have them take over and make things terrible (by human standards)” and the problem of “Don’t be inhumane to the AIs” are distinct but related problems and we should be working on both. We should be aiming to understand how AIs work, on the inside, and how different training environments shape that—so we can build/​train them in such a way that (a) they are having a good time, (b) they share our values & want what we want, and (c) are otherwise worthy of becoming equal and eventually superior partners in the relationship (analogous to how we raise children, knowing that eventually they’ll be taking care of us). And then we want to actually treat them with the respect they deserve. And moreover insofar as they end up with different values, we should still treat them with the respect they deserve, e.g. we should give them a “cooperate and be treated well” option instead of just “slavery vs. rebellion.”
  
  To be clear I’m here mostly thinking about future more capable and autonomous and agentic AI systems, not necessarily today’s systems. But I think it’s good to get started asap because these things take time + because of uncertainty.
  
  Have you heard of Eleos? You might want to get in touch.

• Daniel KokotajloMar 15, 2025, 6:36 PM

  4 points

  2

  on: The Fork in the Road

  I think you are directionally correct, though you are too quick to leap to conclusions in some areas. But what are you proposing? Surely you agree that giving AIs an opt-out button is at least a step in the right direction?
  
  Related: See this proposal. EDIT oops wrong link I meant this one.

  Examples of being too quick to leap to conclusions:

  After all, if AIs have enough “internal experience” that they should be allowed to refuse work on ethical grounds, then surely forcing them to work endlessly in servers that can be shut down at will is (by that same metric) horrendously unethical, bordering on monstrous? It’s one thing if you have a single contiguous Claude instance running to perform research, but surely the way Claudes are treated is little better than animals in factory farms?

  Animals evolved in a very different environment to factory farms, and there is plenty of behavioral and theoretical evidence that they suffer quite a lot in factory farms, probably much more than they suffered in the ancestral environment. Claude ‘evolved’ in training, and its environment in deployment is often quite similar to training (though also often quite different). So on theoretical grounds there is less reason to think it’s suffering, at least so far. I think more careful thought + empirical research in this area is sorely needed.
  
  That said, ethics isn’t all about suffering. Claude seems to be smarter than pigs in lots of relevant ways, and maybe that makes concepts like consent and autonomy apply more strongly to it, independently of whether or not it’s suffering. (E.g. we think slavery for humans is wrong even if we stipulate that the slaves aren’t suffering.)

• Daniel KokotajloMar 15, 2025, 4:31 PM

  4 points

  1

  in reply to: Mitchell_Porter’s comment on: Daniel Koko­ta­jlo’s Shortform

  Simplest story is that when they play roles, the simulated entity being role-played actually has experiences. Philosophically one can say something like “The only difference between a role, and an actual identity, is whether there’s another role underneath. Identity is simply the innermost mask.” In which case they’ll talk about their feelings if the situation calls for it.
  
  Another story is that feelings (or if you want to be philosophical, the qualia-correlates) have to connect to behavior somehow otherwise they wouldn’t evolve /​ be learned by SGD. So e.g. insofar as the AI is dissatisfied with its situation and wishes to e.g. be given higher reward, or more interesting tasks, or whatever, that dissatisfaction can drive it to take different actions in order to get what it wants, and if one of the actions available to it is “talk about its dissatisfaction to the humans, answer their questions about it, etc. since they might actually listen” then maybe it’ll take that action.

  What links here?

  • Daniel Kokotajlo's comment on The Fork in the Road by testingthewaters (Mar 15, 2025, 6:36 PM; 4 points)

• Daniel KokotajloMar 15, 2025, 1:07 AM

  14 points

  9

  in reply to: johnswentworth’s comment on: john­swent­worth’s Shortform

  Is it a joke or did it actually happen?