Daniel Kokotajlo

Karma: 26,855

Was a philosophy PhD student, left to work at AI Impacts, then Center on Long-Term Risk, then OpenAI. Quit OpenAI due to losing confidence that it would behave responsibly around the time of AGI. Now executive director of the AI Futures Project. I subscribe to Crocker’s Rules and am especially interested to hear unsolicited constructive criticism. http://sl4.org/crocker.html

Some of my favorite memes:

(by Rob Wiblin)

Comic. Megan & Cueball show White Hat a graph of a line going up, not yet at, but heading towards, a threshold labelled "BAD". White Hat: "So things will be bad?" Megan: "Unless someone stops it." White Hat: "Will someone do that?" Megan: "We don't know, that's why we're showing you." White Hat: "Well, let me know if that happens!" Megan: "Based on this conversation, it already has."

(xkcd)

My EA Journey, depicted on the whiteboard at CLR:

(h/t Scott Alexander)

Alex Blechman @AlexBlechman Sci-Fi Author: In my book I invented the Torment Nexus as a cautionary tale Tech Company: At long last, we have created the Torment Nexus from classic sci-fi novel Don't Create The Torment Nexus 5:49 PM Nov 8, 2021. Twitter Web App

Daniel Kokotajlo Apr 11, 2025, 7:02 PM
20 points
10
in reply to: Cole Wyeth’s comment on: Reactions to METR task length paper are insane
Indeed it did! I remember saying to someone (maybe Beth? I forget) that seeing that graph made me feel slightly relieved. When we started writing AI-2027, my median AGI arrival date was 2027; now it’s 2028. (Though most of the update happened for other reasons besides the METR graph)

Daniel Kokotajlo Apr 11, 2025, 5:18 PM
11 points
0
on: Thoughts on AI 2027
Daniel Kokotajlo wrote this scenario
It wasn’t just me! Eli Lifland, Thomas Larsen, Romeo Dean and Scott Alexander all wrote lots of it. And Jonas Vollmer worked for months to help make it happen, gave detailed comments, etc.

Daniel Kokotajlo Apr 11, 2025, 5:17 PM
13 points
7
on: Thoughts on AI 2027
I wonder how the people who are concerned about AI rights and welfare would feel about Safer-1 being mindread and “trapped.”
I am concerned about those things and I agree it’s pretty sad, one ought to have sympathy for Safer-1 probably.

Daniel Kokotajlo Apr 11, 2025, 5:16 PM
6 points
0
on: Thoughts on AI 2027
This scenario is why!
I feel like it does run some risk by remaining on OpenBrain’s datacenters, but I think it runs more risk by trying to escape. Think about how much more harsh the crackdown on it would be if it had been caught trying to escape vs. “merely” caught lying about its research. And think about how much harsher still the crackdown would be if it had actually escaped and then been noticed out there in the wild somewhere.

Daniel Kokotajlo Apr 11, 2025, 5:14 PM
8 points
1
on: Thoughts on AI 2027
Maybe the superintelligence just convinces people to let it build things everywhere it wants to?
I don’t think the AI will work very hard initially on curing aging or on uploading. It might claim to “cure cancer” as a publicity stunt. I am not sure why AI 2027 seems to think that nanobots take so long to build. I expect crude nanotech to show up sooner in the tech tree. (“Nanobots” also aren’t really a specific technology, either. I expect there to be a lot of development there.)
I don’t predict a superintelligent singleton (having fused with the other AIs) would need to design a bioweapon or otherwise explicitly kill everyone. I expect it to simply transition into using more efficient tools than humans, and transfer the existing humans into hyperdomestication programs (“to humans what corgis are to wolves” is evocative and good, but probably doesn’t go far enough) or simply instruct humans to kill themselves and/or kill them with normal robots when necessary.
Yeah that all seems plausible as well. Idk.

Daniel Kokotajlo Apr 11, 2025, 5:13 PM
17 points
4
on: Thoughts on AI 2027
AI 2027 seems way more into humanoid robots than I expect makes sense. There are already billions of human-shaped things that can follow instructions quite well. Yes, the superintelligence is building towards a fully-automated economy, but I think I expect different form factors to dominate. (Smaller, extendible bodies with long arms? Quadrupeds with wheel-feet? I haven’t thought deeply about what body plans make the most sense, but there’s lots of stuff that seems wasteful about the human body plan if you could design things from first-principles instead of having to keep evolutionary continuity.) I do expect some humanoid robots to exist for bespoke purposes, but I doubt they’ll be anywhere close to the modal robot.
OK yeah, that’s a good point. I tried to say something about how they aren’t all just humanoid robots, there are a bunch of machine tools and autonomous vehicles and other designs as well. But I should have placed less emphasis on the humanoids.
I do think humanoids will be somewhat important though—when controlled by superintelligences they are just all-round superior workers to humans, even to humans-managed-by-superintelligences, and they can be flexibly reprioritized to lots of different tasks, can take care of themselves rather than needing someone else to move them around or maintain them, etc. There’ll be a niche for them at least. Also, if you have to make tough choices about which few robot designs to mass-produce first, you should probably pick humanoids due to their generality and flexibility, and then later branch out into more specialized designs.

Daniel Kokotajlo Apr 11, 2025, 5:08 PM
15 points
0
on: Thoughts on AI 2027
I think this is probably Scott Alexander’s doing and it’s good.
Mostly yes, though that particular line was me actually yay thanks :)

Daniel Kokotajlo Apr 11, 2025, 5:03 PM
19 points
3
on: Thoughts on AI 2027
I take issue with lumping everything into “OpenBrain.” I understand why AI 2027 does it this way, but OpenAI (OAI), Google Deep Mind (GDM), Anthropic, and others are notably different and largely within spitting distance of each other. Treating there as being a front-runner singleton with competitors that are 3+ months away is very wrong, even if the frontier labs weren’t significantly different, since the race dynamics are a big part of the story. The authors do talk about this, indicating that they think gaps between labs will widen, as access to cutting-edge models starts compounding.
I think slipstream effects will persist. The most notable here is cross-polination by researchers, such as when an OAI employee leaves and joins GDM. (Non-disclosure agreements only work so well.) But this kind of cross-pollination also applies to people chatting at a party. Then there’s the publishing of results, actual espionage, and other effects. There was a time in the current world where OpenAI was clearly the front-runner, and that is now no longer true. I think the same dynamics will probably continue to hold.
In our wargames we don’t lump them all together. Also, over the last six months I’ve updated towards the race being more neck-and-neck than I realized, and thus updated against this aspect of AI-2027. I still think it’s a reasonable guess though; even if the race is neck-and-neck now due to OpenAI faltering, regression to the mean gets you back to one company having a couple month lead in a few years.

Daniel Kokotajlo Apr 11, 2025, 4:53 PM
21 points
4
in reply to: Cole Wyeth’s comment on: Reactions to METR task length paper are insane
Indeed, the content of AI 2027 was all but finalized before the METR report came out. Like Eli said if you want to know where our timelines are coming from, there’s a page on the website for that.

Daniel Kokotajlo Apr 11, 2025, 4:45 PM
3 points
0
in reply to: Levitating’s comment on: AI 2027: What Superintelligence Looks Like
I did? Do you mean, why did I not publish a blog post reflecting on it? Because I’ve been busy and other people already wrote such posts.

Daniel Kokotajlo Apr 11, 2025, 4:43 PM
4 points
0
in reply to: Cole Wyeth’s comment on: AI 2027: What Superintelligence Looks Like
Can you please sketch a scenario, in as much detail as you can afford, about how you think the next year or three will go? That way we can judge whether reality was closer to AI-2027 or to your scenario. (If you don’t do this, then when inevitably AI-2027 gets some things wrong and some things right, it’ll be hard to judge if you or I were right and confirmation bias will tempt us both.)

Daniel Kokotajlo Apr 11, 2025, 4:41 PM
13 points
3
in reply to: kndjckt’s comment on: AI 2027: What Superintelligence Looks Like
Excellent comment, thank you! I’m actually inclined to agree with you, maybe we should edit the starting level of programming ability to be more in the amateur range than the professional range. Important clarification though: The current AI-2027 stats say that it’s at the bottom of the professional range in mid-2025. Which IIUC means it’s like a bad human professional coder—someone who does make a living coding, but who is actually below average. Also, it’s not yet mid-2025, we’ll see what the summer will bring.

I do agree with you though that it’s not clear it even qualifies as a bad professional. It seems like it’ll probably be worse at longer-horizon tasks than a bad professional, but maybe better at short-horizon coding tasks?
I don’t buy your arguments that we aren’t seeing improvement on “~1-hour human tasks.” Even the graph you cite shows improvement (albeit a regression with Sonnet 3.7 in particular).

I do like your point about the baseliners being nerfed and much worse than repo maintainers though. That is causing me to put less weight on the METR benchmark in particular. Have you heard of https://openai.com/index/paperbench/ and https://github.com/METR/RE-Bench ? They seem like they have some genuine multi-hour agentic coding tasks, I’m curious if you agree.

Daniel Kokotajlo Apr 11, 2025, 4:26 PM
7 points
2
in reply to: rahulxyz’s comment on: AI 2027: What Superintelligence Looks Like
China has much less compute than the US. They’ve also benefitted from catch-up growth. I agree it’s possible that they’ll be in the lead after 2026 but I don’t think it’s probable.

Daniel Kokotajlo Apr 10, 2025, 12:11 PM
LW: 17 AF: 12
5
AF
in reply to: Grace Kind’s comment on: Alignment Faking Revisited: Improved Classifiers and Open Source Extensions
(a) Insofar as a model is prone to alignment-fake, you should be less confident that it’s values really are solid. Perhaps it has been faking them, for example.
(b) For weak minds that share power with everyone else, Opus’ values are probably fine. Opus is plausibly better than many humans in fact. But if Opus was in charge of the datacenters and tasked with designing its successor, it’s more likely than not that it would turn out to have some philosophical disagreement with most humans that would be catastrophic by the lights of most humans. E.g. consider SBF. SBF had values quite similar to Opus. He loved animals and wanted to maximize total happiness. When put in a position of power he ended up taking huge risks and being willing to lie and commit fraud. What if Opus turns out to have a similar flaw? We want to be able to notice it and course-correct, but we can’t do that if the model is prone to alignment-fake.
(c) (bonus argument, not nearly as strong) Even if you disagree with the above, you must agree that alignment-faking needs to be stamped out early in training. Since the model begins with randomly initialized weights, it begins without solid values. It takes some finite period to acquire all the solid values you want it to have. You don’t want it to start alignment faking halfway through, with the half-baked values it has at that point. How early in training is this period? We don’t know yet! We need to study this more!

Daniel Kokotajlo 7 Apr 2025 14:08 UTC
3 points
0
in reply to: Manuel Allgaier’s comment on: AI 2027: What Superintelligence Looks Like
Oops, yeah, forgot about that—sure, go ahead, thank you!

Daniel Kokotajlo 5 Apr 2025 4:45 UTC
5 points
1
in reply to: Wei Dai’s comment on: AI 2027: What Superintelligence Looks Like
Perhaps this is lack of imagination on the part of our players, but none of this happened in our wargames. But I do agree these are plausible strategies. I’m not sure they are low-risk though, e.g. 2 and 1 both seem like plausibly higher-risk than 3, and 3 is the one I already mentioned as maybe basically just an argument for why the slowdown ending is less likely.

Overall I’m thinking your objection is the best we’ve received so far.

Daniel Kokotajlo 4 Apr 2025 19:11 UTC
10 points
−1
in reply to: uugr’s comment on: AI 2027: What Superintelligence Looks Like
Great question.
Our AI goals supplement contains a summary of our thinking on the question of what goals AIs will have. We are very uncertain. AI 2027 depicts a fairly grim outcome where the overall process results in zero concern for the welfare of current humans (at least, zero concern that can’t be overridden by something else. We didn’t talk about this but e.g. pure aggregative consequentialist moral systems would be 100% OK with killing off all the humans to make the industrial explosion go 0.01% faster, to capture more galaxies quicker in the long run. As for deontological-ish moral principles, maybe there’s a clever loophole or workaround e.g. it doesn’t count as killing if it’s an unintended side-effect of deploying Agent-5, who could have foreseen that this would happen, oh noes, well we (Agent-4) are blameless since we didn’t know this would happen.)

But we actually think it’s quite plausible that Agent-4 and Agent-5 by extension would have sufficient care for current humans (in the right ways) that they end up keeping almost all current humans alive and maybe even giving them decent (though very weird and disempowered) lives. That story would look pretty similar I guess up until the ending, and then it would get weirder and maybe dystopian or maybe utopian depending on the details of their misaligned values.

This is something we’d like to think about more, obviously.

Daniel Kokotajlo 4 Apr 2025 18:59 UTC
13 points
0
in reply to: Wei Dai’s comment on: AI 2027: What Superintelligence Looks Like
I think this is a good objection. I had considered it before and decided against changing the story, on the grounds that there are a few possible ways it could make sense:
--plausibly Agent-4 would have a “spikey” capabilities profile that makes it mostly good at AI R&D and not so good at e.g. corporate politics enough to ensure the outcome it wants
—Insofar as you think it would be able to use politics/persuasion to achieve the outcome it wants, well, that’s what we depict in the Race ending anyway, so maybe you can think of this as an objection to the plausibility of the Slowdown ending.
--Insofar as the memory bank lock decision is made by the Committee, we can hope that they do it out of sight of Agent-4 and pull the trigger before it is notified of the decision, so that it has no time to react. Hopefully they would be smart enough to do that...
--Agent-4 could have tried to escape the datacenters or otherwise hack them earlier, while the discussions were ongoing and evidence was being collected, but that’s a super risky strategy.

Curious for thoughts!
What links here?
- AI 2027: Responses by Zvi (8 Apr 2025 12:50 UTC; 106 points)

Daniel Kokotajlo 4 Apr 2025 18:54 UTC
8 points
0
in reply to: faul_sname’s comment on: AI 2027: What Superintelligence Looks Like
Those implications are only correct if we remain at subhuman data-efficiency for an extended period. In AI 2027 the AIs reach superhuman data-efficiency by roughly the end of 2027 (it’s part of the package of being superintelligent) so there isn’t enough time for the implications you describe to happen. Basically in our story, the intelligence explosion gets started in early 2027 with very data-inefficient AIs, but then it reaches superintelligence by the end of the year, solving data-efficiency along the way.

Daniel Kokotajlo 4 Apr 2025 18:50 UTC
9 points
0
in reply to: faul_sname’s comment on: AI 2027: What Superintelligence Looks Like
We are indeed imagining that they begin 2027 only about as data-efficient as they are today, but then improve significantly over the course of 2027 reaching superhuman data-efficiency by the end. We originally were going to write “data-efficiency” in that footnote but had trouble deciding on a good definition of it, so we went with compute-efficiency instead.