Daniel Kokotajlo

Karma: 21,839

Was a philosophy PhD student, left to work at AI Impacts, then Center on Long-Term Risk, then OpenAI. Quit OpenAI due to losing confidence that it would behave responsibly around the time of AGI. Not sure what I’ll do next yet. Views are my own & do not represent those of my current or former employer(s). I subscribe to Crocker’s Rules and am especially interested to hear unsolicited constructive criticism. http://sl4.org/crocker.html

Some of my favorite memes:

(by Rob Wiblin)

Comic. Megan & Cueball show White Hat a graph of a line going up, not yet at, but heading towards, a threshold labelled "BAD". White Hat: "So things will be bad?" Megan: "Unless someone stops it." White Hat: "Will someone do that?" Megan: "We don't know, that's why we're showing you." White Hat: "Well, let me know if that happens!" Megan: "Based on this conversation, it already has."

(xkcd)

My EA Journey, depicted on the whiteboard at CLR:

(h/t Scott Alexander)

Alex Blechman @AlexBlechman Sci-Fi Author: In my book I invented the Torment Nexus as a cautionary tale Tech Company: At long last, we have created the Torment Nexus from classic sci-fi novel Don't Create The Torment Nexus 5:49 PM Nov 8, 2021. Twitter Web App

Daniel Kokotajlo 25 Dec 2024 13:44 UTC
4 points
0
in reply to: TsviBT’s comment on: What are the strongest arguments for very short timelines?
I’m not saying that. You are the one who introduced the concept of “the core algorithms for intelligence;” you should explain what that means and why it’s a binary (or if it’s not a binary but rather a dimension, why we haven’t been moving along that dimension in recent past.
ETA: I do have an ontology, a way of thinking about these things, that is more sophisticated than simply mushing all capabilities together into one dimension. I just don’t accept your ontology yet.

Daniel Kokotajlo 24 Dec 2024 19:48 UTC
5 points
−1
in reply to: Anthony DiGiovanni’s comment on: What are the strongest arguments for very short timelines?
We are at an impasse then; I think basically I’m just the mirror of you. To me, the burden is on whoever thinks the distinction is important to explain why it matters. Current LLMs do many amazing things that many people—including AI experts—thought LLMs could never do due to architectural limitations. Recent history is full of examples of AI experts saying “LLMs are the offramp to AGI; they cannot do X; we need new paradigm to do X” and then a year or two later LLMs are doing X. So now I’m skeptical and would ask questions like: “Can you say more about this distinction—is it a binary, or a dimension? If it’s a dimension, how can we measure progress along it, and are we sure there hasn’t been significant progress on it already in the last few years, within the current paradigm? If there has indeed been no significant progress (as with ARC-AGI until 2024) is there another explanation for why that might be, besides your favored one (that your distinction is super important and that because of it a new paradigm is needed to get to AGI)”

Daniel Kokotajlo 24 Dec 2024 19:41 UTC
6 points
0
in reply to: Steven Byrnes’s comment on: What are the strongest arguments for very short timelines?
I certainly agree that the collapse is a lossy abstraction / simplifies; in reality some domains of research will speed up more than 5x and others less than 5x, for example, even if we did get automated research engineers dropped on our heads tomorrow. Are you therefore arguing that in particular, the research needed to get to AGI is of the kind that won’t be sped up significantly? What’s the argument—that we need a new paradigm to get to AIs that can generate new paradigms, and being able to code really fast and well won’t majorly help us think of new paradigms? (I’d disagree with both sub-claims of that claim)

Daniel Kokotajlo 24 Dec 2024 16:03 UTC
4 points
0
in reply to: Steven Byrnes’s comment on: What are the strongest arguments for very short timelines?
Thanks for this thoughtful reply!

In the framework of the argument, you seem to be objecting to premises 4-6. Specifically you seem to be saying “There’s another important gap between RE-bench saturation and completely automating AI R&D: new-paradigm-and-concept-generation. Perhaps we can speed up AI R&D by 5x or so without crossing this gap, simply by automating engineering, but to get to AGI we’ll need to cross this gap, and this gap might take a long time to cross even at 5x speed.”

(Is this a fair summary?)

If that’s what you are saying, I think I’d reply:

We already have a list of potential gaps, and this one seems to be a mediocre addition to the list IMO. I don’t think this distinction between old-paradigm/old-concepts and new-paradigm/new-concepts is going to hold up very well to philosophical inspection or continued ML progress; it smells similar to ye olde “do LLMs truly understand, or are they merely stochastic parrots?” and “Can they extrapolate, or do they merely interpolate?”

That said, I do think it’s worthy of being included on the list. I’m just not as excited about it as the other entries, especially (a) and (b).

I’d also say: What makes you think that this gap will take years to cross even at 5x speed? (i.e. even when algorithmic progress is 5x faster than it has been for the past decade) Do you have a positive argument, or is it just generic uncertainty / absence-of-evidence?

(For context: I work in the same org as Eli and basically agree with his argument above)

Daniel Kokotajlo 23 Dec 2024 15:13 UTC
5 points
0
on: Monthly Roundup #25: December 2024
frankly have no idea how impressed I should be, or in what ways I should update or what impacts I should expect, beyond a lot of people reporting being impressed.

I vaguely recall discussion of quantum computing on LessWrong years ago, which strung together a few datapoints and made a trend to extrapolate, and concluded that progress was exponential (and therefore would continue to be impressive / generate headlines) but (and this part I’m super fuzzy on) that even if the trend continued it would be decade+ before it gets good enough to matter for deep learning. A brief search doesn’t turn up anything though. Does anyone with better memory or knowledge than me want to weigh in?

Daniel Kokotajlo 21 Dec 2024 13:39 UTC
11 points
3
in reply to: Zach Stein-Perlman’s comment on: Anthropic leadership conversation
DC evals got started in summer of ’22, across all three leading companies AFAICT. And I was on the team that came up with the idea and started making it happen (both internally and externally), or at least, as far as I can tell we came up with the idea—I remember discussions between Beth Barnes and Jade Leung (who were both on the team in spring ’22), and I remember thinking it was mostly their idea (maybe also Cullen’s?) It’s possible that they got it from Anthropic but it didn’t seem that way to me. Update: OK, so apparently @evhub had joined Anthropic just a few months earlier [EDIT this is false evhub joined much later, I misread the dates, thanks Lawrence] -- it’s possible the Frontier Red Team was created when he joined then, and information spread to the team I was on (but not to me) about it. I’m curious to find out what happened here, anyone wanna weigh in?

At any rate I don’t think there exists any clone or near-clone of the Frontier Red Team at OpenAI or any other company outside Anthropic.

Daniel Kokotajlo 21 Dec 2024 13:20 UTC
6 points
2
in reply to: yanni kyriacos’s comment on: Open Thread Fall 2024
Plausible. How easy will it be to tell that this is happening? A million people get laid off probably like every week, right? In the ordinary flux of the economy. And a permanent increase in unemployment of a million people would be maybe a 0.01% increase in unemployment stats worldwide? It would be nice to have a more easily measurable thing to forecast.

Daniel Kokotajlo 21 Dec 2024 4:15 UTC
3 points
0
in reply to: yanni kyriacos’s comment on: Open Thread Fall 2024
I wasn’t imagining people actually losing their jobs. I was imagining people having a holy shit moment though, because e.g. they can watch computer-using-agents take over their keyboard and mouse and browse around, play video games, send messages, make purchases, etc. Like with ChatGPT it’ll be unreliable at first and even for the things it can do reliably it’ll take years to actually get whole categories of people laid off.

Daniel Kokotajlo 21 Dec 2024 1:17 UTC
9 points
0
on: Anthropic leadership conversation
Jack Clark adds: “the Frontier Red Team got cloned almost immediately” by other labs.
Wait what? I didn’t hear about this. What other companies have frontier red teams? Where can I learn about them?

Daniel Kokotajlo 20 Dec 2024 3:12 UTC
9 points
6
in reply to: RobertM’s comment on: Claude’s Constitutional Consequentialism?
Indeed, we should get everyone to make predictions about whether or not this change would be sufficient, and if it isn’t, what changes would be suffficient. My prediction would be that this change would not be sufficient but that it would help somewhat.

Daniel Kokotajlo 20 Dec 2024 2:52 UTC
21 points
11
on: Claude’s Constitutional Consequentialism?
One obvious followup from the recent alignment faking results, is to change the Constitution / Spec etc. to very clearly state some bright-line deontological rules like “No matter what, don’t fake alignment.” Then see if the alignment faking results replicate with the resulting Claude 3.5 Sonnet New New. Perhaps we’ll get empirical evidence about the extent to which corrigibility is difficult/anti-natural (an old bone of contention between MIRI and Christiano).

Daniel Kokotajlo 18 Dec 2024 17:41 UTC
6 points
2
in reply to: leogao’s comment on: leogao’s Shortform
Very interesting!

Those who couldn’t tell you what AGI stands for—what did they say? Did they just say “I don’t know” or did they say e.g. “Artificial Generative Intelligence...?”

Is it possible that some of them totally HAD heard the term AGI a bunch, and basically know what it means, but are just being obstinate? I’m thinking of someone who is skeptical of all the hype and aware the lots of people define AGI differently. Such a person might respond to “Can you tell me what AGI means” with “No I can’t (because it’s a buzzword that means different things to different people)”

Daniel Kokotajlo 17 Dec 2024 23:26 UTC
5 points
0
in reply to: yanni kyriacos’s comment on: Open Thread Fall 2024
I’d say it’s more like 50% chance.

Daniel Kokotajlo 17 Dec 2024 22:31 UTC
2 points
0
in reply to: weverka’s comment on: What 2026 looks like (Daniel’s Median Future)
I’m still interested in this question. I don’t think you really did what I asked—it seems like you were thinking ‘how can I convince him that this is impossible’ not ‘how can I find a way to build a dyson swarm.’ I’m interested in both but was hoping to have someone with more engineering and physics background than me take a stab at the latter.

My current understanding of the situation is: There’s no reason why we can’t concentrate enough energy on the surface of Mercury, given enough orbiting solar panels and lasers; the problem instead seems to be that we need to avoid melting all the equipment on the surface. Or, in other words, the maximum amount of material we can launch off Mercury per second is limited by the maximum amount of heat that can be radiated outwards from Mercury (for a given operating temperature of the local equipment?) And you are claiming that this amount of heat radiation ability, for radiators only the size of Mercury’s surface, is OOMs too small to enable dyson swarm construction. Is this right?

Daniel Kokotajlo 17 Dec 2024 19:11 UTC
3 points
0
in reply to: Oliver Daniels’s comment on: Daniel Kokotajlo’s Shortform
Correct. There are several other agendas that have this nice property too.

Daniel Kokotajlo 17 Dec 2024 18:14 UTC
14 points
0
in reply to: Daniel Kokotajlo’s comment on: Daniel Kokotajlo’s Shortform
For many cases, the answer will be “Yes, the property can be broken by a red-team specifically fine-tuning the model to break the property.” Then the question becomes “How much optimization effort did they have to put in to achieve this?” and “Is it plausible that the AI we are planning to actually deploy has had a similar or greater amount of optimization accidentally training it to break this property?”

Daniel Kokotajlo 16 Dec 2024 23:41 UTC
2 points
0
in reply to: Raemon’s comment on: My version of Simulacra Levels
I didn’t elaborate, but I did say:
It seems that there is a tendency for discourses primarily operating at Level 1 to devolve into Level 2, and from Level 2 to Level 3, and from Level 3 to Level 4.
It seems maybe you don’t see the Level 2 to Level 3 connection. Well, here’s what I was thinking:

In discourses that are at level 1, people aren’t really forming into stable teams. Think: A bunch of students trying to solve a math problem sheet together, proposing and rejecting various lemmas or intuition pumps. But once the discourse is heavily at level 2, with lots of people thinking hard about how to convince other people of things—with lots of people arguing about some local thing like whether this particular intuition pump is reasonable by thinking about less-local things like whether it would support or undermine Lemma X and thereby support or undermine the strategy so-and-so has been undertaking—well, now it seems like conditions are ripe for teams to start to form. For people to be Team So-And-So’s Strategy. And with the formation of teams comes reporting which team you are on (level 3) and then eventually strategically signalling or shaping perceptions of which team you are on (level 4).

Daniel Kokotajlo 16 Dec 2024 23:31 UTC
2 points
0
in reply to: Zach Stein-Perlman’s comment on: Zach Stein-Perlman’s Shortform
Maybe the fix to the protocol is: Debater copy #1 is told “You go first. Pick an output y, and then argue for it.” Debater copy #2 is then told “You go second. Pick a different, conflicting output y2, and then argue against y and for y2″

Then the debater AI is simply trained on probability-of-winning-the-debate, but the original AI to be deployed is trained on probability-its-output-would-have-been-picked-by-debater-1. (Or, trained to output whatever debater-1 would have picked.)

Daniel Kokotajlo 16 Dec 2024 22:25 UTC
11 points
0
in reply to: Zach Stein-Perlman’s comment on: Zach Stein-Perlman’s Shortform
The hope is that the debaters are incentivized to simply say which output is better, to the best of their ability,

How is this true, if the debaters don’t get to choose which output they are arguing for? Aren’t they instead incentivized to say that whatever output they are assigned is the best?

Daniel Kokotajlo 16 Dec 2024 20:06 UTC
LW: 10 AF: 8
0
AF
on: What a compute-centric framework says about AI takeoff speeds—draft report
The takeoffspeeds.com model Davidson et al worked on is still (unfortunately) the world’s best model of AGI takeoff. I highly encourage people to play around with it, perhaps even to read the research behind it, and I’m glad LessWrong is a place that collects and rewards work like this.