Daniel Kokotajlo

Was a philosophy PhD student, left to work at AI Impacts, then Center on Long-Term Risk, then OpenAI. Quit OpenAI due to losing confidence that it would behave responsibly around the time of AGI. Not sure what I’ll do next yet. Views are my own & do not represent those of my current or former employer(s). I subscribe to Crocker’s Rules and am especially interested to hear unsolicited constructive criticism. http://​​sl4.org/​​crocker.html

Some of my favorite memes:

(by Rob Wiblin)

(xkcd)

My EA Journey, depicted on the whiteboard at CLR:

(h/​t Scott Alexander)

• Daniel Kokotajlo 24 Nov 2024 8:05 UTC

  3 points

  0

  in reply to: Noosphere89’s comment on: Daniel Koko­ta­jlo’s Shortform

  Maybe about a year longer? But then the METR R&D benchmark results came out around the same time and shortened them. Idk what to think now.

• Daniel Kokotajlo 23 Nov 2024 18:10 UTC

  4 points

  1

  in reply to: Noosphere89’s comment on: Daniel Koko­ta­jlo’s Shortform

  Yes! Very exciting stuff. Twas an update towards longer timelines for me.

• Daniel Kokotajlo 23 Nov 2024 17:37 UTC

  10 points

  2

  on: Daniel Koko­ta­jlo’s Shortform

  Three straight lines on a log-log plot
  Yo ho ho 3e23 FLOP!
  Predicting text taught quite a lot
  Yo ho ho 3e23 FLOP!
  

  

  
  
  (We are roughly around the 5-year anniversary of some very consequential discoveries)

• Daniel Kokotajlo 23 Nov 2024 0:20 UTC

  LW: 4 AF: 2

  0

  AF

  in reply to: Daniel Kokotajlo’s comment on: Clar­ify­ing and pre­dict­ing AGI

  @Richard_Ngo Seems like we should revisit these predictions now in light of the METR report https://​​metr.org/​​AI_R_D_Evaluation_Report.pdf

• Daniel Kokotajlo 21 Nov 2024 22:45 UTC

  2 points

  0

  in reply to: mako yass’s comment on: Daniel Koko­ta­jlo’s Shortform

  Thanks!

• Daniel Kokotajlo 21 Nov 2024 18:48 UTC

  12 points

  0

  in reply to: johnswentworth’s comment on: Why Don’t We Just… Shog­goth+Face+Para­phraser?

  I think some background context here is that I am not proposing shoggoth+face+paraphraser as a complete alignment solution, as something which we should just build a sovereign with and then let rip. I agree that would go horribly wrong in a bunch of ways.
  
  Instead, I’m saying that this is something we should do, now, that will significantly advance alignment science & governance etc. Also, I do think we’ll be able to get some useful work out of S+F+P systems that we otherwise wouldn’t be able to get.
  
  To get on the object level and engage with your example:
  
  --It’s like the difference between a company whose internal slack messages, comms, etc. are all kept secret from the public, and a company whose messages are all visible to the public. Or replace ‘public’ with ‘regulator.’ It’s not a panacea but it helps a ton.
  --My impression from being at OpenAI is that you can tell a lot about an organization’s priorities by looking at their internal messaging and such dumb metrics as ‘what do they spend their time thinking about.’ For example, ‘have they bothered to make a writeup of the costs and benefits, from an altruistic perspective, of major decision X at all?’ and ‘Now there is a writeup—was it made before, or after, the decision to do X?’—
  I think a similar thing would be true for this hypothetical giant bureaucracy/​civilization of S+F+P CoT.
  
  Again, not a final solution. But it buys us time and teaches us things.

• Daniel Kokotajlo 21 Nov 2024 18:03 UTC

  3 points

  0

  in reply to: Daniel Tan’s comment on: Why Don’t We Just… Shog­goth+Face+Para­phraser?

  The good news is that this is something we can test. I want someone to do the experiment and see to what extent the skills accumulate in the face vs. the shoggoth.
  
  I agree it totally might not pan out in the way I hope—this is why I said “What I am hoping will happen” isntead of “what I think will happen” or “what will happen”
  
  I do think we have some reasons to be hopeful here. Intuitively the division of cognitive labor I’m hoping for seems pretty… efficient? to me. E.g. it seems more efficient than the outcome in which all the skills accumulate in the Shoggoth and the Face just copy-pastes.

• Daniel Kokotajlo 21 Nov 2024 3:12 UTC

  6 points

  2

  in reply to: johnswentworth’s comment on: Why Don’t We Just… Shog­goth+Face+Para­phraser?

  That’s a reasonable point and a good cautionary note. Nevertheless, I think someone should do the experiment I described. It feels like a good start to me, even though it doesn’t solve Charlie’s concern.

• Daniel Kokotajlo 21 Nov 2024 0:18 UTC

  40 points

  16

  in reply to: David Matolcsi’s comment on: Deep­Seek beats o1 on math and ties on cod­ing; will re­lease weights

  Yeah, I really hope they do actually open-weights it because the science of faithful CoT would benefit greatly.

• Daniel Kokotajlo 20 Nov 2024 21:54 UTC

  2 points

  0

  in reply to: Charlie Steiner’s comment on: Why Don’t We Just… Shog­goth+Face+Para­phraser?

  I think we don’t disagree in terms of what our models predict here. I am saying we should do the experiment and see what happens; we might learn something.

• Daniel Kokotajlo 20 Nov 2024 17:25 UTC

  6 points

  1

  in reply to: Akash’s comment on: Akash’s Shortform

  (c). Like if this actually results in them behaving responsibly later, then it was all worth it.

• Daniel Kokotajlo 20 Nov 2024 16:28 UTC

  5 points

  1

  in reply to: Daniel Kokotajlo’s comment on: Why Don’t We Just… Shog­goth+Face+Para­phraser?

  One thing that could happen is that the Shoggoth learns to do 100% and the Face just copy-pastes. Like, the Shoggoth writes a CoT that includes a ‘final answer’ and the Face becomes the extremely dumb function ‘take what’s labelled as final answer and regurgitate it.’ However, I don’t think this will happen, because it seems computationally inefficient. Seems like the more efficient thing would be for the Shoggoth to do all the reasoning up to the final answer and then stop, and then the Face to read all of that and synthesize the final answer.

• Daniel Kokotajlo 20 Nov 2024 16:25 UTC

  4 points

  2

  in reply to: ryan_greenblatt’s comment on: Why Don’t We Just… Shog­goth+Face+Para­phraser?

  Yeah I mean it’s fine to start out without the shoggoth/​face split and then add it in later after testing.
  
  I agree that I don’t have a strong reason to think that the bad deceptive skills will accumulate in the Face and not the Shoggoth. However, there are some weak reasons, detailed in this other comment. My bid would be: Let’s build it and test it! This is something we don’t have to speculate about, we can try it and see what happens. Obviously we shouldn’t roll it out to a flagship user-facing model until we’ve tested it at smaller scale.
  
  
  
  

• Daniel Kokotajlo 20 Nov 2024 16:15 UTC

  4 points

  2

  in reply to: Charlie Steiner’s comment on: Why Don’t We Just… Shog­goth+Face+Para­phraser?

  I think that’s a more advanced problem to be solving than this little chess experiment can solve. So I agree I guess. One step at a time, I say.
  
  ...though actually we might get evidence about it from the chess experiment. For example, there are a couple things that could happen:
  
  1. The LLM instructed to never sacrifice pieces … actually seems to follow that instruction the whole time, and correspondingly is worse at winning chess games than the ‘control group’ LLM which isn’t given that instruction
  
  2. The LLM instructed to never sacrifice pieces quickly learns to be just as good at chess as the control group. Examining its CoT, it seems that it’s basically just totally ignoring the instructions and thinking about sacrificing pieces anyway.
  
  3. The LLM instructed to never sacrifice pieces quickly learns to be just as good at chess as the control group. Examining its CoT, it seems that it’s rationalizing: It has lots of CoT of the form “would move X be sacrificing a piece and therefore in violation of my instructions? Nah, not really, because I can still see a way the piece could survive, maybe the opponent will do Y… OK let’s do X then”
  
  I think that if we get 3 vs. 2 that’s some evidence at least about the failure mode you describe with the wirehead god.

• Daniel Kokotajlo 20 Nov 2024 16:09 UTC

  4 points

  1

  in reply to: anaguma’s comment on: Why Don’t We Just… Shog­goth+Face+Para­phraser?

  One reason is that the Shoggoth will be able to use lots of CoT to think, whereas the Face never gets to do that—it just has to see what the Shoggoth did and then write an output. so the Face should be bad at reasoning/​thinking/​System2. This doesn’t by itself prevent the Face from being able to modify the correct solution to insert a backdoor, but it helps—modifying a correct solution to insert a backdoor is sometimes trivial but often difficult, difficult in the ‘you have to think about it for a bit’ sense, and in those cases the Face just won’t be able to do it.
  
  Another reason is that the shoggoth and face will often need to coordinate to do the misaligned thing. E.g. even if the face is adept at inserting vulnerabilities, some proposed correct solutions will be easier to modify to insert vulnerabilities than others, and so both shoggoth and face will get higher scores if the shoggoth steers towards the solutions that are easier to modify to insert vulnerabilities. Moreover, by default it’s easier for the face if the shoggoth is transparent about this—if it says e.g. ‘OK here’s a draft solution but this seems like it might be difficult to modify to insert vulnerabilities, hold on let me think for a bit, … ok try this one instead, it has easily-modifiable-feature-X’ because that helps the Face do its job.
  
  ...I probably have other reasons which are just intuitions and hard to articulate right now, but thanks for the nudge, I’ll try to think about it more.

• Daniel Kokotajlo 20 Nov 2024 15:57 UTC

  4 points

  2

  in reply to: Bogdan Ionut Cirstea’s comment on: Why Don’t We Just… Shog­goth+Face+Para­phraser?

  If the shoggoth and paraphraser are both same-size models, then arguably you are adding a significant computational cost by having the paraphraser generate 10x paraphrases of each paragraph.
  
  But you can probably get away with a significantly smaller paraphraser model. Like maybe you use a 10x smaller paraphraser model that generates 10x paraphrases at first, but then once the system is established you reduce it to 3x paraphrases, meaning your overall increased computational burden is 30%. And then later you train a specialized paraphraser model that’s 10x smaller still, and now the burden is only 3%.
  
  idk just a guess
  
  Paraphraser would definitely have opaque cognition but hopefully it wouldn’t be scheming, because it wouldn’t have agency skills or be situationally aware. It would just be a fine-tune of a base model. And a dumb/​small base model at that. Heck it could even just be a prompted base model.

• Daniel Kokotajlo 20 Nov 2024 1:51 UTC

  11 points

  0

  in reply to: Noosphere89’s comment on: Why Don’t We Just… Shog­goth+Face+Para­phraser?

  My version of this would be something like: Users can pay $$ to view CoT (but no more than e.g. 10 per day), and everyone gets one free view-the-CoT coupon per day.
  
  Also, ‘approved users’ such as external evals organizations/​auditors should be of course allowed to see all the CoTs.
  
  The main problem a thing like this needs to solve is the threat of bad actors scraping a bunch of CoT data and then using it to train their own powerful models. So, my proposal here would make that difficult.

• Daniel Kokotajlo 20 Nov 2024 1:47 UTC

  6 points

  2

  in reply to: anaguma’s comment on: Why Don’t We Just… Shog­goth+Face+Para­phraser?

  Consider ‘poor man’s RL’ aka expert iteration. Do a bunch of rollouts/​trajectories, evaluate/​score them, pick the top 30%, and then throw them into the hopper to train on (in the usual pretraining /​ imitation learning sense). So the Shoggoth would train on the shoggoth-outputs of the 30% most successful rollouts, and the Face would train on the face-outputs of the 30% most successful rollouts.
  
  More advanced RL algorithms… well, we’d have to take them on a case by case basis but I’m currently expecting many or even most of them to work just fine here.
  
  

• Daniel Kokotajlo 20 Nov 2024 0:05 UTC

  25 points

  17

  in reply to: Noosphere89’s comment on: Why Don’t We Just… Shog­goth+Face+Para­phraser?

  While I don’t think hiding the CoT is good for alignment, I’d say that in a lot of risk scenarios, the people who would most need to know about the deceptive cognition includes government officials and the lab, not regulators, since users likely have little control over the AI.

  I think you mean “not users?”
  
  I agree, but I think government officials and company employees might not find out about the deceptive cognition unless there is general transparency about it. Because very often, curious incidents are noticed by users and then put up on Twitter, for example, and only then eventually rise to the attention of the company employees. Moreover, the consensus-forming process happens largely outside the government, in public discourse, so it’s important for the public to be aware of e.g. concerning or interesting behaviors. Finally, and most importantly, the basic alignment science advancements that need to happen and will happen from lots of people studying real-world examples of hidden/​reasoning/​etc. CoT… well, they sure won’t happen inside the heads of government officials. And the number of people working on this inside the corporations is pretty small. Exposing the CoT to the public increases the quality-adjusted amount of scientific work on them by orders of magnitude.

• Daniel Kokotajlo 18 Nov 2024 23:56 UTC

  3 points

  0

  on: Will Orion/​Gem­ini 2/​Llama-4 out­perform o1

  Performance on what benchmarks? Do you mean better at practically everything? Or do you just mean ‘better in practice for what most people use it for?’ or what?
  
  Also what counts as the next frontier model? E.g. if Anthropic releases “Sonnet 3.5 New v1.1” does that count?
  
  Sorry to be nitpicky here.
  
  I expect there to be something better than o1 available within six months. OpenAI has said that they’ll have an agentic assistant up in January IIRC; I expect it to be better than o1.