I’m confused. Aren’t the organizations pushing hardest to build AGI (OpenAI, DeepMind, Anthropic) led or founded by people who were among the earliest to be convinced of AI risk? Ah, found some additional context in the Twitter thread.
Kerry Vaughan:
That’s really interesting!
Take OpenAI as an example, since conversations about AI X-risk seemed pretty foundational to its creation, I would have thought this was relatively widely discussed there.
Is that just not the case?
Michael Page:
[Add necessary caveats, disclaimers, etc.] In short, correct.
So, apparently the people who started OpenAI to reduce AI risk, failed to make sure that the people they hired to build AGI are actually convinced of AI risk?!
I could imagine that OpenAI getting top talent to ensure their level of research achievements while also filtering people they hire by their seriousness about reducing civilization-level risks is too hard. Or at least it could easily have been infeasible 4 years ago.
I know a couple of people at DeepMind and none of them have reducing civilization-level risks as one of their primary motivations for working there, as I believe is the case with most of DeepMind.
The original stated rationale behind OpenAI was https://www.wired.com/2015/12/how-elon-musk-and-y-combinator-plan-to-stop-computers-from-taking-over/. I gather that another big rationale behind OpenAI was ‘Elon was scared of Demis doing bad things with AGI’; and another big rationale was ‘Sam was already interested in doing cool stuff with AI regardless of the whole AI-risk thing’. (Let me know if you think any of this summary is misleading or wrong.)
Since then, Elon has left, and Sam and various other individuals at OpenAI seem to have improved their models of AGI risk a lot. But this does seem like a very different situation than ‘founding an org based on an at-all good understanding of AI risk, filtering for staff based on such an understanding, etc.’
UPDATE: I did mean this comment as a genuine question, not a rhetorical one. I’m actually curious about what underlies these perceptions of OpenAI. Feel free to private message me if you’d prefer not to reply in public. Another possible reason that occurred to me after posting this comment is that OpenAI was the first serious competitor to DeepMind in pursuing AGI, so people may resent them making AGI research more competitive and hence arguably creating pressure to rush decisions and cut corners on safety.
Why do so many people on LessWrong say that OpenAI doesn’t prioritize AI x-risk?
I’m not saying that they definitely do. But since they were founded with AI risks in mind and have had some great safety researchers over the years, I am curious which events lead people to frequently say that they are not prioritizing it. I know a lot of their safety researchers left at one point, which is a bad sign about their safety prioritization, but not conclusive.
I get the sense a lot of people were upset when they changed from a non-profit to a capped-profit structure. But that doesn’t say anything directly about how they prioritize x-risk, and the fact that they didn’t go to a 100% for-profit structure shows they are at least prioritizing misuse risks.
OpenAI has had at least a few safety-related publications since 2020:
I’m confused. Aren’t the organizations pushing hardest to build AGI (OpenAI, DeepMind, Anthropic) led or founded by people who were among the earliest to be convinced of AI risk? Ah, found some additional context in the Twitter thread.
Kerry Vaughan:
Michael Page:
So, apparently the people who started OpenAI to reduce AI risk, failed to make sure that the people they hired to build AGI are actually convinced of AI risk?!
I could imagine that OpenAI getting top talent to ensure their level of research achievements while also filtering people they hire by their seriousness about reducing civilization-level risks is too hard. Or at least it could easily have been infeasible 4 years ago.
I know a couple of people at DeepMind and none of them have reducing civilization-level risks as one of their primary motivations for working there, as I believe is the case with most of DeepMind.
The original stated rationale behind OpenAI was https://www.wired.com/2015/12/how-elon-musk-and-y-combinator-plan-to-stop-computers-from-taking-over/. I gather that another big rationale behind OpenAI was ‘Elon was scared of Demis doing bad things with AGI’; and another big rationale was ‘Sam was already interested in doing cool stuff with AI regardless of the whole AI-risk thing’. (Let me know if you think any of this summary is misleading or wrong.)
Since then, Elon has left, and Sam and various other individuals at OpenAI seem to have improved their models of AGI risk a lot. But this does seem like a very different situation than ‘founding an org based on an at-all good understanding of AI risk, filtering for staff based on such an understanding, etc.’
This link is dead for me. I found this link that points to the same article.
Thanks! Edited.
I think one thing is that they had a different idea of what AI risk entails.
What idea would that be, misuse risks rather than misalignment risks?
UPDATE: I did mean this comment as a genuine question, not a rhetorical one. I’m actually curious about what underlies these perceptions of OpenAI. Feel free to private message me if you’d prefer not to reply in public. Another possible reason that occurred to me after posting this comment is that OpenAI was the first serious competitor to DeepMind in pursuing AGI, so people may resent them making AGI research more competitive and hence arguably creating pressure to rush decisions and cut corners on safety.
Why do so many people on LessWrong say that OpenAI doesn’t prioritize AI x-risk?
I’m not saying that they definitely do. But since they were founded with AI risks in mind and have had some great safety researchers over the years, I am curious which events lead people to frequently say that they are not prioritizing it. I know a lot of their safety researchers left at one point, which is a bad sign about their safety prioritization, but not conclusive.
I get the sense a lot of people were upset when they changed from a non-profit to a capped-profit structure. But that doesn’t say anything directly about how they prioritize x-risk, and the fact that they didn’t go to a 100% for-profit structure shows they are at least prioritizing misuse risks.
OpenAI has had at least a few safety-related publications since 2020:
https://openai.com/blog/language-model-safety-and-misuse/
https://openai.com/blog/summarizing-books/
https://openai.com/blog/learning-to-summarize-with-human-feedback/
https://openai.com/blog/microscope/