Eliezer Yudkowsky writes on twitter:
Nothing else Elon Musk has done can possibly make up for how hard the “OpenAI” launch trashed humanity’s chances of survival; previously there was a nascent spirit of cooperation, which Elon completely blew up to try to make it all be about who, which monkey, got the poison banana, and by spreading and advocating the frame that everybody needed their own “demon” (Musk’s old term) in their house, and anybody who talked about reducing proliferation of demons must be a bad anti-openness person who wanted to keep all the demons for themselves.
Nobody involved with OpenAI’s launch can reasonably have been said to have done anything else of relative importance in their lives. The net impact of their lives is their contribution to the huge negative impact of OpenAI’s launch, plus a rounding error.
Previously all the AGI people were at the same conference talking about how humanity was going to handle this together. Elon Musk didn’t like Demis Hassabis, so he blew that up. That’s the impact of his life. The end.
I’ve found myself repeatedly uncertain about what to make of OpenAI and their impact. The most recent LessWrong discussion that I’m aware of has happened on the post Will OpenAI-s work unintentionally increase existential risk, but the arguments for negative impact are different from the thing Eliezer named.
I’m also not entirely sure whether publicly debating sensitive questions like whether a person or organization accidentally increased existential risk is a good idea in the first place. However, spontaneously bringing up the issue on a twitter thread is unlikely to be optimal. At the very least, it should be beneficial to discuss the meta question, i.e., how we should or shouldn’t talk about this. With that in mind, here are three things I would like to understand better:
Concretely speaking, should we be hesitant to talk about this? If so, what kind of discussions are okay?
And—conditional on discussing them being a good idea:
-
What is the more detailed story of how the “nascent spirit of cooperation” has degraded or changed since the inception of OpenAI?
-
What interventions are possible here, if any? (Is it really that difficult to organize some kind of outreach to Elon to try and reverse some of the effects? Naively speaking, my impression has been that our community is sufficiently well connected to do this, and that Elon is amenable to arguments.)
I’m less interested in estimating the total impact of any specific person.
I think we should not be hesitant to talk about this in public. I used to be of the opposite opinion, believing-as-if there was a benevolent conspiracy that figured out which conversations could/couldn’t nudge AI politics in useful ways, whose upsides were more important than the upsides of LWers/etc. knowing what’s up. I now both believe less in such a conspiracy, and believe more that we need public fora in which to reason because we do not have functional private fora with memory (in the way that a LW comment thread has memory) that span across organizations.
It’s possible I’m still missing something, but if so it would be nice to have it spelled out publicly what exactly I am missing.
I agree with Lincoln Quirk’s comment that things could turn into a kind of culture war, and that this would be harmful. It seems to me it’s worth responding to this by trying unusually hard (on this or other easily politicizable topics) to avoid treating arguments like soldiers. But it doesn’t seem worthwhile to me to refrain from honest attempts to think in public.
Ineffective, because the people arguing on the forum are lacking knowledge about the situation. They don’t understand OpenAI’s incentive structure, plan, etc. Thus any plans they put forward will be in all likelihood useless to OpenAI.
Risky, because (some combination of):
it is emotionally difficult to hear that one of your friends is plotting against you (and openAI is made up of humans, many of whom came out of this community)
it’s especially hard if your friend is misinformed and plotting against you; and I think it likely that the openAI people believe that Yudkowsky/LW commentators are misinformed or at least under-informed (and they are probably right about this)
to manage that emotional situation, you may want to declare war back on them, cut off contact, etc.; any of these actions if declared as an internal policy would be damaging to the future relationship between openAI and the LW world
openAI has already had a ton of PR issues over the last few years and so they probably have a pretty well developed muscle for dealing internally with bad PR, which this would fall under. If true, the muscle probably looks like internal announcements with messages like “ignore those people/stop listening to them, they don’t understand what we do, we’re managing all these concerns and those people are over indexing on them anyway”
the evaporative cooling effect may eject some people who were already on the fence about leaving, but the people who remain will be more committed to the original mission, more “anti LW” and less inclined to listen to us in the future
hearing bad arguments makes one more resistant to similar (but better) arguments in the future
I want to state for the record that I think OpenAI is sincerely trying to make the world a better place, and I appreciate their efforts. I don’t have a settled opinion on the sign of their impact so far.
What should be done instead of a public forum? I don’t necessarily think there needs to be a “conspiracy”, but I do think that it’s a heck of a lot better to have one-on-one meetings with people to convince them of things. At my company, when sensitive things need to be decided or acted on, a bunch of slack DMs fly around until one person is clearly the owner of the problem; they end up in charge of having the necessary private conversations (and keeping stakeholders in the loop). Could this work with LW and OpenAI? I’m not sure.
What’s standing in the way of these being created?
Mostly time and attention. This has been on the list of things the LessWrong team has considered working on and there’s just a lot of competing priorities.
Hmm, I was imagining that in Anna’s view, it’s not just about what concrete social media or other venues exist, but about some social dynamic that makes even the informal benevolent conspiracy part impossible or undesirable.
How open do we think OpenAI would be to additional research showing the dangers of AGI? If OpenAI is pursuing a perilous course, perhaps this community should prioritize doing the kind of research that would persuade them to slow down. Sam Altman came across to me at the two SSC talks he gave as being highly rational as this community would define rational.
If this is the correct path, we would benefit from people who have worked at OpenAI explaining what kind of evidence would be needed to influence them towards Eliezer’s view of AGI.
I don’t know what ‘we’ think, but as a person somewhat familiar with OpenAI employees and research output, they are definitely willing to pursue safety and transparency research that’s relevant to existential risk, and I don’t really know how one could do that without opening oneself up to producing research that provides evidence of AI danger.
I just saw this, but this feels like a better-late-than-never situation. I think hard conversations about the possibilities of increasing existential risk should happen.
I work at OpenAI. I have worked at OpenAI for over five years.
I think we definitely should be willing and able to have these sorts of conversations in public, mostly for the reasons other people have listed. I think AnnaSalamon is the answer I agree with most.
I want to also add this has made me deeply uncomfortable and anxious countless times over the past few years. It can be a difficult thing to navigate well or navigate efficiently. I feel like I’ve gotten better at it, and better at knowing/managing myself. I see newer colleagues also suffering from this. I try to help them when I can.
I’m not sure this is the right answer for all context, but I am optimistic for this one. I’ve found the rationality community and the lesswrong community to be much better than average at dealing with bad faith arguments, and for cutting towards the truth. I think there are communities where it would go poorly enough that it could be net-negative to have the conversation.
Side note: I really don’t have a lot of context about the Elon Musk connection, and the guy has not really been involved for years. I think the “what things (including things OpenAI is doing) might increase existential risk” is an important conversation to have when analyzing forecasts, predictions, and research plans. I am less optimistic about “what tech executives think about other tech executives” going well.
I’d like to put in my vote for “this should not be discussed in public forums”. Whatever is happening, the public forum debate will have no impact on it; but it does create the circumstances for a culture war that seems quite bad.
I think this is wrong. I think a lot of people who care about AI Alignment read LessWrong and might change their relationship to Open AI depending on what is said here.
I disagree with Lincoln’s comment, but I’m confused that when I read it just now it was at −2; it seems like a substantive comment/opinion that deserves to be heard and part of the conversation.
If comments expressing some folks’ actual point of view are downvoted below the visibility threshold, it’ll be hard to have good substantive conversation.
I’m skeptical of OpenAI’s net impact on the spirit of cooperation because I’m skeptical about the counterfactual prospects of cooperation in the last 6 years had OpenAI not been founded.
The 2000s and early 2010s centralized and intermediated a lot of stuff online, where we trusted centralized parties to be neutral arbiters. We are now experiencing the after effects of that naivete, where Reddit, Twitter and Facebook are censoring certain parties on social media, and otherwise neutral infrastructure like AWS or Cloudflare kick off disfavored parties. I am at the point where I am scared of centralized infrastructure and try to minimize my reliance on it because it allows third parties to apply pressure on me.
These general trends are much larger than Elon Musk and would have happened without him. I’m uncertain to what extent Elon was just reacting to this trend and was ahead of the curve.
Darkly humorous, but OpenAI’s recent destruction of otherwise entirely benign users of GPT-3 is teaching people to not rely on centralized AI. Now that OpenAI is large, the Blue Egregore can apply media pressure on the tech company to extract submission, in this case censorship demands. The downstream effects of this demand to think of the non-existent children was the destruction of trust in AI Dungeon leading people to mostly switch to various GPT-J alternatives (including local setups) and the shutdown of the Samantha, which Mr. Rohrer is taking hard.
Releasing GPT-3 non-trivially increased the odds of doomsday. So yeah they are not good actors.
Can you elaborate on that? It seems non-obvious; I feel like I could tell a different story
… like, OpenAI has much fewer resources than Deepmind, so if Deepmind released GPT-3 first, and if it is the case that scaling up a language model far enough gives you AGI, there would have been less time in between knowing what AGI will look like and having AGI.