Like, I am quite worried that we will end up with some McCarthy-esque immune reaction to EA people in the US and the UK government where people will be like “wait, what the fuck, how did it happen that this weirdly intense social group with strong shared ideology is now suddenly having such an enormous amount of power in government? Wow, I need to kill this thing with fire, because I don’t even know how to track where it is, or who is involved, so paranoia is really the only option”.
This is looking increasingly prescient.
[Edit to add context]
Not saying this is happening now, but after the board decisions at OpenAI, I could imagine more people taking notice. Hopefully the sentiment then will just be open discourse and acknowledging that there’s now this interesting ideology besides partisan politics and other kinds of lobbying/influence-seeking that are already commonplace. But to get there, I think it’s plausible that EA has some communications- and maybe trust-building work to do.
Just for the record, if the current board thing turns out to be something like a play of power from EAs in AI Safety trying to end up more in control (by e.g. planning to facilitate a merger or a much closer collaboration with Anthropic), and the accusations of lying to the board turn out to be a nothing-burger, then I would consider this a very central example of the kind of political play I was worried would happen (and indeed involved Helen who is one of the top EA DC people).
Correspondingly I assign decently high (20-25%) probability to that indeed being what happened, in which case I would really like the people involved to be held accountable (and for us to please stop the current set of strategies that people are running that give rise to this kind of thing).
As you’d probably agree with, it’s plausible that Sutskever was able to convince the board about specific concerns based on his understanding of the technology (risk levels and timelines) or his day-to-day experience at OpenAI and direct interactions with Sam Altman. If that’s what happened, then it wouldn’t be fair [to say that] any EA-minded board members just acted in an ideology-driven way. (Worth pointing out for people who don’t know this that Sutskever has no ties to EA; it just seems like he shares concerns about the dangers from AI.)
But let’s assume that it comes out that EA board members played a really significant role or were even thinking about something like this before Sutskever brought up concerns. “Play of power” evokes connotations of opportunism and there being no legitimacy for the decision other than that the board thought they could get away with it. This sort of concern you’re describing would worry me a whole lot more if OpenAI had a typical board and corporate structure.
However, since they have a legal structure and mission that emphasizes benefitting humanity as a whole and not shareholders, I’d say situations like the one here are (in theory) exactly why the board was set up that way. The board’s primary task is overseeing the CEO. To achieve OpenAI’s mission, the CEO needs to have the type of personality and thinking habits so he will likely converge toward whatever the best-informed views are about AI risks (and benefits) and how to mitigate (and actualize) them. The CEO shouldn’t be someone who is unlikely to engage in the sort of cognition that one would perform if one cared greatly about long-run outcomes rather than near-term status and took seriously the chance of being wrong about one’s AI risk and timeline assumptions. Regardless of what’s actually true about Altman, it seems like the board came to a negative conclusion about his suitability. In terms of how they made this update, we can envision some different scenarios, some of them would seem unfair to Altman and “ideology-driven” in a sinister way, while others would seem legitimate. (The following scenarios will take for granted that the thing that happened had elements of a “AI safety coup,” as opposed to a “Sutskever coup” or “something else entirely.” Again, I’m not saying that any of this is confirmed; I’m just going with the hypothesis where the EA involvement has the most potential for controversy.) So, here are three variants of how the board could have updated that Altman is not suitable for the mission:
(1) The responsible board members (could just be a subset of the ones that voted against Altman rather than all four of them) never gave him much of a chance. They learned that Altman is less concerned about AI notkilleveryoneism than they would’ve liked, so they took an opportunity to try to oust him. (This is bad because it’s ideology-driven rather than truth-seeking.)
(2) The responsible board members did give Altman a chance initially, but he deceived them in a smoking-gun-type breach of trust.
(3) The responsible board members did gave Altman a chance initially, but they became increasingly disillusioned through a more insincere-vibes-based and gradual erosion of trust, perhaps accompanied by disappointments from empty promises/assurances about, e.g., taking safety testing more seriously for future models, avoiding racing dynamics/avoiding giving out too much info on how to speed up AI through commercialization/rollouts, etc. (I’m only speculating here with the examples I’m giving, but the point is that if the board is unusually active about looking into stuff, it’s conceivable that they maybe-justifiably reached this sort of update even without any smoking-gun-type breach of trust.)
Needless to say, (1) would be very bad board behavior and would put EA in a bad light. (2) would be standard stuff about what boards are there for, but seems somewhat unlikely to have happened here based on the board not being able to easily give more info to the public about what Altman did wrong (as well as the impression I get that they don’t hold much leverage in the negotations now). (3) seems most likely to me and also quite complex to make judgments about the specifics, because lots of things can fall into (3). (3) requires an unusually “active/observant” board. This isn’t necessarily bad. I basically want to flag that I see lots of (3)-type scenarios where the board acted with integrity and courage, but also (admittedly) probably displayed some inexperience by not preparing for the power struggle that results after a decision like this, and by (possibly?) massively mishandling communications, using wording that may perfectly describe what happened when the description is taken literally, but is very misleading when we apply the norms about how parting ways announcements are normally written in very tactful corporate speak. (See also Eliezer’s comment here.) Alternatively, it’s also possible that a (3)-type scenario happened, but the specific incremental updates were uncharitable towards Altman due to being tempted by “staging a coup,” or stuff like that. It gets messy when you have to evaluate someone’s leadership fit where they have a bunch of uncontested talents but also some orange flags and you have decide what sort of strengths or weaknesses are most essential for the mission.
For me the key variable is whether they took a decision that would have put someone substantially socially closer to them in charge, with some veneer of safety motivation, but where the ultimate variance in their decision would counterfactually be driven by social proximity and pre-existing alliances.
A concrete instance of this would be if the plan with the firing was to facilitate some merge with Anthropic, or to promote someone like Dario to the new CEO position, who the board members (which were chosen by Holden) have a much tighter relationship to.
My current model is that Holden chose them. Tasha in 2018, Helen in 2021 when he left and chose Helen as his successor board member.
I don’t know, but I think it was close to a unilateral decision from his side (like I don’t think anyone at Open AI had much reason to trust Helen outside of Holden’s endorsement, so my guess is he had a lot of leeway).
Thanks! And why did Holden have the ability to choose board members (and be on the board in the first place)?
I remember hearing that this was in exchange for OP investment into OpenAI, but I also remember Dustin claiming that OpenAI didn’t actually need any OP money (would’ve just gotten the money easily from another investor).
Is your model essentially that the OpenAI folks just got along with Holden and thought he/OP were reasonable, or is there a different reason Holden ended up having so much influence over the board?
My model is that this was a mixture of a reputational trade (OpenAI got to hire a bunch of talent from EA spaces and make themselves look responsible to the world) and just actually financial ($30M was a substantial amount of money at that point in time).
Sam Altman has many times said he quite respects Holden, so that made up a large fraction of the variance. See e.g. this tweet:
(i used to be annoyed at being the villain of the EAs until i met their heroes*, and now i’m lowkey proud of it
*there are a few EA heroes i think are really great, eg Holden)
[...] reputational trade (OpenAI got to hire a bunch of talent from EA spaces and make themselves look responsible to the world) [...]
Yes, I think “reputational trade,” i.e., something that’s beneficial for both parties, is an important part of the story that the media hasn’t really picked up on. EAs were focused on the dangers and benefits from AI way before anyone else, so it carries quite some weight when EA opinion leaders put an implicit seal of approval on the new AI company.
There’s a tension between (1) previously having held back on natural-seeming criticism of OpenAI (“putting the world at risk for profits” or “they plan on wielding this immense power of building god/single-handedly starting something bigger than the next Industrial Revolution/making all jobs obsolete and solving all major problems”) because they have the seal of approval from this public good, non-profit, beneficial-mission-focused board structure,
and
(2) being outraged when this board structure does something that it was arguably intended to do (at least under some circumstances).
(Of course, the specifics of how and why things happened matter a lot, and maybe most people aren’t outraged because the board did something, but rather because of how they did it or based on skepticism about reasons and justifications. On those later points, I sympathize more with people who are outraged or concerned that something didn’t go right. But we don’t know all the details yet.)
(Of course, the specifics of how and why things happened matter a lot, and maybe most people aren’t outraged because the board did something, but rather because of how they did it or based on skepticism about reasons and justifications. On those later points, I sympathize more with people who are outraged or concerned that something didn’t go right. But we don’t know all the details yet.)
Almost all the outrage I am seeing is about how this firing was conducted. I think if the board had a proper report ready that outlined why they think OpenAI was acting recklessly, and if they had properly consulted with relevant stakeholders before doing this, I think the public reaction would be very different.
I agree there are also some random people on the internet who are angry about the board taking any action even though the company is going well in financial terms, but most of the well-informed and reasonably people I’ve seen are concerned about the way this was rushed and how the initial post seemed to pretty clearly imply that Sam had done some pretty serious deception, without anything to back that up with.
FWIW, I think it’s likely that they thought about this decision for quite some time and systematically – I mean the initial announcement did mention something about a “deliberative review process by the board.” But yeah, we don’t get to see any of what they thought about or who (if anyone) they consulted for gathering further evidence or for verifying claims by Sutskever. Unfortunately, we don’t know yet. And I concede that given the little info we have, it takes charitable priors to end up with “my view.” (I put it in quotation marks because it’s not like I have more than 50% confidence in it. Mostly, I want to flag that this view is still very much on the table.)
Also, on the part about “imply that Sam had done some pretty serious deception, without anything to back that up with.” I’m >75% that either Eliezer nailed it in this tweet, or they actually have evidence about something pretty serious but decided not to disclose it for reasons that have to do with the nature of the thing that happened. (I guess the third option is they self-deceived into thinking their reasons to fire Altman will seem serious/compelling [or at least defensible] to everyone to whom they give more info, when in fact the reasoning is more subtle/subjective/depends on additional assumptions that many others wouldn’t share. This could then have become apparent to them when they had to explain their reasoning to OpenAI staff later on, and they aborted the attempt in the middle of it when they noticed it wasn’t hitting well, leaving the other party confused. I don’t think that would necessarily imply anything bad about the board members’ character, though it is worth noting that if someone self-deceives in that way too strongly or too often, it makes for a common malefactor pattern, and obviously it wouldn’t reflect well on their judgment in this specific instance. One reason I consider this hypothesis less likely than the others is because it’s rare for several people – the four board members – to all make the same mistake about whether their reasoning will seem compelling to others, and for none of them to realize that it’s better to err on the side of caution and instead say something like “we noticed we have strong differences in vision with Sam Altman,” or something like that.)
My current model is that this is unlikely to have been planned long in-advance. For example, for unrelated reasons I was planning to have a call with Helen last week, and she proposed a meeting time of last Thursday (when I responded with my availability for Thursday early in the week, she did not respond). She did then not actually schedule the final meeting time and didn’t respond to my last email, but this makes me think that at least early in the week, she did not expect to be busy on Thursday.
There are also some other people who I feel like I would expect to know about this if it had been planned who have been expressing their confusion and bafflement at what is going on on Twitter and various Slacks I am in. I think if this was planned, it was planned as a background thing, and then came to a head suddenly, with maybe 1-2 days notice, but it doesn’t seem like more.
This is looking increasingly prescient.
[Edit to add context]
Not saying this is happening now, but after the board decisions at OpenAI, I could imagine more people taking notice. Hopefully the sentiment then will just be open discourse and acknowledging that there’s now this interesting ideology besides partisan politics and other kinds of lobbying/influence-seeking that are already commonplace. But to get there, I think it’s plausible that EA has some communications- and maybe trust-building work to do.
Just for the record, if the current board thing turns out to be something like a play of power from EAs in AI Safety trying to end up more in control (by e.g. planning to facilitate a merger or a much closer collaboration with Anthropic), and the accusations of lying to the board turn out to be a nothing-burger, then I would consider this a very central example of the kind of political play I was worried would happen (and indeed involved Helen who is one of the top EA DC people).
Correspondingly I assign decently high (20-25%) probability to that indeed being what happened, in which case I would really like the people involved to be held accountable (and for us to please stop the current set of strategies that people are running that give rise to this kind of thing).
As you’d probably agree with, it’s plausible that Sutskever was able to convince the board about specific concerns based on his understanding of the technology (risk levels and timelines) or his day-to-day experience at OpenAI and direct interactions with Sam Altman. If that’s what happened, then it wouldn’t be fair [to say that] any EA-minded board members just acted in an ideology-driven way. (Worth pointing out for people who don’t know this that Sutskever has no ties to EA; it just seems like he shares concerns about the dangers from AI.)
But let’s assume that it comes out that EA board members played a really significant role or were even thinking about something like this before Sutskever brought up concerns. “Play of power” evokes connotations of opportunism and there being no legitimacy for the decision other than that the board thought they could get away with it. This sort of concern you’re describing would worry me a whole lot more if OpenAI had a typical board and corporate structure.
However, since they have a legal structure and mission that emphasizes benefitting humanity as a whole and not shareholders, I’d say situations like the one here are (in theory) exactly why the board was set up that way. The board’s primary task is overseeing the CEO. To achieve OpenAI’s mission, the CEO needs to have the type of personality and thinking habits so he will likely converge toward whatever the best-informed views are about AI risks (and benefits) and how to mitigate (and actualize) them. The CEO shouldn’t be someone who is unlikely to engage in the sort of cognition that one would perform if one cared greatly about long-run outcomes rather than near-term status and took seriously the chance of being wrong about one’s AI risk and timeline assumptions. Regardless of what’s actually true about Altman, it seems like the board came to a negative conclusion about his suitability. In terms of how they made this update, we can envision some different scenarios, some of them would seem unfair to Altman and “ideology-driven” in a sinister way, while others would seem legitimate. (The following scenarios will take for granted that the thing that happened had elements of a “AI safety coup,” as opposed to a “Sutskever coup” or “something else entirely.” Again, I’m not saying that any of this is confirmed; I’m just going with the hypothesis where the EA involvement has the most potential for controversy.) So, here are three variants of how the board could have updated that Altman is not suitable for the mission:
(1) The responsible board members (could just be a subset of the ones that voted against Altman rather than all four of them) never gave him much of a chance. They learned that Altman is less concerned about AI notkilleveryoneism than they would’ve liked, so they took an opportunity to try to oust him. (This is bad because it’s ideology-driven rather than truth-seeking.)
(2) The responsible board members did give Altman a chance initially, but he deceived them in a smoking-gun-type breach of trust.
(3) The responsible board members did gave Altman a chance initially, but they became increasingly disillusioned through a more insincere-vibes-based and gradual erosion of trust, perhaps accompanied by disappointments from empty promises/assurances about, e.g., taking safety testing more seriously for future models, avoiding racing dynamics/avoiding giving out too much info on how to speed up AI through commercialization/rollouts, etc. (I’m only speculating here with the examples I’m giving, but the point is that if the board is unusually active about looking into stuff, it’s conceivable that they maybe-justifiably reached this sort of update even without any smoking-gun-type breach of trust.)
Needless to say, (1) would be very bad board behavior and would put EA in a bad light. (2) would be standard stuff about what boards are there for, but seems somewhat unlikely to have happened here based on the board not being able to easily give more info to the public about what Altman did wrong (as well as the impression I get that they don’t hold much leverage in the negotations now). (3) seems most likely to me and also quite complex to make judgments about the specifics, because lots of things can fall into (3). (3) requires an unusually “active/observant” board. This isn’t necessarily bad. I basically want to flag that I see lots of (3)-type scenarios where the board acted with integrity and courage, but also (admittedly) probably displayed some inexperience by not preparing for the power struggle that results after a decision like this, and by (possibly?) massively mishandling communications, using wording that may perfectly describe what happened when the description is taken literally, but is very misleading when we apply the norms about how parting ways announcements are normally written in very tactful corporate speak. (See also Eliezer’s comment here.) Alternatively, it’s also possible that a (3)-type scenario happened, but the specific incremental updates were uncharitable towards Altman due to being tempted by “staging a coup,” or stuff like that. It gets messy when you have to evaluate someone’s leadership fit where they have a bunch of uncontested talents but also some orange flags and you have decide what sort of strengths or weaknesses are most essential for the mission.
For me the key variable is whether they took a decision that would have put someone substantially socially closer to them in charge, with some veneer of safety motivation, but where the ultimate variance in their decision would counterfactually be driven by social proximity and pre-existing alliances.
A concrete instance of this would be if the plan with the firing was to facilitate some merge with Anthropic, or to promote someone like Dario to the new CEO position, who the board members (which were chosen by Holden) have a much tighter relationship to.
Clarification/history question: How were these board members chosen?
My current model is that Holden chose them. Tasha in 2018, Helen in 2021 when he left and chose Helen as his successor board member.
I don’t know, but I think it was close to a unilateral decision from his side (like I don’t think anyone at Open AI had much reason to trust Helen outside of Holden’s endorsement, so my guess is he had a lot of leeway).
Thanks! And why did Holden have the ability to choose board members (and be on the board in the first place)?
I remember hearing that this was in exchange for OP investment into OpenAI, but I also remember Dustin claiming that OpenAI didn’t actually need any OP money (would’ve just gotten the money easily from another investor).
Is your model essentially that the OpenAI folks just got along with Holden and thought he/OP were reasonable, or is there a different reason Holden ended up having so much influence over the board?
My model is that this was a mixture of a reputational trade (OpenAI got to hire a bunch of talent from EA spaces and make themselves look responsible to the world) and just actually financial ($30M was a substantial amount of money at that point in time).
Sam Altman has many times said he quite respects Holden, so that made up a large fraction of the variance. See e.g. this tweet:
Yes, I think “reputational trade,” i.e., something that’s beneficial for both parties, is an important part of the story that the media hasn’t really picked up on. EAs were focused on the dangers and benefits from AI way before anyone else, so it carries quite some weight when EA opinion leaders put an implicit seal of approval on the new AI company.
There’s a tension between
(1) previously having held back on natural-seeming criticism of OpenAI (“putting the world at risk for profits” or “they plan on wielding this immense power of building god/single-handedly starting something bigger than the next Industrial Revolution/making all jobs obsolete and solving all major problems”) because they have the seal of approval from this public good, non-profit, beneficial-mission-focused board structure,
and
(2) being outraged when this board structure does something that it was arguably intended to do (at least under some circumstances).
(Of course, the specifics of how and why things happened matter a lot, and maybe most people aren’t outraged because the board did something, but rather because of how they did it or based on skepticism about reasons and justifications. On those later points, I sympathize more with people who are outraged or concerned that something didn’t go right. But we don’t know all the details yet.)
Almost all the outrage I am seeing is about how this firing was conducted. I think if the board had a proper report ready that outlined why they think OpenAI was acting recklessly, and if they had properly consulted with relevant stakeholders before doing this, I think the public reaction would be very different.
I agree there are also some random people on the internet who are angry about the board taking any action even though the company is going well in financial terms, but most of the well-informed and reasonably people I’ve seen are concerned about the way this was rushed and how the initial post seemed to pretty clearly imply that Sam had done some pretty serious deception, without anything to back that up with.
Okay, that’s fair.
FWIW, I think it’s likely that they thought about this decision for quite some time and systematically – I mean the initial announcement did mention something about a “deliberative review process by the board.” But yeah, we don’t get to see any of what they thought about or who (if anyone) they consulted for gathering further evidence or for verifying claims by Sutskever. Unfortunately, we don’t know yet. And I concede that given the little info we have, it takes charitable priors to end up with “my view.” (I put it in quotation marks because it’s not like I have more than 50% confidence in it. Mostly, I want to flag that this view is still very much on the table.)
Also, on the part about “imply that Sam had done some pretty serious deception, without anything to back that up with.” I’m >75% that either Eliezer nailed it in this tweet, or they actually have evidence about something pretty serious but decided not to disclose it for reasons that have to do with the nature of the thing that happened. (I guess the third option is they self-deceived into thinking their reasons to fire Altman will seem serious/compelling [or at least defensible] to everyone to whom they give more info, when in fact the reasoning is more subtle/subjective/depends on additional assumptions that many others wouldn’t share. This could then have become apparent to them when they had to explain their reasoning to OpenAI staff later on, and they aborted the attempt in the middle of it when they noticed it wasn’t hitting well, leaving the other party confused. I don’t think that would necessarily imply anything bad about the board members’ character, though it is worth noting that if someone self-deceives in that way too strongly or too often, it makes for a common malefactor pattern, and obviously it wouldn’t reflect well on their judgment in this specific instance. One reason I consider this hypothesis less likely than the others is because it’s rare for several people – the four board members – to all make the same mistake about whether their reasoning will seem compelling to others, and for none of them to realize that it’s better to err on the side of caution and instead say something like “we noticed we have strong differences in vision with Sam Altman,” or something like that.)
My current model is that this is unlikely to have been planned long in-advance. For example, for unrelated reasons I was planning to have a call with Helen last week, and she proposed a meeting time of last Thursday (when I responded with my availability for Thursday early in the week, she did not respond). She did then not actually schedule the final meeting time and didn’t respond to my last email, but this makes me think that at least early in the week, she did not expect to be busy on Thursday.
There are also some other people who I feel like I would expect to know about this if it had been planned who have been expressing their confusion and bafflement at what is going on on Twitter and various Slacks I am in. I think if this was planned, it was planned as a background thing, and then came to a head suddenly, with maybe 1-2 days notice, but it doesn’t seem like more.