Meta: I think these kinds of posts should include some sort of disclaimer acknowledging that you are an OpenAI employee & also mentioning whether or not the post was reviewed by OpenAI staff, OpenAI comms, etc.
I imagine you didn’t do this because many people who read this forum are aware of this fact (and it’s on your profile– it’s not like you’re trying to hide it), but I suspect this information could be useful for newcomers who are engaging with this kind of material.
Yeah, this omission felt pretty glaring to me. OpenAI is explicitly aiming to build “the most powerful technology humanity has yet invented.” Obviously that doesn’t mean Richard is wrong that the AI safety community is too power-seeking, but I would sure have appreciated him acknowledging/grappling with the fact that the company he works for is seeking to obtain more power than any group of people in history by a gigantic margin.
An elephant in the room (IMO) is that moving forward, OpenAI probably benefits from a world in which the AI safety community does not have much influence.
There’s a fine line between “play nice with others and be more cooperative” and “don’t actually advocate for policies that you think would help the world, and only do things that the Big Companies and Their Allies are comfortable with.”
Again, I don’t think Richard sat in his room and thought “how do I spread a meme that is good for my company.” I think he’s genuinely saying what he believes and giving advice that he thinks will be useful to the AI safety community and improve society’s future .
But I also think that one of the reasons why Richard still works at OpenAI is because he’s the kind of agent who genuinely believes things that tend to be pretty aligned with OpenAI’s interests, and I suspect his perspective is informed by having lots of friends/colleagues at OpenAI.
Someone who works for a tobacco company can still have genuinely useful advice for the community of people concerned about the health effects of smoking. But I still think it’s an important epistemic norm that they add (at least) a brief disclaimer acknowledging that they work for a tobacco company.
(And the case becomes even stronger in the event that they have to get approval from the tobacco company comms team, or they filter out any ideas that they have that could get them in trouble with the company. Or perhaps before writing/publishing a post they consider the fact that other people have been fired from their company for sharing information that was against company interests, that the CEO attempted to remove a board member under the justification that she published a paper that went against company interests, that the company has previously used history of using highly restrive NDAs to prevent people from saying things that go against company interests.)
But I also think that one of the reasons why Richard still works at OpenAI is because he’s the kind of agent who genuinely believes things that tend to be pretty aligned with OpenAI’s interests, and I suspect his perspective is informed by having lots of friends/colleagues at OpenAI.
Added a disclaimer, as suggested. It seems like a good practice for this sort of post. Though note that I disagree with this paragraph; I don’t think “being the kind of agent who X” or “being informed by many people at Y” are good reasons to give disclaimers. Whereas I do buy that “they filter out any ideas that they have that could get them in trouble with the company” is an important (conscious or unconscious) effect, and worth a disclaimer.
I’ve also added this note to the text:
Note that most big companies (especially AGI companies) are strongly structurally power-seeking too, and this is a big reason why society at large is so skeptical of and hostile to them. I focused on AI safety in this post both because companies being power-seeking is an idea that’s mostly “priced in”, and because I think that these ideas are still useful even when dealing with other power-seeking actors.
I appreciate you adding the note, though I do think the situation is far more unusual than described. I agree it’s widely priced in that companies in general seek power, but I think probably less so that the author of this post personally works for a company which is attempting to acquire drastically more power than any other company ever, and that much of the behavior the post describes as power-seeking amounts to “people trying to stop the author and his colleagues from attempting that.”
(I think “being the kind of agent who survives the selection process” can sometimes be an important epistemic thing to consider, though mostly when thinking about how systems work and what kinds of people/views those systems promote. Agreed that “being informed by many people who Y” is a rather weak one & certainly would not on its own warrant a disclosure.)
I think “being the kind of agent who survives the selection process” can sometimes be an important epistemic thing to consider
I’m not claiming it’s zero information, but there are lots of things that convey non-zero information which it’d be bad to set disclosure norms based on. E.g. “I’ve only ever worked at nonprofits” should definitely affect your opinion of someone’s epistemics (e.g. when they’re trying to evaluate corporate dynamics) but once we start getting people to disclose that sort of thing there’s no clear stopping point. So mostly I want the line to be “current relevant conflicts of interest”.
My take atm is “seems right that this shouldn’t be a permanent norm, there are definitely costs of disclaimer-ratcheting that are pretty bad. I think it might still be the right thing to do of your own accord in some cases, which is, like, superogetory.”
I think there’s maybe a weird thing with this post, where, it’s trying to be the timeless, abstract version of itself. It’s certainly easier to write the timeless abstract version than the “digging into specific examples and calling people out” version. But, I think the digging into specific examples is actually kind of important here – it’s easy to come away with vague takeaways that disagree, where everyone nods along but then mostly thinks it’s Those Other Guys who are being power seeking.
Given that it’s probably 10-50x harder to write the Post With Specific Examples, I think actually a pretty okay outcome is “ship the vague post, and let discussion in the comments get into the inside-baseball-details.” And, then, it’d be remiss for the post-author’s role in the ecosystem not coming up as an example to dig into.
Meta: I think these kinds of posts should include some sort of disclaimer acknowledging that you are an OpenAI employee & also mentioning whether or not the post was reviewed by OpenAI staff, OpenAI comms, etc.
I imagine you didn’t do this because many people who read this forum are aware of this fact (and it’s on your profile– it’s not like you’re trying to hide it), but I suspect this information could be useful for newcomers who are engaging with this kind of material.
Yeah, this omission felt pretty glaring to me. OpenAI is explicitly aiming to build “the most powerful technology humanity has yet invented.” Obviously that doesn’t mean Richard is wrong that the AI safety community is too power-seeking, but I would sure have appreciated him acknowledging/grappling with the fact that the company he works for is seeking to obtain more power than any group of people in history by a gigantic margin.
An elephant in the room (IMO) is that moving forward, OpenAI probably benefits from a world in which the AI safety community does not have much influence.
There’s a fine line between “play nice with others and be more cooperative” and “don’t actually advocate for policies that you think would help the world, and only do things that the Big Companies and Their Allies are comfortable with.”
Again, I don’t think Richard sat in his room and thought “how do I spread a meme that is good for my company.” I think he’s genuinely saying what he believes and giving advice that he thinks will be useful to the AI safety community and improve society’s future .
But I also think that one of the reasons why Richard still works at OpenAI is because he’s the kind of agent who genuinely believes things that tend to be pretty aligned with OpenAI’s interests, and I suspect his perspective is informed by having lots of friends/colleagues at OpenAI.
Someone who works for a tobacco company can still have genuinely useful advice for the community of people concerned about the health effects of smoking. But I still think it’s an important epistemic norm that they add (at least) a brief disclaimer acknowledging that they work for a tobacco company.
(And the case becomes even stronger in the event that they have to get approval from the tobacco company comms team, or they filter out any ideas that they have that could get them in trouble with the company. Or perhaps before writing/publishing a post they consider the fact that other people have been fired from their company for sharing information that was against company interests, that the CEO attempted to remove a board member under the justification that she published a paper that went against company interests, that the company has previously used history of using highly restrive NDAs to prevent people from saying things that go against company interests.)
Added a disclaimer, as suggested. It seems like a good practice for this sort of post. Though note that I disagree with this paragraph; I don’t think “being the kind of agent who X” or “being informed by many people at Y” are good reasons to give disclaimers. Whereas I do buy that “they filter out any ideas that they have that could get them in trouble with the company” is an important (conscious or unconscious) effect, and worth a disclaimer.
I’ve also added this note to the text:
I appreciate you adding the note, though I do think the situation is far more unusual than described. I agree it’s widely priced in that companies in general seek power, but I think probably less so that the author of this post personally works for a company which is attempting to acquire drastically more power than any other company ever, and that much of the behavior the post describes as power-seeking amounts to “people trying to stop the author and his colleagues from attempting that.”
Thanks!
(I think “being the kind of agent who survives the selection process” can sometimes be an important epistemic thing to consider, though mostly when thinking about how systems work and what kinds of people/views those systems promote. Agreed that “being informed by many people who Y” is a rather weak one & certainly would not on its own warrant a disclosure.)
I’m not claiming it’s zero information, but there are lots of things that convey non-zero information which it’d be bad to set disclosure norms based on. E.g. “I’ve only ever worked at nonprofits” should definitely affect your opinion of someone’s epistemics (e.g. when they’re trying to evaluate corporate dynamics) but once we start getting people to disclose that sort of thing there’s no clear stopping point. So mostly I want the line to be “current relevant conflicts of interest”.
My take atm is “seems right that this shouldn’t be a permanent norm, there are definitely costs of disclaimer-ratcheting that are pretty bad. I think it might still be the right thing to do of your own accord in some cases, which is, like, superogetory.”
I think there’s maybe a weird thing with this post, where, it’s trying to be the timeless, abstract version of itself. It’s certainly easier to write the timeless abstract version than the “digging into specific examples and calling people out” version. But, I think the digging into specific examples is actually kind of important here – it’s easy to come away with vague takeaways that disagree, where everyone nods along but then mostly thinks it’s Those Other Guys who are being power seeking.
Given that it’s probably 10-50x harder to write the Post With Specific Examples, I think actually a pretty okay outcome is “ship the vague post, and let discussion in the comments get into the inside-baseball-details.” And, then, it’d be remiss for the post-author’s role in the ecosystem not coming up as an example to dig into.