I downvoted this comment because it felt uncomfortably scapegoat-y to me. If you think the OpenAI grant was a big mistake, it’s important to have a detailed investigation of what went wrong, and that sort of detailed investigation is most likely to succeed if you have cooperation from people who are involved. I’ve been reading a fair amount about what it takes to instill a culture of safety in an organization, and nothing I’ve seen suggests that scapegoating is a good approach.
Writing a postmortem is not punishment—it is a learning opportunity for the entire company.
...
Blameless postmortems are a tenet of SRE culture. For a postmortem to be truly blameless, it must focus on identifying the contributing causes of the incident without indicting any individual or team for bad or inappropriate behavior. A blamelessly written postmortem assumes that everyone involved in an incident had good intentions and did the right thing with the information they had. If a culture of finger pointing and shaming individuals or teams for doing the “wrong” thing prevails, people will not bring issues to light for fear of punishment.
Blameless culture originated in the healthcare and avionics industries where mistakes can be fatal. These industries nurture an environment where every “mistake” is seen as an opportunity to strengthen the system. When postmortems shift from allocating blame to investigating the systematic reasons why an individual or team had incomplete or incorrect information, effective prevention plans can be put in place. You can’t “fix” people, but you can fix systems and processes to better support people making the right choices when designing and maintaining complex systems.
...
Removing blame from a postmortem gives people the confidence to escalate issues without fear. It is also important not to stigmatize frequent production of postmortems by a person or team. An atmosphere of blame risks creating a culture in which incidents and issues are swept under the rug, leading to greater risk for the organization [Boy13].
...
We can say with confidence that thanks to our continuous investment in cultivating a postmortem culture, Google weathers fewer outages and fosters a better user experience.
If you start with the assumption that there was a moral failing on the part of the grantmakers, and you are wrong, there’s a good chance you’ll never learn that.
If you start with the assumption that there was a moral failing on the part of the grantmakers, and you are wrong, there’s a good chance you’ll never learn that.
I think you are misinterpreting the grandparent comment. I do not read any mention of a ‘moral failing’ in that comment. You seem worried because of the commenter’s clear description of what they think would be a sensible step for us to take given what they believe are egregious flaws in the decision-making processes of the people involved. I don’t think there’s anything wrong with such claims.
Again: You can care about people while also seeing their flaws and noticing how they are hurting you and others you care about. You can be empathetic to people having flawed decision making and care about them, while also wanting to keep them away from certain decision-making positions.
If you think the OpenAI grant was a big mistake, it’s important to have a detailed investigation of what went wrong, and that sort of detailed investigation is most likely to succeed if you have cooperation from people who are involved.
Oh, interesting. Who exactly do you think influential people like Holden Karnofsky and Paul Christiano are accountable to, exactly? This “detailed investigation” you speak of, and this notion of a “blameless culture”, makes a lot of sense when you are the head of an organization and you are conducting an investigation as to the systematic mistakes made by people who work for you, and who you are responsible for. I don’t think this situation is similar enough that you can use these intuitions blandly without thinking through the actual causal factors involved in this situation.
Note that I don’t necessarily endorse the grandparent comment claims. This is a complex situation and I’d spend more time analyzing it and what occurred.
Enforcing social norms to prevent scapegoating also destroys information that is valuable for accurate credit assignment and causally modelling reality.
I read the Ben Hoffman post you linked. I’m not finding it very clear, but the gist seems to be something like: Statements about others often import some sort of good/bad moral valence; trying to avoid this valence can decrease the accuracy of your statements.
If OP was optimizing purely for descriptive accuracy, disregarding everyone’s feelings, that would be one thing. But the discussion of “repercussions” before there’s been an investigation goes into pure-scapegoating territory if you ask me.
I do not read any mention of a ‘moral failing’ in that comment.
If OP wants to clarify that he doesn’t think there was a moral failing, I expect that to be helpful for a post-mortem. I expect some other people besides me also saw that subtext, even if it’s not explicit.
You can be empathetic to people having flawed decision making and care about them, while also wanting to keep them away from certain decision-making positions.
“Keep people away” sounds like moral talk to me. If you think someone’s decisionmaking is actively bad, i.e. you’d better off reversing any advice from them, then maybe you should keep them around so you can do that! But more realistically, someone who’s fucked up in a big way will probably have learned from that, and functional cultures don’t throw away hard-won knowledge.
Imagine a world where AI is just an inherently treacherous domain, and we throw out the leadership whenever they make a mistake. So we get a continuous churn of inexperienced leaders in an inherently treacherous domain—doesn’t sound like a recipe for success!
Oh, interesting. Who exactly do you think influential people like Holden Karnofsky and Paul Christiano are accountable to, exactly? This “detailed investigation” you speak of, and this notion of a “blameless culture”, makes a lot of sense when you are the head of an organization and you are conducting an investigation as to the systematic mistakes made by people who work for you, and who you are responsible for. I don’t think this situation is similar enough that you can use these intuitions blandly without thinking through the actual causal factors involved in this situation.
I agree that changes things. I’d be much more sympathetic to the OP if they were demanding an investigation or an apology.
But the discussion of “repercussions” before there’s been an investigation goes into pure-scapegoating territory if you ask me.
Just to be clear, OP themselves seem to think that what they are saying will have little effect on the status quo. They literally called it “Very Spicy Take”. Their intention was to allow them to express how they felt about the situation. I’m not sure why you find this threatening, because again, the people they think ideally wouldn’t continue to have influence over AI safety related decisions are incredibly influential and will very likely continue to have the influence they currently possess. Almost everyone else in this thread implicitly models this fact as they are discussing things related to the OP comment.
There is not going to be any scapegoating that will occur. I imagine that everything I say is something I would say in person to the people involved, or to third parties, and not expect any sort of coordinated action to reduce their influence—they are that irreplaceable to the community and to the ecosystem.
So basically, I think it is a bad idea and you think we can’t do it anyway. In that case let’s stop calling for it, and call for something more compassionate and realistic like a public apology.
I’ll bet an apology would be a more effective way to pressure OpenAI to clean up its act anyways. Which is a better headline—“OpenAI cofounder apologizes for their role in creating OpenAI”, or some sort of internal EA movement drama? If we can generate a steady stream of negative headlines about OpenAI, there’s a chance that Sam is declared too much of a PR and regulatory liability. I don’t think it’s a particularly good plan, but I haven’t heard a better one.
Can you not be close friends with someone while also expecting them to be bad at self-control when it comes to alcohol? Or perhaps they are great at technical stuff like research but pretty bad at negotiation, especially when dealing with experienced adverserial situations such as when talking to VCs?
If you think someone’s decisionmaking is actively bad, i.e. you’d better off reversing any advice from them, then maybe you should keep them around so you can do that!
It is not that people people’s decision-making skill is optimized such that you can consistently reverse someone’s opinion to get something that accurately tracks reality. If that was the case then they are implicitly tracking reality very well already. Reversed stupidity is not intelligence.
But more realistically, someone who’s fucked up in a big way will probably have learned from that, and functional cultures don’t throw away hard-won knowledge.
Again you seem to not be trying to track the context of our discussion here. This advice again is usually said when it comes to junior people embedded in an institution, because the ability to blame someone and / or hold them responsible is a power that senior / executive people hold. This attitude you describe makes a lot of sense when it comes to people who are learning things, yes. I don’t know if you can plainly bring it into this domain, and you even acknowledge this in the next few lines.
Imagine a world where AI is just an inherently treacherous domain, and we throw out the leadership whenever they make a mistake.
I think it is incredibly unlikely that the rationalist community has an ability to ‘throw out’ the ‘leadership’ involved here. I find this notion incredibly silly, given the amount of influence OpenPhil has over the alignment community, especially through their funding (including the pipeline, such as MATS).
It is not that people people’s decision-making skill is optimized such that you can consistently reverse someone’s opinion to get something that accurately tracks reality. If that was the case then they are implicitly tracking reality very well already. Reversed stupidity is not intelligence.
Sure, I think this helps tease out the moral valence point I was trying to make. “Don’t allow them near” implies their advice is actively harmful, which in turn suggests that reversing it could be a good idea. But as you say, this is implausible. A more plausible statement is that their advice is basically noise—you shouldn’t pay too much attention to it. I expect OP would’ve said something like that if they were focused on descriptive accuracy rather than scapegoating.
Another way to illuminate the moral dimension of this conversation: If we’re talking about poor decision-making, perhaps MIRI and FHI should also be discussed? They did a lot to create interest in AGI, and MIRI failed to create good alignment researchers by its own lights. Now after doing advocacy off and on for years, and creating this situation, they’re pivoting to 100% advocacy.
Could MIRI be made up of good people who are “great at technical stuff”, yet apt to shoot themselves in the foot when it comes to communicating with the public? It’s hard for me to imagine an upvoted post on this forum saying “MIRI shouldn’t be allowed anywhere near AI safety communications”.
I downvoted this comment because it felt uncomfortably scapegoat-y to me. If you think the OpenAI grant was a big mistake, it’s important to have a detailed investigation of what went wrong, and that sort of detailed investigation is most likely to succeed if you have cooperation from people who are involved. I’ve been reading a fair amount about what it takes to instill a culture of safety in an organization, and nothing I’ve seen suggests that scapegoating is a good approach.
https://sre.google/sre-book/postmortem-culture/
If you start with the assumption that there was a moral failing on the part of the grantmakers, and you are wrong, there’s a good chance you’ll never learn that.
Enforcing social norms to prevent scapegoating also destroys information that is valuable for accurate credit assignment and causally modelling reality.
I think you are misinterpreting the grandparent comment. I do not read any mention of a ‘moral failing’ in that comment. You seem worried because of the commenter’s clear description of what they think would be a sensible step for us to take given what they believe are egregious flaws in the decision-making processes of the people involved. I don’t think there’s anything wrong with such claims.
Again: You can care about people while also seeing their flaws and noticing how they are hurting you and others you care about. You can be empathetic to people having flawed decision making and care about them, while also wanting to keep them away from certain decision-making positions.
Oh, interesting. Who exactly do you think influential people like Holden Karnofsky and Paul Christiano are accountable to, exactly? This “detailed investigation” you speak of, and this notion of a “blameless culture”, makes a lot of sense when you are the head of an organization and you are conducting an investigation as to the systematic mistakes made by people who work for you, and who you are responsible for. I don’t think this situation is similar enough that you can use these intuitions blandly without thinking through the actual causal factors involved in this situation.
Note that I don’t necessarily endorse the grandparent comment claims. This is a complex situation and I’d spend more time analyzing it and what occurred.
I read the Ben Hoffman post you linked. I’m not finding it very clear, but the gist seems to be something like: Statements about others often import some sort of good/bad moral valence; trying to avoid this valence can decrease the accuracy of your statements.
If OP was optimizing purely for descriptive accuracy, disregarding everyone’s feelings, that would be one thing. But the discussion of “repercussions” before there’s been an investigation goes into pure-scapegoating territory if you ask me.
If OP wants to clarify that he doesn’t think there was a moral failing, I expect that to be helpful for a post-mortem. I expect some other people besides me also saw that subtext, even if it’s not explicit.
“Keep people away” sounds like moral talk to me. If you think someone’s decisionmaking is actively bad, i.e. you’d better off reversing any advice from them, then maybe you should keep them around so you can do that! But more realistically, someone who’s fucked up in a big way will probably have learned from that, and functional cultures don’t throw away hard-won knowledge.
Imagine a world where AI is just an inherently treacherous domain, and we throw out the leadership whenever they make a mistake. So we get a continuous churn of inexperienced leaders in an inherently treacherous domain—doesn’t sound like a recipe for success!
I agree that changes things. I’d be much more sympathetic to the OP if they were demanding an investigation or an apology.
Just to be clear, OP themselves seem to think that what they are saying will have little effect on the status quo. They literally called it “Very Spicy Take”. Their intention was to allow them to express how they felt about the situation. I’m not sure why you find this threatening, because again, the people they think ideally wouldn’t continue to have influence over AI safety related decisions are incredibly influential and will very likely continue to have the influence they currently possess. Almost everyone else in this thread implicitly models this fact as they are discussing things related to the OP comment.
There is not going to be any scapegoating that will occur. I imagine that everything I say is something I would say in person to the people involved, or to third parties, and not expect any sort of coordinated action to reduce their influence—they are that irreplaceable to the community and to the ecosystem.
So basically, I think it is a bad idea and you think we can’t do it anyway. In that case let’s stop calling for it, and call for something more compassionate and realistic like a public apology.
I’ll bet an apology would be a more effective way to pressure OpenAI to clean up its act anyways. Which is a better headline—“OpenAI cofounder apologizes for their role in creating OpenAI”, or some sort of internal EA movement drama? If we can generate a steady stream of negative headlines about OpenAI, there’s a chance that Sam is declared too much of a PR and regulatory liability. I don’t think it’s a particularly good plan, but I haven’t heard a better one.
Can you not be close friends with someone while also expecting them to be bad at self-control when it comes to alcohol? Or perhaps they are great at technical stuff like research but pretty bad at negotiation, especially when dealing with experienced adverserial situations such as when talking to VCs?
It is not that people people’s decision-making skill is optimized such that you can consistently reverse someone’s opinion to get something that accurately tracks reality. If that was the case then they are implicitly tracking reality very well already. Reversed stupidity is not intelligence.
Again you seem to not be trying to track the context of our discussion here. This advice again is usually said when it comes to junior people embedded in an institution, because the ability to blame someone and / or hold them responsible is a power that senior / executive people hold. This attitude you describe makes a lot of sense when it comes to people who are learning things, yes. I don’t know if you can plainly bring it into this domain, and you even acknowledge this in the next few lines.
I think it is incredibly unlikely that the rationalist community has an ability to ‘throw out’ the ‘leadership’ involved here. I find this notion incredibly silly, given the amount of influence OpenPhil has over the alignment community, especially through their funding (including the pipeline, such as MATS).
Sure, I think this helps tease out the moral valence point I was trying to make. “Don’t allow them near” implies their advice is actively harmful, which in turn suggests that reversing it could be a good idea. But as you say, this is implausible. A more plausible statement is that their advice is basically noise—you shouldn’t pay too much attention to it. I expect OP would’ve said something like that if they were focused on descriptive accuracy rather than scapegoating.
Another way to illuminate the moral dimension of this conversation: If we’re talking about poor decision-making, perhaps MIRI and FHI should also be discussed? They did a lot to create interest in AGI, and MIRI failed to create good alignment researchers by its own lights. Now after doing advocacy off and on for years, and creating this situation, they’re pivoting to 100% advocacy.
Could MIRI be made up of good people who are “great at technical stuff”, yet apt to shoot themselves in the foot when it comes to communicating with the public? It’s hard for me to imagine an upvoted post on this forum saying “MIRI shouldn’t be allowed anywhere near AI safety communications”.