I also feel really frustrated that you wrote this, Anna. I think there are a number of obvious and significant disanalogies between the situations at Leverage versus MIRI/CFAR. There’s a lot to say here, but a few examples which seem especially salient:
To the best of my knowledge, the leadership of neither MIRI nor CFAR has ever slept with any subordinates, much less many of them.
While I think staff at MIRI and CFAR do engage in motivated reasoning sometimes wrt PR, neither org engaged in anything close to the level of obsessive, anti-epistemic reputational control alleged in Zoe’s post. MIRI and CFAR staff were not required to sign NDAs agreeing they wouldn’t talk badly about the org—in fact, at least in my experience with CFAR, staff much more commonly share criticism of the org than praise. CFAR staff were regularly encouraged to share their ideas at workshops and on LessWrong, to get public feedback. And when we did mess up, we tried quite hard to publicly and accurately describe our wrongdoing—e.g., Anna and I spent low-hundreds of hours investigating/thinking through the Brent affair, and tried so hard to avoid accidentally doing anti-epistemic reputational control (this was our most common topic of conversation during this process) that in my opinion, our writeup about it actually makes CFAR seem much more culpable than I think it was.
As I understand it, there were ~3 staff historically whose job descriptions involved debugging in some way which you, Anna, now feel uncomfortable with/think was fucky. But to the best of your knowledge, these situations caused much less harm than e.g. Zoe seems to have experienced, and the large majority of staff did not experience this—in general staff rarely explicitly debugged each other, and when it did happen it was clearly opt-in, and fairly symmetrical (e.g., in my personal conversations with you Anna, I’d guess the ratio of you something-like-debugging me to the reverse is maybe 3/2?).
CFAR put really a lot of time and effort into trying to figure out how to teach rationality techniques, and how to talk with people about x-risk, without accidentally doing something fucky to people’s psyches. Our training curriculum for workshop mentors includes extensive advice on ways to avoid accidentally causing psychological harm. Harm did happen sometimes, which was why our training emphasized it so heavily. But we really fucking tried, and my sense is that we actually did very well on the whole at establishing institutional and personal knowledge about how to be gentle with people in these situations; personally, it’s the skillset I’d most worry about the community losing if CFAR shut down and more events started being run by other orgs.
Insofar as you agree with the above, Anna, I’d appreciate you stating that clearly, since I think saying “the OP speaks for me” implies you think the core analogy described in the OP was non-misleading.
Yeah, sorry. I agree that my comment “the OP speaks for me” is leading a lot of people to false views that I should correct. It’s somehow tricky because there’s a different thing I worry will be obscured by my doing this, but I’ll do it anyhow as is correct and try to come back for that different thing later.
To the best of my knowledge, the leadership of neither MIRI nor CFAR has ever slept with a subordinate, much less many of them.
Agreed.
While I think staff at CFAR and MIRI probably engaged in motivated reasoning sometimes wrt PR, neither org engaged in anything close to the level of obsessive, anti-epistemic reputational control alleged in Zoe’s post. CFAR and MIRI staff were certainly not required to sign NDAs agreeing they wouldn’t talk badly about the org—in fact, in my experience CFAR staff much more commonly share criticism of the org than praise. CFAR staff were regularly encouraged to share their ideas at workshops and on LessWrong, to get public feedback. And when we did mess up, we tried extremely hard to publicly and accurately describe our wrongdoing—e.g., Anna and I personally spent hundreds of hours investigating/thinking about the Brent affair, and tried so hard to avoid accidentally doing anti-epistemic reputational control that in my opinion, our writeup about it actually makes CFAR seem much more culpable than I think it was.
I agree that there’s a large difference in both philosophy of how/whether to manage reputation, and amount of control exhibited/attempted about how staff would talk about the organizations, with Leverage doing a lot of that and CFAR doing less of it than most organizations.
As I understand it, there were ~3 staff historically whose job description involved debugging in some way which you, Anna, now feel uncomfortable with/think was fucky. But to the best of your knowledge, these situations caused much less harm than e.g. Zoe seems to have experienced, and the large majority of staff did not experience this—in general staff rarely explicitly debugged each other, and when it did happen it was clearly opt-in, and fairly symmetrical (e.g., in my personal conversations with you Anna, I’d guess the ratio of you something-like-debugging me to the reverse is maybe 3/2?).
I think this understates both how many people it happened with, and how fucky it sometimes was. (Also, it was job but not “job description”, although I think Zoe’s was “job description”). I think this one was actually worse in some of the early years, vs your model of it. My guess is indeed that it involved fewer hours than Zoe, and was overall less deliberately part of a dynamic quite as fucky as Zoe’s, but as I mentioned to you on the phone, an early peripheral staff member left CFAR for a mental institution in a way that seemed plausibly to do with how debugging and trials worked, and definitely to do with workplace stress of some sort, as well as with a preexisting condition they entered with and didn’t tell us about. (We would’ve handled this better later, I think.) There are some other situations that were also I think pretty fucked up, in the sense of “I think the average person would experience some horror/indignation if they took in what was happening.”
I can also think of stories of real scarring outside the three people I was counting.
I… do think it was considerably less weird looking, and less overtly fucked-up looking, than the descriptions I have (since writing my “this post speaks for me” comment) gotten of Leverage in the 2018-2019 era.
Also, most people at CFAR, especially in recent years, I think suffered none or nearly none of this. (I believe the same was true for parts of Leverage, though not sure.)
So, if we are playing the “compare how bad Leverage and CFAR are along each axis” game (which is not the main thing I took the OP to be doing, at all, nor the main thing I was trying to agree with, at all), I do think Leverage is worse than CFAR on this axis but I think the “per capita” damage of this sort that hit CFAR staff in the early years (“per capita” rather than cumulative, because Leverage had many more people) was maybe about a tenth of my best guess at what was up in the near-Zoe parts of Leverage in 2018-2019, which is a lot but, yes, different.
CFAR put really a lot of time and effort into trying to figure out how to teach rationality techniques, and how to talk with people about x-risk, without accidentally doing something fucky to people’s psyches. Our training curriculum for workshop mentors includes extensive advice on ways to avoid accidentally causing psychological harm. Harm did happen sometimes, which was why our training emphasized it so heavily. But we really fucking tried, and my sense is that we actually did very well on the whole at establishing institutional and personal knowledge about how to be gentle with people in these situations; personally, it’s the skillset I’d most worry about the community losing if CFAR shut down and more events started being run by other orgs.
We indeed put a lot of effort into this, and got some actual skill and good institutional habits out.
Perhaps this is an opportunity to create an internal document on “unhealthy behaviors” that would list the screwups and the lessons learned, and read it together regularly, like a safety training? (Analogically to how organizations that get their computers hacked or documents stolen, describe how it happened as a part of their safety training.) Perhaps with anonymous feedback whether someone has a concern that MIRI or CFAR is slipping into some bad pattern again.
Also, it might be useful to hire an external psychologist who would in regular intervals have a discussion with MIRI/CFAR employees. And to provide this document to the psychologist, so they know what risks to focus on. (Furthermore I think the psychologist should not be a rationalist; to provide a better outside view.)
For starters, someone could create the first version of the document by extracting information from this debate.
EDIT: Oops, on second reading of your comment, it seems like you already have something like this. Uhm, maybe a good opportunity to update/extend the document?
*
As a completely separate topic, it would be nice to have a table with the following columns: “Safety concern”, “What happened in MIRI/CFAR”, “What happened in Leverage (as far as we know)”, “Similarities”, “Differences”. But this is much less important, in long term.
I endorse Adam’s commentary, though I did not feel the frustration Eli and Adam report, possibly because I know Anna well enough that I reflexively did the caveating in my own brain rather than modeling the audience.
To the best of my knowledge, the leadership of neither MIRI nor CFAR has ever slept with a subordinate, much less many of them.
This issue doesn’t seem particularly important to me but the comparison you’re making is a good example of a more general problem I want to talk about.
My impression is that the decision structure of CFAR was much less legible & transparent than that of Leverage, so that it would be harder to determine who might be treated as subordinate to whom in what context. In addition, my impression from the years I was around is that Leverage didn’t preside over as much of an external scene, - Leverage followers had formalized roles as members of the organization, while CFAR had a “community,” many of whom were workshop alumni.
And when we did mess up, we tried extremely hard to publicly and accurately describe our wrongdoing—e.g., Anna and I personally spent hundreds of hours investigating/thinking about the Brent affair, and tried so hard to avoid accidentally doing anti-epistemic reputational control that in my opinion, our writeup about it actually makes CFAR seem much more culpable than I think it was.
Am I missing something here? The communication I read from CFAR seemed like it was trying to reveal as little as it could get away with, gradually saying more (and taking a harsher stance towards Brent) in response to public pressure, not like it was trying to help me, a reader, understand what had happened.
Anna and I personally spent hundreds of hours investigating/thinking about the Brent affair… our writeup about it...
Am I missing something here? The communication I read from CFAR seemed like it was trying to reveal as little as it could get away with...
FWIW, I think you and Adam are talking about two different pieces of communication. I think you are thinking of the communication leading up to the big community-wide discussion that happened in Sept 2018, while Adam is thinking specifically of CFAR’s follow-up communication months after that — in particular this post. (It would have been in between those two times when Adam and Anna did all that thinking that he was talking about.)
I agree manager/staff relations have often been less clear at CFAR than is typical. But I’m skeptical that’s relevant here, since as far as I know there aren’t really even borderline examples of this happening. The closest example to something like this I can think of is that staff occasionally invite their partners to attend or volunteer at workshops, which I think does pose some risk of fucky power dynamics, albeit dramatically less risk imo than would be posed by “the clear leader of an organization, who’s revered by staff as a world-historically important philosopher upon whose actions the fate of the world rests, and who has unilateral power to fire any of them, sleeps with many employees.”
Am I missing something here? The communication I read from CFAR seemed like it was trying to reveal as little as it could get away with, gradually saying more (and taking a harsher stance towards Brent) in response to public pressure, not like it was trying to help me, a reader, understand what had happened.
As lead author on the Brent post, I felt bummed reading this. I tried really hard to avoid letting my care for/interest in CFAR affect my descriptions of what happened, or my choices about what to describe. Anna and I spent quite large amounts of time—at least double-digit hours, I think probably triple-digit—searching for ways our cognition might be biased or motivated or PR-like, and trying to correct for that. We debated and introspected about it, ran drafts by friends of ours who seemed unusually likely to call us on bullshit, etc.
Looking back, my sense remains that we basically succeeded—i.e., that we described the situation about as accurately and neutrally as we could have. If I’m wrong about this… well, it wasn’t for lack of trying.
Looking back, my sense remains that we basically succeeded—i.e., that we described the situation about as accurately and neutrally as we could have. If I’m wrong about this… well, all I can say is that it wasn’t for lack of trying.
I think CFAR ultimately succeeded in providing a candid and good faith account of what went wrong, but the time it took to get there (i.e. 6 months between this and the initial update/apology) invites adverse inferences like those in the grandparent.
A lot of the information ultimately disclosed in March would definitely have been known to CFAR in September, such as Brent’s prior involvement as a volunteer/contractor for CFAR, his relationships/friendships with current staff, and the events as ESPR. The initial responses remained coy on these points, and seemed apt to give the misleading impression CFAR’s mistakes were (relatively) much milder than they in fact were. I (among many) contacted CFAR leadership to urge them to provide more candid and complete account when I discovered some of this further information independently.
I also think, similar to how it would be reasonable to doubt ‘utmost corporate candour’ back then given initial partial disclosure, it’s reasonable to doubt CFAR has addressed the shortcomings revealed given the lack of concrete follow-up. I also approached CFAR leadership when CFAR’s 2019 Progress Report and Future Plans initially made no mention of what happened with Brent, nor what CFAR intended to improve in response to it. What was added in is not greatly reassuring:
And after spending significant time investigating our mistakes with regard to Brent, we reformed our hiring, admissions and conduct policies, to reduce the likelihood such mistakes reoccur.
A cynic would note this is ‘marking your own homework’, but cynicism is unnecessary to recommend more self-scepticism. I don’t doubt the Brent situation indeed inspired a lot of soul searching and substantial, sincere efforts to improve. What is more doubtful (especially given the rest of the morass of comments) is whether these efforts actually worked. Although there is little prospect of satisfying me, more transparency over what exactly has changed—and perhaps third party oversight and review—may better reassure others.
It would help if they actually listed and gave examples of exactly what kind of mental manipulation they were doing to people other than telling them to take drugs. These comments seem to dance around the exactly details of what happened and only talk about the group dynamics between people as a result of these mysterious actions/events.
I also feel really frustrated that you wrote this, Anna. I think there are a number of obvious and significant disanalogies between the situations at Leverage versus MIRI/CFAR. There’s a lot to say here, but a few examples which seem especially salient:
To the best of my knowledge, the leadership of neither MIRI nor CFAR has ever slept with any subordinates, much less many of them.
While I think staff at MIRI and CFAR do engage in motivated reasoning sometimes wrt PR, neither org engaged in anything close to the level of obsessive, anti-epistemic reputational control alleged in Zoe’s post. MIRI and CFAR staff were not required to sign NDAs agreeing they wouldn’t talk badly about the org—in fact, at least in my experience with CFAR, staff much more commonly share criticism of the org than praise. CFAR staff were regularly encouraged to share their ideas at workshops and on LessWrong, to get public feedback. And when we did mess up, we tried quite hard to publicly and accurately describe our wrongdoing—e.g., Anna and I spent low-hundreds of hours investigating/thinking through the Brent affair, and tried so hard to avoid accidentally doing anti-epistemic reputational control (this was our most common topic of conversation during this process) that in my opinion, our writeup about it actually makes CFAR seem much more culpable than I think it was.
As I understand it, there were ~3 staff historically whose job descriptions involved debugging in some way which you, Anna, now feel uncomfortable with/think was fucky. But to the best of your knowledge, these situations caused much less harm than e.g. Zoe seems to have experienced, and the large majority of staff did not experience this—in general staff rarely explicitly debugged each other, and when it did happen it was clearly opt-in, and fairly symmetrical (e.g., in my personal conversations with you Anna, I’d guess the ratio of you something-like-debugging me to the reverse is maybe 3/2?).
CFAR put really a lot of time and effort into trying to figure out how to teach rationality techniques, and how to talk with people about x-risk, without accidentally doing something fucky to people’s psyches. Our training curriculum for workshop mentors includes extensive advice on ways to avoid accidentally causing psychological harm. Harm did happen sometimes, which was why our training emphasized it so heavily. But we really fucking tried, and my sense is that we actually did very well on the whole at establishing institutional and personal knowledge about how to be gentle with people in these situations; personally, it’s the skillset I’d most worry about the community losing if CFAR shut down and more events started being run by other orgs.
Insofar as you agree with the above, Anna, I’d appreciate you stating that clearly, since I think saying “the OP speaks for me” implies you think the core analogy described in the OP was non-misleading.
Yeah, sorry. I agree that my comment “the OP speaks for me” is leading a lot of people to false views that I should correct. It’s somehow tricky because there’s a different thing I worry will be obscured by my doing this, but I’ll do it anyhow as is correct and try to come back for that different thing later.
Agreed.
I agree that there’s a large difference in both philosophy of how/whether to manage reputation, and amount of control exhibited/attempted about how staff would talk about the organizations, with Leverage doing a lot of that and CFAR doing less of it than most organizations.
I think this understates both how many people it happened with, and how fucky it sometimes was. (Also, it was job but not “job description”, although I think Zoe’s was “job description”). I think this one was actually worse in some of the early years, vs your model of it. My guess is indeed that it involved fewer hours than Zoe, and was overall less deliberately part of a dynamic quite as fucky as Zoe’s, but as I mentioned to you on the phone, an early peripheral staff member left CFAR for a mental institution in a way that seemed plausibly to do with how debugging and trials worked, and definitely to do with workplace stress of some sort, as well as with a preexisting condition they entered with and didn’t tell us about. (We would’ve handled this better later, I think.) There are some other situations that were also I think pretty fucked up, in the sense of “I think the average person would experience some horror/indignation if they took in what was happening.”
I can also think of stories of real scarring outside the three people I was counting.
I… do think it was considerably less weird looking, and less overtly fucked-up looking, than the descriptions I have (since writing my “this post speaks for me” comment) gotten of Leverage in the 2018-2019 era.
Also, most people at CFAR, especially in recent years, I think suffered none or nearly none of this. (I believe the same was true for parts of Leverage, though not sure.)
So, if we are playing the “compare how bad Leverage and CFAR are along each axis” game (which is not the main thing I took the OP to be doing, at all, nor the main thing I was trying to agree with, at all), I do think Leverage is worse than CFAR on this axis but I think the “per capita” damage of this sort that hit CFAR staff in the early years (“per capita” rather than cumulative, because Leverage had many more people) was maybe about a tenth of my best guess at what was up in the near-Zoe parts of Leverage in 2018-2019, which is a lot but, yes, different.
We indeed put a lot of effort into this, and got some actual skill and good institutional habits out.
Perhaps this is an opportunity to create an internal document on “unhealthy behaviors” that would list the screwups and the lessons learned, and read it together regularly, like a safety training? (Analogically to how organizations that get their computers hacked or documents stolen, describe how it happened as a part of their safety training.) Perhaps with anonymous feedback whether someone has a concern that MIRI or CFAR is slipping into some bad pattern again.
Also, it might be useful to hire an external psychologist who would in regular intervals have a discussion with MIRI/CFAR employees. And to provide this document to the psychologist, so they know what risks to focus on. (Furthermore I think the psychologist should not be a rationalist; to provide a better outside view.)
For starters, someone could create the first version of the document by extracting information from this debate.
EDIT: Oops, on second reading of your comment, it seems like you already have something like this. Uhm, maybe a good opportunity to update/extend the document?
*
As a completely separate topic, it would be nice to have a table with the following columns: “Safety concern”, “What happened in MIRI/CFAR”, “What happened in Leverage (as far as we know)”, “Similarities”, “Differences”. But this is much less important, in long term.
I endorse Adam’s commentary, though I did not feel the frustration Eli and Adam report, possibly because I know Anna well enough that I reflexively did the caveating in my own brain rather than modeling the audience.
This issue doesn’t seem particularly important to me but the comparison you’re making is a good example of a more general problem I want to talk about.
My impression is that the decision structure of CFAR was much less legible & transparent than that of Leverage, so that it would be harder to determine who might be treated as subordinate to whom in what context. In addition, my impression from the years I was around is that Leverage didn’t preside over as much of an external scene, - Leverage followers had formalized roles as members of the organization, while CFAR had a “community,” many of whom were workshop alumni.
Am I missing something here? The communication I read from CFAR seemed like it was trying to reveal as little as it could get away with, gradually saying more (and taking a harsher stance towards Brent) in response to public pressure, not like it was trying to help me, a reader, understand what had happened.
FWIW, I think you and Adam are talking about two different pieces of communication. I think you are thinking of the communication leading up to the big community-wide discussion that happened in Sept 2018, while Adam is thinking specifically of CFAR’s follow-up communication months after that — in particular this post. (It would have been in between those two times when Adam and Anna did all that thinking that he was talking about.)
Yeah, this was the post I meant.
I agree manager/staff relations have often been less clear at CFAR than is typical. But I’m skeptical that’s relevant here, since as far as I know there aren’t really even borderline examples of this happening. The closest example to something like this I can think of is that staff occasionally invite their partners to attend or volunteer at workshops, which I think does pose some risk of fucky power dynamics, albeit dramatically less risk imo than would be posed by “the clear leader of an organization, who’s revered by staff as a world-historically important philosopher upon whose actions the fate of the world rests, and who has unilateral power to fire any of them, sleeps with many employees.”
As lead author on the Brent post, I felt bummed reading this. I tried really hard to avoid letting my care for/interest in CFAR affect my descriptions of what happened, or my choices about what to describe. Anna and I spent quite large amounts of time—at least double-digit hours, I think probably triple-digit—searching for ways our cognition might be biased or motivated or PR-like, and trying to correct for that. We debated and introspected about it, ran drafts by friends of ours who seemed unusually likely to call us on bullshit, etc.
Looking back, my sense remains that we basically succeeded—i.e., that we described the situation about as accurately and neutrally as we could have. If I’m wrong about this… well, it wasn’t for lack of trying.
I think CFAR ultimately succeeded in providing a candid and good faith account of what went wrong, but the time it took to get there (i.e. 6 months between this and the initial update/apology) invites adverse inferences like those in the grandparent.
A lot of the information ultimately disclosed in March would definitely have been known to CFAR in September, such as Brent’s prior involvement as a volunteer/contractor for CFAR, his relationships/friendships with current staff, and the events as ESPR. The initial responses remained coy on these points, and seemed apt to give the misleading impression CFAR’s mistakes were (relatively) much milder than they in fact were. I (among many) contacted CFAR leadership to urge them to provide more candid and complete account when I discovered some of this further information independently.
I also think, similar to how it would be reasonable to doubt ‘utmost corporate candour’ back then given initial partial disclosure, it’s reasonable to doubt CFAR has addressed the shortcomings revealed given the lack of concrete follow-up. I also approached CFAR leadership when CFAR’s 2019 Progress Report and Future Plans initially made no mention of what happened with Brent, nor what CFAR intended to improve in response to it. What was added in is not greatly reassuring:
A cynic would note this is ‘marking your own homework’, but cynicism is unnecessary to recommend more self-scepticism. I don’t doubt the Brent situation indeed inspired a lot of soul searching and substantial, sincere efforts to improve. What is more doubtful (especially given the rest of the morass of comments) is whether these efforts actually worked. Although there is little prospect of satisfying me, more transparency over what exactly has changed—and perhaps third party oversight and review—may better reassure others.
It would help if they actually listed and gave examples of exactly what kind of mental manipulation they were doing to people other than telling them to take drugs. These comments seem to dance around the exactly details of what happened and only talk about the group dynamics between people as a result of these mysterious actions/events.