Anna, I feel frustrated that you wrote this. Unless I have severely misunderstood you, this seems extremely misleading.
For context, before this post was published Anna and I discussed the comparison between MIRI/CFAR and Leverage.
At that time, you, Anna, posited a high level dynamic involving “narrative pyramid schemes” accelerating, and then going bankrupt, at about the same time. I agreed that this seemed like it might have something to it, but emphasized that, despite some high level similarities, what happened at MIRI/CFAR was meaningfully different from, and much much less harmful than, what Zoe described in her post.
We then went through a specific operationalization of one of the specific claimed parallels (specifically the frequency and oppressiveness of superior-to-subordinate debugging), and you agreed that while the CFAR case was, quantitatively, an order of magnitude better than what Zoe describes. We talked more generally about some of the other parallels, and you generally agreed that the specific harms were much greater in the Leverage case.
(And just now, I talked with another CFAR staff member who reported that the two of you went point by point, and for each one you agreed that the CFAR/MIRI situation was much less bad than the Leverage case. [edit: I misunderstood. They only went through 5 points, out of many, but out of those 5 Anna agreed that the Leverage case was broadly worse.])
I think that you believe, as I do, that there were some high-level structural similarities between the dynamics at MIRI/CFAR and at Leverage, and also what happened at Leverage was substantially worse* than what happened at MIRI/CFAR.
Do you believe that?
If so, can you please clearly say so?
I feel like not clearly stating that second part is extremely and damagingly misleading.
What is at stake is not just the abstract dynamics, but also the concrete question of how alarmed, qualitatively, people around here should be. It seems to me that you, with this comment, are implying that it is appropriate to be about as alarmed by Zoe’s report of Leverage as by this description of MIRI/CFAR. Which seems wrong to me.
[Edit: * - This formally read “an order of magnitude worse”.
I think this is correct, for a number of common sense metrics (ie “there was at least 10x as many hours of superior-subordinate debugging at Leverage, where this seemed to be an institutionalized practice making up a lot of a person’s day, compared to CFAR, where this did happen sometimes what wasn’t a core feature of the org. (This is without taking into account the differences in how harmful those hours were. The worst case of which I’m aware of this happening at CFAR was less harmful than Zoe’s account.)
I think across most metrics named, Leverage had a worse or stronger version of thing, with a few exceptions. MIRI’s (but not CFAR’s, mostly) narrative had more urgency about it than Leverage’s for instance, because of AI timeline considerations, and overall the level of “intensity” or “pressure” around MIRI and Leverage might have been similar? I’m not close enough to either org to say with confidence.
But overall, I think it is weird to talk about “orders of magnitude” without referring to a specific metric, since it has the veneer of rigor without really adding much substance. I’m hoping that this edit adds some of that substance and I’m walking my claim back to the vaguer “substantially worse”, with the caveat that I am generally in favor of, and open to sharing more specific quantitative estimates on specific operationalizations if asked.]
I think that you believe, as I do, that there were some high-level structural similarities between the dynamics at MIRI/CFAR and at Leverage, and also what happened at Leverage was an order of magnitude worse than what happened at MIRI/CFAR.
Leverage_2018-2019 sounds considerably worse than Leverage 2013-2016.
My current guess is that if you took a random secular American to be your judge, or a random LWer, and you let them watch the life of a randomly chosen member of the Leverage psychology team from 2018-2019 (which I’m told is the worst part) and also of a randomly chosen staff member at either MIRI or CFAR, they would be at least 10x more horrified by the experience of the one in the Leverage psychology team.
I somehow don’t know how to say in my own person “was an order of magnitude worse”, but I can say the above. The reason I don’t know how to say “was an order of magnitude worse” is because it honestly looks to me (as to Jessica in the OP) like many places are pretty bad for many people, in the sense of degrading their souls via deceptions, manipulations, and other ethical violations. I’m not sure if this view of mine will sound over-the-top/dismissable or we-all-already-know-that/dismissible, or something else, but I have in mind such things as:
It seems to me that many many kids enter school with a desire to learn and an ability to trust their own mind, and leave school with a weird kind of “make sure you don’t get it wrong” that inhibits trying and doing. Some of this is normal aging, but my best guess is that an important chunk is more like cultural damage.
Many teenagers can do philosophy, stretch, try to think about the world. Most of the same folks at 30 or 40 can’t, outside of the ~one specific disciplines in which they’re a professional. They don’t let themselves.
Lots of upper middle class adults hardly know how to have conversations, of the “talk from the person inside who is actually home, asking what they want to know instead of staying safe, hitting new unpredictable thoughts/conversations” sense. This is a change from childhood. Again, this is probably partly aging, but I suspect cultural damage, and I’ve been told a couple times (including by folks who have no contact with Vassar or anyone else in this community) that this is less true for working class folks than for upper middle class folks, which if true is evidence for it being partly cultural damage though I should check this better.
Some staff IMO initially expect that folks at CFAR or Google or the FDA or wherever will be trying to do something real, and then come to later relate to it more like belief-in-belief, and to lots of other things too, with language coming to seem more like a mechanism for coordinating our belief-in-beliefs, and less like light with which one can talk and reason. And with things in general coming to seem kind of remote and as though you can’t really hope for anything real.
Anyhow. This essay wants to be larger than I’m willing to make this comment-reply before sleeping, so I’ll just keep doing it poorly/briefly, and hope to have more conversation later not necessarily under Jessica’s OP. But my best guess is that both CFAR of most of the last ten years, and the average workplace, are:
a) On the one hand, quite a bit less overtly hellish than the Leverage psychology teams of 2018-2019; but nevertheless maybe full of secret bits of despair and giving-up-on bits of our birthrights, in ways that are mostly not consciously noticed;
b) More than 1/10th as damaging to most employees’ basic human capacities, compared to Leverage_2018-2019.
Why do I think b? Partly because of my observations on what happens to people in the broader world (including control groups of folks who do their own thing among good people and end up fine, but I might be rigging my data and playing “no true scottsmen” games to get rid of the rest, and misconstruing natural aging or something). And partly because I chatted with several people in the past week who spent time at Leverage, and they all seemed like they had intact souls, to me, although my soul-ometer is not necessarily that accurate etc.
But, anyhow, I agree that most people would see what you’re saying, I’m just seeing something else and I care about it and I’m sorry if I said it in a confusing/misleading way but it is actually pretty hard to talk about.
These claims seem rather extreme and unsupported to me:
“Lots of upper middle class adults hardly know how to have conversations...”
“the average workplace [is] more than 1/10th as damaging to most employees’ basic human capacities, compared to Leverage_2018-2019.”
I suggest if you write a toplevel post, you search for evidence for/against them.
Elaborating a bit on my reasons for skepticism:
It seems like for the past 10+ years, you’ve been mostly interacting with people in CFAR-adjacent contexts. I’m not sure what your source of knowledge is on “average” upper middle class adults/workplaces. My personal experience is normal people are comfortable having non-superficial conversations if you convince them you aren’t weird first, and normal workplaces are pretty much fine. (I might be overselecting on smaller companies where people have a sense of humor.)
A specific concrete piece of evidence: Joe Rogan has one of the world’s most popular podcasts, and the episodes I’ve heard very much seem to me like they’re “hitting new unpredictable thoughts”. Rogan is notorious for talking to guests about DMT, for instance.
The two observations seems a bit inconsistent, if you’ll grant that working class people generally have worse working conditions than upper middle class people—you’d expect them to experience more workplace abuse and therefore have more trauma. (In which context would an abusive boss be more likely to get called out successfully: a tech company or a restaurant?)
I’ve noticed a pattern where people like Vassar will make extreme claims without much supporting evidence and people will respond with “wow, what an interesting guy” instead of asking for evidence. I’m trying to push back against that.
I can imagine you’d be tempted to rationalize that whatever pathological stuff is/was present at CFAR is also common in the general population / organizations in general.
RE: “Lots of upper middle class adults hardly know how to have conversations...”
I will let Anna speak for herself, but I have evidence of my own to bring… maybe not directly about the thing she’s saying but nearby things.
I have noticed friends who jumped up to upper middle class status due to suddenly coming into a lot of wealth (prob from crypto stuff). I noticed that their conversations got worse (from my POV).
In particular: They were more self-preoccupied. They discussed more banal things. They spent a lot of time optimizing things that mostly seemed trivial to me (like what to have for dinner). When I brought up more worldly topics of conversation, someone expressed a kind of “wow I haven’t thought about the world in such a long time, it’d be nice to think about the world more.” Their tone was a tad wistful and they looked at me like they could learn something from me, but also they weren’t going to try very hard and we both knew it. I felt like they were in a wealth/class bubble that insulated them from many of the world’s problems and suffering. It seemed like they’d lost touch with their real questions and deep inner longings. I don’t think this was as true of them before, but maybe I wasn’t paying sufficient attention before, I dunno.
It’s like their life path switched from ‘seeking’ to ‘maintaining’. They walked far enough, and they picked a nice spot, and now that’s where they at.
I used to work in tech. My coworkers were REALLY preoccupied with trivial things like Pokemon Go, sports, video games, what to eat/drink, new toys and gadgets, how to make more money, Marvel movies, career advancement. Almost to the point of obsession. It was like an adult playground atmosphere… pretty fun, pretty pleasant, and pretty banal. Our job was great. The people were great. The money was great. And I personally had to get the f out of there.
This isn’t to say that they aren’t capable of having ‘real conversations’ about the world at times. But on the day-to-day level, I sensed an overwhelming force trying to keep them from looking at what the world is actually like, the part they’re playing in it, what really matters, etc. It felt like a dream world.
They also tended to have an alcohol or drug ‘habit’ or ‘hobby’ of some kind. Pot or alcohol; take your pick.
My more NY-flavored / finance-or-marketing-or-whatever-flavored friends like to drink, own nice watches, wear nice suits, have nice apartments, etc. Different flavor from the West Coast tech scene, but the same thing going on. They appear happy, happier than before. But also… eh. Their preoccupations again seem not-very-alive and have an artificial smell. They seem a bit blocked from having interesting and life-changing thoughts.
I don’t really judge the people I am talking about. I am sad about the situation but don’t feel like they’re doing something wrong.
I think the upper middle class capitalist dream is not all it is cracked up to be, and I would encourage people to try it out if they want to… but also to get over it once they’re done trying it? It’s nice for a while, and I like my friends having nice things and having money and stuff. But I don’t think it’s very character-building or teaching them new things or answering their most important questions. I also don’t like the way it insulates people from noticing how much death, suffering, and injustice there is going on.
Oh yeah they also spent a lot of time trying to have the right or correct opinions. So they would certainly talk about ‘the world’ but mostly for the sake of having “right opinions” about it. Not so that they could necessarily, like, have insights into it or feel connected to what was happening. It was a game with not very high or real stakes for them. They tended to rehash the SAME arguments over and over with each other.
This all sounds super fascinating to me, but perhaps a new post would be better for this.
My current best guess is that some people are “intrinsically” interested in the world, and for others the interest is only “instrumental”. The intrinsically interested are learning things about the real world because it is fascinating and because it is real. The instrumentally interested are only learning about things they assume might be necessary for satisfying their material needs. Throwing lots of money at them will remove chains from the former, but will turn off the engine for the latter.
For me another shocking thing about people in tech is how few of them are actually interested in the tech. Again, seems to be this intrinsical/instrumental distinction. The former group studies Haskell or design patterns or whatever. The latter group is only interested in things that can currently increase their salary, and even there they are mostly looking for shortcuts. Twenty years ago, programmers were considered nerdy. These days, programmers who care about e.g. clean code are considered too nerdy by most programmers.
I also don’t like the way it insulates people from noticing how much death, suffering, and injustice there is going on.
I often communicate with people outside my bubble, so my personal wealth does not isolate me from hearing about their suffering. If I won a lottery, I would probably spend more time helping people, because that’s the type of thing I sometimes do, and I would now have more free time for that. I would expect this to be even stronger for any effective altruist.
(There is a voice in my head telling me that this all might be a fundamental attribution error, that I am assuming fixed underlying personality traits that only get better expressed as people get rich, and underestimate the effect of the environment, such as peer pressure of other rich people.)
Your next comment (people for whom having “right opinions” is super important) sounds to me like managers. Having an opinion different from other managers is a liability; it signals that you are either not flexible enough or can’t see what your superiors want you to think.
Bit of a nitpick, but FYI I think you’re using “worldly” here in almost the opposite of the way it’s usually used. It seems like you mean “weighty” or “philosophical” or something to do with the big questions in life. Whereas traditionally, the term means:
of or concerned with material values or ordinary life rather than a spiritual existence
On that definition I’d say it was your friends who wanted to talk about worldly stuff, while you wanted to push the conversation in a non-worldly direction! (As I understand, the meaning originally comes from contrasting “the world” and the church.)
Oh, hmmmmm. Sorry for lack of clarity. I don’t remember exactly what the topic I brought up was. I just know it wasn’t very ‘local’. Could have been philosophical / deep. Could have been geopolitical / global / big picture.
I used to think the ability to have deep conversations is an indicator of how “alive” a person is, but now I think that view is wrong. It’s better to look at what the person has done and is doing. Surprisingly there’s little correlation: I often come across people who are very measured in conversation, but turn out to have amazing skills and do amazing things.
Assuming that language is about coordination instead of object level world modeling, why should we be surprised that there’s little correlation between these two very different things?
Because object level world modeling is vastly easier and more unconstrained when you can draw on the sight of other minds, so a live world-modeler who can’t talk to people has something going wrong (whether in them or in the environment).
I also feel really frustrated that you wrote this, Anna. I think there are a number of obvious and significant disanalogies between the situations at Leverage versus MIRI/CFAR. There’s a lot to say here, but a few examples which seem especially salient:
To the best of my knowledge, the leadership of neither MIRI nor CFAR has ever slept with any subordinates, much less many of them.
While I think staff at MIRI and CFAR do engage in motivated reasoning sometimes wrt PR, neither org engaged in anything close to the level of obsessive, anti-epistemic reputational control alleged in Zoe’s post. MIRI and CFAR staff were not required to sign NDAs agreeing they wouldn’t talk badly about the org—in fact, at least in my experience with CFAR, staff much more commonly share criticism of the org than praise. CFAR staff were regularly encouraged to share their ideas at workshops and on LessWrong, to get public feedback. And when we did mess up, we tried quite hard to publicly and accurately describe our wrongdoing—e.g., Anna and I spent low-hundreds of hours investigating/thinking through the Brent affair, and tried so hard to avoid accidentally doing anti-epistemic reputational control (this was our most common topic of conversation during this process) that in my opinion, our writeup about it actually makes CFAR seem much more culpable than I think it was.
As I understand it, there were ~3 staff historically whose job descriptions involved debugging in some way which you, Anna, now feel uncomfortable with/think was fucky. But to the best of your knowledge, these situations caused much less harm than e.g. Zoe seems to have experienced, and the large majority of staff did not experience this—in general staff rarely explicitly debugged each other, and when it did happen it was clearly opt-in, and fairly symmetrical (e.g., in my personal conversations with you Anna, I’d guess the ratio of you something-like-debugging me to the reverse is maybe 3/2?).
CFAR put really a lot of time and effort into trying to figure out how to teach rationality techniques, and how to talk with people about x-risk, without accidentally doing something fucky to people’s psyches. Our training curriculum for workshop mentors includes extensive advice on ways to avoid accidentally causing psychological harm. Harm did happen sometimes, which was why our training emphasized it so heavily. But we really fucking tried, and my sense is that we actually did very well on the whole at establishing institutional and personal knowledge about how to be gentle with people in these situations; personally, it’s the skillset I’d most worry about the community losing if CFAR shut down and more events started being run by other orgs.
Insofar as you agree with the above, Anna, I’d appreciate you stating that clearly, since I think saying “the OP speaks for me” implies you think the core analogy described in the OP was non-misleading.
Yeah, sorry. I agree that my comment “the OP speaks for me” is leading a lot of people to false views that I should correct. It’s somehow tricky because there’s a different thing I worry will be obscured by my doing this, but I’ll do it anyhow as is correct and try to come back for that different thing later.
To the best of my knowledge, the leadership of neither MIRI nor CFAR has ever slept with a subordinate, much less many of them.
Agreed.
While I think staff at CFAR and MIRI probably engaged in motivated reasoning sometimes wrt PR, neither org engaged in anything close to the level of obsessive, anti-epistemic reputational control alleged in Zoe’s post. CFAR and MIRI staff were certainly not required to sign NDAs agreeing they wouldn’t talk badly about the org—in fact, in my experience CFAR staff much more commonly share criticism of the org than praise. CFAR staff were regularly encouraged to share their ideas at workshops and on LessWrong, to get public feedback. And when we did mess up, we tried extremely hard to publicly and accurately describe our wrongdoing—e.g., Anna and I personally spent hundreds of hours investigating/thinking about the Brent affair, and tried so hard to avoid accidentally doing anti-epistemic reputational control that in my opinion, our writeup about it actually makes CFAR seem much more culpable than I think it was.
I agree that there’s a large difference in both philosophy of how/whether to manage reputation, and amount of control exhibited/attempted about how staff would talk about the organizations, with Leverage doing a lot of that and CFAR doing less of it than most organizations.
As I understand it, there were ~3 staff historically whose job description involved debugging in some way which you, Anna, now feel uncomfortable with/think was fucky. But to the best of your knowledge, these situations caused much less harm than e.g. Zoe seems to have experienced, and the large majority of staff did not experience this—in general staff rarely explicitly debugged each other, and when it did happen it was clearly opt-in, and fairly symmetrical (e.g., in my personal conversations with you Anna, I’d guess the ratio of you something-like-debugging me to the reverse is maybe 3/2?).
I think this understates both how many people it happened with, and how fucky it sometimes was. (Also, it was job but not “job description”, although I think Zoe’s was “job description”). I think this one was actually worse in some of the early years, vs your model of it. My guess is indeed that it involved fewer hours than Zoe, and was overall less deliberately part of a dynamic quite as fucky as Zoe’s, but as I mentioned to you on the phone, an early peripheral staff member left CFAR for a mental institution in a way that seemed plausibly to do with how debugging and trials worked, and definitely to do with workplace stress of some sort, as well as with a preexisting condition they entered with and didn’t tell us about. (We would’ve handled this better later, I think.) There are some other situations that were also I think pretty fucked up, in the sense of “I think the average person would experience some horror/indignation if they took in what was happening.”
I can also think of stories of real scarring outside the three people I was counting.
I… do think it was considerably less weird looking, and less overtly fucked-up looking, than the descriptions I have (since writing my “this post speaks for me” comment) gotten of Leverage in the 2018-2019 era.
Also, most people at CFAR, especially in recent years, I think suffered none or nearly none of this. (I believe the same was true for parts of Leverage, though not sure.)
So, if we are playing the “compare how bad Leverage and CFAR are along each axis” game (which is not the main thing I took the OP to be doing, at all, nor the main thing I was trying to agree with, at all), I do think Leverage is worse than CFAR on this axis but I think the “per capita” damage of this sort that hit CFAR staff in the early years (“per capita” rather than cumulative, because Leverage had many more people) was maybe about a tenth of my best guess at what was up in the near-Zoe parts of Leverage in 2018-2019, which is a lot but, yes, different.
CFAR put really a lot of time and effort into trying to figure out how to teach rationality techniques, and how to talk with people about x-risk, without accidentally doing something fucky to people’s psyches. Our training curriculum for workshop mentors includes extensive advice on ways to avoid accidentally causing psychological harm. Harm did happen sometimes, which was why our training emphasized it so heavily. But we really fucking tried, and my sense is that we actually did very well on the whole at establishing institutional and personal knowledge about how to be gentle with people in these situations; personally, it’s the skillset I’d most worry about the community losing if CFAR shut down and more events started being run by other orgs.
We indeed put a lot of effort into this, and got some actual skill and good institutional habits out.
Perhaps this is an opportunity to create an internal document on “unhealthy behaviors” that would list the screwups and the lessons learned, and read it together regularly, like a safety training? (Analogically to how organizations that get their computers hacked or documents stolen, describe how it happened as a part of their safety training.) Perhaps with anonymous feedback whether someone has a concern that MIRI or CFAR is slipping into some bad pattern again.
Also, it might be useful to hire an external psychologist who would in regular intervals have a discussion with MIRI/CFAR employees. And to provide this document to the psychologist, so they know what risks to focus on. (Furthermore I think the psychologist should not be a rationalist; to provide a better outside view.)
For starters, someone could create the first version of the document by extracting information from this debate.
EDIT: Oops, on second reading of your comment, it seems like you already have something like this. Uhm, maybe a good opportunity to update/extend the document?
*
As a completely separate topic, it would be nice to have a table with the following columns: “Safety concern”, “What happened in MIRI/CFAR”, “What happened in Leverage (as far as we know)”, “Similarities”, “Differences”. But this is much less important, in long term.
I endorse Adam’s commentary, though I did not feel the frustration Eli and Adam report, possibly because I know Anna well enough that I reflexively did the caveating in my own brain rather than modeling the audience.
To the best of my knowledge, the leadership of neither MIRI nor CFAR has ever slept with a subordinate, much less many of them.
This issue doesn’t seem particularly important to me but the comparison you’re making is a good example of a more general problem I want to talk about.
My impression is that the decision structure of CFAR was much less legible & transparent than that of Leverage, so that it would be harder to determine who might be treated as subordinate to whom in what context. In addition, my impression from the years I was around is that Leverage didn’t preside over as much of an external scene, - Leverage followers had formalized roles as members of the organization, while CFAR had a “community,” many of whom were workshop alumni.
And when we did mess up, we tried extremely hard to publicly and accurately describe our wrongdoing—e.g., Anna and I personally spent hundreds of hours investigating/thinking about the Brent affair, and tried so hard to avoid accidentally doing anti-epistemic reputational control that in my opinion, our writeup about it actually makes CFAR seem much more culpable than I think it was.
Am I missing something here? The communication I read from CFAR seemed like it was trying to reveal as little as it could get away with, gradually saying more (and taking a harsher stance towards Brent) in response to public pressure, not like it was trying to help me, a reader, understand what had happened.
Anna and I personally spent hundreds of hours investigating/thinking about the Brent affair… our writeup about it...
Am I missing something here? The communication I read from CFAR seemed like it was trying to reveal as little as it could get away with...
FWIW, I think you and Adam are talking about two different pieces of communication. I think you are thinking of the communication leading up to the big community-wide discussion that happened in Sept 2018, while Adam is thinking specifically of CFAR’s follow-up communication months after that — in particular this post. (It would have been in between those two times when Adam and Anna did all that thinking that he was talking about.)
I agree manager/staff relations have often been less clear at CFAR than is typical. But I’m skeptical that’s relevant here, since as far as I know there aren’t really even borderline examples of this happening. The closest example to something like this I can think of is that staff occasionally invite their partners to attend or volunteer at workshops, which I think does pose some risk of fucky power dynamics, albeit dramatically less risk imo than would be posed by “the clear leader of an organization, who’s revered by staff as a world-historically important philosopher upon whose actions the fate of the world rests, and who has unilateral power to fire any of them, sleeps with many employees.”
Am I missing something here? The communication I read from CFAR seemed like it was trying to reveal as little as it could get away with, gradually saying more (and taking a harsher stance towards Brent) in response to public pressure, not like it was trying to help me, a reader, understand what had happened.
As lead author on the Brent post, I felt bummed reading this. I tried really hard to avoid letting my care for/interest in CFAR affect my descriptions of what happened, or my choices about what to describe. Anna and I spent quite large amounts of time—at least double-digit hours, I think probably triple-digit—searching for ways our cognition might be biased or motivated or PR-like, and trying to correct for that. We debated and introspected about it, ran drafts by friends of ours who seemed unusually likely to call us on bullshit, etc.
Looking back, my sense remains that we basically succeeded—i.e., that we described the situation about as accurately and neutrally as we could have. If I’m wrong about this… well, it wasn’t for lack of trying.
Looking back, my sense remains that we basically succeeded—i.e., that we described the situation about as accurately and neutrally as we could have. If I’m wrong about this… well, all I can say is that it wasn’t for lack of trying.
I think CFAR ultimately succeeded in providing a candid and good faith account of what went wrong, but the time it took to get there (i.e. 6 months between this and the initial update/apology) invites adverse inferences like those in the grandparent.
A lot of the information ultimately disclosed in March would definitely have been known to CFAR in September, such as Brent’s prior involvement as a volunteer/contractor for CFAR, his relationships/friendships with current staff, and the events as ESPR. The initial responses remained coy on these points, and seemed apt to give the misleading impression CFAR’s mistakes were (relatively) much milder than they in fact were. I (among many) contacted CFAR leadership to urge them to provide more candid and complete account when I discovered some of this further information independently.
I also think, similar to how it would be reasonable to doubt ‘utmost corporate candour’ back then given initial partial disclosure, it’s reasonable to doubt CFAR has addressed the shortcomings revealed given the lack of concrete follow-up. I also approached CFAR leadership when CFAR’s 2019 Progress Report and Future Plans initially made no mention of what happened with Brent, nor what CFAR intended to improve in response to it. What was added in is not greatly reassuring:
And after spending significant time investigating our mistakes with regard to Brent, we reformed our hiring, admissions and conduct policies, to reduce the likelihood such mistakes reoccur.
A cynic would note this is ‘marking your own homework’, but cynicism is unnecessary to recommend more self-scepticism. I don’t doubt the Brent situation indeed inspired a lot of soul searching and substantial, sincere efforts to improve. What is more doubtful (especially given the rest of the morass of comments) is whether these efforts actually worked. Although there is little prospect of satisfying me, more transparency over what exactly has changed—and perhaps third party oversight and review—may better reassure others.
It would help if they actually listed and gave examples of exactly what kind of mental manipulation they were doing to people other than telling them to take drugs. These comments seem to dance around the exactly details of what happened and only talk about the group dynamics between people as a result of these mysterious actions/events.
Anna, I feel frustrated that you wrote this. Unless I have severely misunderstood you, this seems extremely misleading.
For context, before this post was published Anna and I discussed the comparison between MIRI/CFAR and Leverage.
At that time, you, Anna, posited a high level dynamic involving “narrative pyramid schemes” accelerating, and then going bankrupt, at about the same time. I agreed that this seemed like it might have something to it, but emphasized that, despite some high level similarities, what happened at MIRI/CFAR was meaningfully different from, and much much less harmful than, what Zoe described in her post.
We then went through a specific operationalization of one of the specific claimed parallels (specifically the frequency and oppressiveness of superior-to-subordinate debugging), and you agreed that while the CFAR case was, quantitatively, an order of magnitude better than what Zoe describes. We talked more generally about some of the other parallels, and you generally agreed that the specific harms were much greater in the Leverage case.
(And just now, I talked with another CFAR staff member who reported that the two of you went point by point, and for each one you agreed that the CFAR/MIRI situation was much less bad than the Leverage case. [edit: I misunderstood. They only went through 5 points, out of many, but out of those 5 Anna agreed that the Leverage case was broadly worse.])
I think that you believe, as I do, that there were some high-level structural similarities between the dynamics at MIRI/CFAR and at Leverage, and also what happened at Leverage was substantially worse* than what happened at MIRI/CFAR.
Do you believe that?
If so, can you please clearly say so?
I feel like not clearly stating that second part is extremely and damagingly misleading.
What is at stake is not just the abstract dynamics, but also the concrete question of how alarmed, qualitatively, people around here should be. It seems to me that you, with this comment, are implying that it is appropriate to be about as alarmed by Zoe’s report of Leverage as by this description of MIRI/CFAR. Which seems wrong to me.
[Edit: * - This formally read “an order of magnitude worse”.
I think this is correct, for a number of common sense metrics (ie “there was at least 10x as many hours of superior-subordinate debugging at Leverage, where this seemed to be an institutionalized practice making up a lot of a person’s day, compared to CFAR, where this did happen sometimes what wasn’t a core feature of the org. (This is without taking into account the differences in how harmful those hours were. The worst case of which I’m aware of this happening at CFAR was less harmful than Zoe’s account.)
I think across most metrics named, Leverage had a worse or stronger version of thing, with a few exceptions. MIRI’s (but not CFAR’s, mostly) narrative had more urgency about it than Leverage’s for instance, because of AI timeline considerations, and overall the level of “intensity” or “pressure” around MIRI and Leverage might have been similar? I’m not close enough to either org to say with confidence.
But overall, I think it is weird to talk about “orders of magnitude” without referring to a specific metric, since it has the veneer of rigor without really adding much substance. I’m hoping that this edit adds some of that substance and I’m walking my claim back to the vaguer “substantially worse”, with the caveat that I am generally in favor of, and open to sharing more specific quantitative estimates on specific operationalizations if asked.]
Leverage_2018-2019 sounds considerably worse than Leverage 2013-2016.
My current guess is that if you took a random secular American to be your judge, or a random LWer, and you let them watch the life of a randomly chosen member of the Leverage psychology team from 2018-2019 (which I’m told is the worst part) and also of a randomly chosen staff member at either MIRI or CFAR, they would be at least 10x more horrified by the experience of the one in the Leverage psychology team.
I somehow don’t know how to say in my own person “was an order of magnitude worse”, but I can say the above. The reason I don’t know how to say “was an order of magnitude worse” is because it honestly looks to me (as to Jessica in the OP) like many places are pretty bad for many people, in the sense of degrading their souls via deceptions, manipulations, and other ethical violations. I’m not sure if this view of mine will sound over-the-top/dismissable or we-all-already-know-that/dismissible, or something else, but I have in mind such things as:
It seems to me that many many kids enter school with a desire to learn and an ability to trust their own mind, and leave school with a weird kind of “make sure you don’t get it wrong” that inhibits trying and doing. Some of this is normal aging, but my best guess is that an important chunk is more like cultural damage.
Many teenagers can do philosophy, stretch, try to think about the world. Most of the same folks at 30 or 40 can’t, outside of the ~one specific disciplines in which they’re a professional. They don’t let themselves.
Lots of upper middle class adults hardly know how to have conversations, of the “talk from the person inside who is actually home, asking what they want to know instead of staying safe, hitting new unpredictable thoughts/conversations” sense. This is a change from childhood. Again, this is probably partly aging, but I suspect cultural damage, and I’ve been told a couple times (including by folks who have no contact with Vassar or anyone else in this community) that this is less true for working class folks than for upper middle class folks, which if true is evidence for it being partly cultural damage though I should check this better.
Some staff IMO initially expect that folks at CFAR or Google or the FDA or wherever will be trying to do something real, and then come to later relate to it more like belief-in-belief, and to lots of other things too, with language coming to seem more like a mechanism for coordinating our belief-in-beliefs, and less like light with which one can talk and reason. And with things in general coming to seem kind of remote and as though you can’t really hope for anything real.
Anyhow. This essay wants to be larger than I’m willing to make this comment-reply before sleeping, so I’ll just keep doing it poorly/briefly, and hope to have more conversation later not necessarily under Jessica’s OP. But my best guess is that both CFAR of most of the last ten years, and the average workplace, are:
a) On the one hand, quite a bit less overtly hellish than the Leverage psychology teams of 2018-2019; but nevertheless maybe full of secret bits of despair and giving-up-on bits of our birthrights, in ways that are mostly not consciously noticed; b) More than 1/10th as damaging to most employees’ basic human capacities, compared to Leverage_2018-2019.
Why do I think b? Partly because of my observations on what happens to people in the broader world (including control groups of folks who do their own thing among good people and end up fine, but I might be rigging my data and playing “no true scottsmen” games to get rid of the rest, and misconstruing natural aging or something). And partly because I chatted with several people in the past week who spent time at Leverage, and they all seemed like they had intact souls, to me, although my soul-ometer is not necessarily that accurate etc.
But, anyhow, I agree that most people would see what you’re saying, I’m just seeing something else and I care about it and I’m sorry if I said it in a confusing/misleading way but it is actually pretty hard to talk about.
Epistemic status of all this: scratchwork, alas.
These claims seem rather extreme and unsupported to me:
“Lots of upper middle class adults hardly know how to have conversations...”
“the average workplace [is] more than 1/10th as damaging to most employees’ basic human capacities, compared to Leverage_2018-2019.”
I suggest if you write a toplevel post, you search for evidence for/against them.
Elaborating a bit on my reasons for skepticism:
It seems like for the past 10+ years, you’ve been mostly interacting with people in CFAR-adjacent contexts. I’m not sure what your source of knowledge is on “average” upper middle class adults/workplaces. My personal experience is normal people are comfortable having non-superficial conversations if you convince them you aren’t weird first, and normal workplaces are pretty much fine. (I might be overselecting on smaller companies where people have a sense of humor.)
A specific concrete piece of evidence: Joe Rogan has one of the world’s most popular podcasts, and the episodes I’ve heard very much seem to me like they’re “hitting new unpredictable thoughts”. Rogan is notorious for talking to guests about DMT, for instance.
The two observations seems a bit inconsistent, if you’ll grant that working class people generally have worse working conditions than upper middle class people—you’d expect them to experience more workplace abuse and therefore have more trauma. (In which context would an abusive boss be more likely to get called out successfully: a tech company or a restaurant?)
I’ve noticed a pattern where people like Vassar will make extreme claims without much supporting evidence and people will respond with “wow, what an interesting guy” instead of asking for evidence. I’m trying to push back against that.
I can imagine you’d be tempted to rationalize that whatever pathological stuff is/was present at CFAR is also common in the general population / organizations in general.
RE: “Lots of upper middle class adults hardly know how to have conversations...”
I will let Anna speak for herself, but I have evidence of my own to bring… maybe not directly about the thing she’s saying but nearby things.
I have noticed friends who jumped up to upper middle class status due to suddenly coming into a lot of wealth (prob from crypto stuff). I noticed that their conversations got worse (from my POV).
In particular: They were more self-preoccupied. They discussed more banal things. They spent a lot of time optimizing things that mostly seemed trivial to me (like what to have for dinner). When I brought up more worldly topics of conversation, someone expressed a kind of “wow I haven’t thought about the world in such a long time, it’d be nice to think about the world more.” Their tone was a tad wistful and they looked at me like they could learn something from me, but also they weren’t going to try very hard and we both knew it. I felt like they were in a wealth/class bubble that insulated them from many of the world’s problems and suffering. It seemed like they’d lost touch with their real questions and deep inner longings. I don’t think this was as true of them before, but maybe I wasn’t paying sufficient attention before, I dunno.
It’s like their life path switched from ‘seeking’ to ‘maintaining’. They walked far enough, and they picked a nice spot, and now that’s where they at.
I used to work in tech. My coworkers were REALLY preoccupied with trivial things like Pokemon Go, sports, video games, what to eat/drink, new toys and gadgets, how to make more money, Marvel movies, career advancement. Almost to the point of obsession. It was like an adult playground atmosphere… pretty fun, pretty pleasant, and pretty banal. Our job was great. The people were great. The money was great. And I personally had to get the f out of there.
This isn’t to say that they aren’t capable of having ‘real conversations’ about the world at times. But on the day-to-day level, I sensed an overwhelming force trying to keep them from looking at what the world is actually like, the part they’re playing in it, what really matters, etc. It felt like a dream world.
They also tended to have an alcohol or drug ‘habit’ or ‘hobby’ of some kind. Pot or alcohol; take your pick.
My more NY-flavored / finance-or-marketing-or-whatever-flavored friends like to drink, own nice watches, wear nice suits, have nice apartments, etc. Different flavor from the West Coast tech scene, but the same thing going on. They appear happy, happier than before. But also… eh. Their preoccupations again seem not-very-alive and have an artificial smell. They seem a bit blocked from having interesting and life-changing thoughts.
I don’t really judge the people I am talking about. I am sad about the situation but don’t feel like they’re doing something wrong.
I think the upper middle class capitalist dream is not all it is cracked up to be, and I would encourage people to try it out if they want to… but also to get over it once they’re done trying it? It’s nice for a while, and I like my friends having nice things and having money and stuff. But I don’t think it’s very character-building or teaching them new things or answering their most important questions. I also don’t like the way it insulates people from noticing how much death, suffering, and injustice there is going on.
Oh yeah they also spent a lot of time trying to have the right or correct opinions. So they would certainly talk about ‘the world’ but mostly for the sake of having “right opinions” about it. Not so that they could necessarily, like, have insights into it or feel connected to what was happening. It was a game with not very high or real stakes for them. They tended to rehash the SAME arguments over and over with each other.
This all sounds super fascinating to me, but perhaps a new post would be better for this.
My current best guess is that some people are “intrinsically” interested in the world, and for others the interest is only “instrumental”. The intrinsically interested are learning things about the real world because it is fascinating and because it is real. The instrumentally interested are only learning about things they assume might be necessary for satisfying their material needs. Throwing lots of money at them will remove chains from the former, but will turn off the engine for the latter.
For me another shocking thing about people in tech is how few of them are actually interested in the tech. Again, seems to be this intrinsical/instrumental distinction. The former group studies Haskell or design patterns or whatever. The latter group is only interested in things that can currently increase their salary, and even there they are mostly looking for shortcuts. Twenty years ago, programmers were considered nerdy. These days, programmers who care about e.g. clean code are considered too nerdy by most programmers.
I often communicate with people outside my bubble, so my personal wealth does not isolate me from hearing about their suffering. If I won a lottery, I would probably spend more time helping people, because that’s the type of thing I sometimes do, and I would now have more free time for that. I would expect this to be even stronger for any effective altruist.
(There is a voice in my head telling me that this all might be a fundamental attribution error, that I am assuming fixed underlying personality traits that only get better expressed as people get rich, and underestimate the effect of the environment, such as peer pressure of other rich people.)
Your next comment (people for whom having “right opinions” is super important) sounds to me like managers. Having an opinion different from other managers is a liability; it signals that you are either not flexible enough or can’t see what your superiors want you to think.
Bit of a nitpick, but FYI I think you’re using “worldly” here in almost the opposite of the way it’s usually used. It seems like you mean “weighty” or “philosophical” or something to do with the big questions in life. Whereas traditionally, the term means:
On that definition I’d say it was your friends who wanted to talk about worldly stuff, while you wanted to push the conversation in a non-worldly direction! (As I understand, the meaning originally comes from contrasting “the world” and the church.)
Oh, hmmmmm. Sorry for lack of clarity. I don’t remember exactly what the topic I brought up was. I just know it wasn’t very ‘local’. Could have been philosophical / deep. Could have been geopolitical / global / big picture.
A couple books suggesting that white collar workplaces are more traumatic than blue collar ones are Moral Mazes (cited by Jessica) and Bullshit Jobs.
I used to think the ability to have deep conversations is an indicator of how “alive” a person is, but now I think that view is wrong. It’s better to look at what the person has done and is doing. Surprisingly there’s little correlation: I often come across people who are very measured in conversation, but turn out to have amazing skills and do amazing things.
Assuming that language is about coordination instead of object level world modeling, why should we be surprised that there’s little correlation between these two very different things?
Because object level world modeling is vastly easier and more unconstrained when you can draw on the sight of other minds, so a live world-modeler who can’t talk to people has something going wrong (whether in them or in the environment).
I also feel really frustrated that you wrote this, Anna. I think there are a number of obvious and significant disanalogies between the situations at Leverage versus MIRI/CFAR. There’s a lot to say here, but a few examples which seem especially salient:
To the best of my knowledge, the leadership of neither MIRI nor CFAR has ever slept with any subordinates, much less many of them.
While I think staff at MIRI and CFAR do engage in motivated reasoning sometimes wrt PR, neither org engaged in anything close to the level of obsessive, anti-epistemic reputational control alleged in Zoe’s post. MIRI and CFAR staff were not required to sign NDAs agreeing they wouldn’t talk badly about the org—in fact, at least in my experience with CFAR, staff much more commonly share criticism of the org than praise. CFAR staff were regularly encouraged to share their ideas at workshops and on LessWrong, to get public feedback. And when we did mess up, we tried quite hard to publicly and accurately describe our wrongdoing—e.g., Anna and I spent low-hundreds of hours investigating/thinking through the Brent affair, and tried so hard to avoid accidentally doing anti-epistemic reputational control (this was our most common topic of conversation during this process) that in my opinion, our writeup about it actually makes CFAR seem much more culpable than I think it was.
As I understand it, there were ~3 staff historically whose job descriptions involved debugging in some way which you, Anna, now feel uncomfortable with/think was fucky. But to the best of your knowledge, these situations caused much less harm than e.g. Zoe seems to have experienced, and the large majority of staff did not experience this—in general staff rarely explicitly debugged each other, and when it did happen it was clearly opt-in, and fairly symmetrical (e.g., in my personal conversations with you Anna, I’d guess the ratio of you something-like-debugging me to the reverse is maybe 3/2?).
CFAR put really a lot of time and effort into trying to figure out how to teach rationality techniques, and how to talk with people about x-risk, without accidentally doing something fucky to people’s psyches. Our training curriculum for workshop mentors includes extensive advice on ways to avoid accidentally causing psychological harm. Harm did happen sometimes, which was why our training emphasized it so heavily. But we really fucking tried, and my sense is that we actually did very well on the whole at establishing institutional and personal knowledge about how to be gentle with people in these situations; personally, it’s the skillset I’d most worry about the community losing if CFAR shut down and more events started being run by other orgs.
Insofar as you agree with the above, Anna, I’d appreciate you stating that clearly, since I think saying “the OP speaks for me” implies you think the core analogy described in the OP was non-misleading.
Yeah, sorry. I agree that my comment “the OP speaks for me” is leading a lot of people to false views that I should correct. It’s somehow tricky because there’s a different thing I worry will be obscured by my doing this, but I’ll do it anyhow as is correct and try to come back for that different thing later.
Agreed.
I agree that there’s a large difference in both philosophy of how/whether to manage reputation, and amount of control exhibited/attempted about how staff would talk about the organizations, with Leverage doing a lot of that and CFAR doing less of it than most organizations.
I think this understates both how many people it happened with, and how fucky it sometimes was. (Also, it was job but not “job description”, although I think Zoe’s was “job description”). I think this one was actually worse in some of the early years, vs your model of it. My guess is indeed that it involved fewer hours than Zoe, and was overall less deliberately part of a dynamic quite as fucky as Zoe’s, but as I mentioned to you on the phone, an early peripheral staff member left CFAR for a mental institution in a way that seemed plausibly to do with how debugging and trials worked, and definitely to do with workplace stress of some sort, as well as with a preexisting condition they entered with and didn’t tell us about. (We would’ve handled this better later, I think.) There are some other situations that were also I think pretty fucked up, in the sense of “I think the average person would experience some horror/indignation if they took in what was happening.”
I can also think of stories of real scarring outside the three people I was counting.
I… do think it was considerably less weird looking, and less overtly fucked-up looking, than the descriptions I have (since writing my “this post speaks for me” comment) gotten of Leverage in the 2018-2019 era.
Also, most people at CFAR, especially in recent years, I think suffered none or nearly none of this. (I believe the same was true for parts of Leverage, though not sure.)
So, if we are playing the “compare how bad Leverage and CFAR are along each axis” game (which is not the main thing I took the OP to be doing, at all, nor the main thing I was trying to agree with, at all), I do think Leverage is worse than CFAR on this axis but I think the “per capita” damage of this sort that hit CFAR staff in the early years (“per capita” rather than cumulative, because Leverage had many more people) was maybe about a tenth of my best guess at what was up in the near-Zoe parts of Leverage in 2018-2019, which is a lot but, yes, different.
We indeed put a lot of effort into this, and got some actual skill and good institutional habits out.
Perhaps this is an opportunity to create an internal document on “unhealthy behaviors” that would list the screwups and the lessons learned, and read it together regularly, like a safety training? (Analogically to how organizations that get their computers hacked or documents stolen, describe how it happened as a part of their safety training.) Perhaps with anonymous feedback whether someone has a concern that MIRI or CFAR is slipping into some bad pattern again.
Also, it might be useful to hire an external psychologist who would in regular intervals have a discussion with MIRI/CFAR employees. And to provide this document to the psychologist, so they know what risks to focus on. (Furthermore I think the psychologist should not be a rationalist; to provide a better outside view.)
For starters, someone could create the first version of the document by extracting information from this debate.
EDIT: Oops, on second reading of your comment, it seems like you already have something like this. Uhm, maybe a good opportunity to update/extend the document?
*
As a completely separate topic, it would be nice to have a table with the following columns: “Safety concern”, “What happened in MIRI/CFAR”, “What happened in Leverage (as far as we know)”, “Similarities”, “Differences”. But this is much less important, in long term.
I endorse Adam’s commentary, though I did not feel the frustration Eli and Adam report, possibly because I know Anna well enough that I reflexively did the caveating in my own brain rather than modeling the audience.
This issue doesn’t seem particularly important to me but the comparison you’re making is a good example of a more general problem I want to talk about.
My impression is that the decision structure of CFAR was much less legible & transparent than that of Leverage, so that it would be harder to determine who might be treated as subordinate to whom in what context. In addition, my impression from the years I was around is that Leverage didn’t preside over as much of an external scene, - Leverage followers had formalized roles as members of the organization, while CFAR had a “community,” many of whom were workshop alumni.
Am I missing something here? The communication I read from CFAR seemed like it was trying to reveal as little as it could get away with, gradually saying more (and taking a harsher stance towards Brent) in response to public pressure, not like it was trying to help me, a reader, understand what had happened.
FWIW, I think you and Adam are talking about two different pieces of communication. I think you are thinking of the communication leading up to the big community-wide discussion that happened in Sept 2018, while Adam is thinking specifically of CFAR’s follow-up communication months after that — in particular this post. (It would have been in between those two times when Adam and Anna did all that thinking that he was talking about.)
Yeah, this was the post I meant.
I agree manager/staff relations have often been less clear at CFAR than is typical. But I’m skeptical that’s relevant here, since as far as I know there aren’t really even borderline examples of this happening. The closest example to something like this I can think of is that staff occasionally invite their partners to attend or volunteer at workshops, which I think does pose some risk of fucky power dynamics, albeit dramatically less risk imo than would be posed by “the clear leader of an organization, who’s revered by staff as a world-historically important philosopher upon whose actions the fate of the world rests, and who has unilateral power to fire any of them, sleeps with many employees.”
As lead author on the Brent post, I felt bummed reading this. I tried really hard to avoid letting my care for/interest in CFAR affect my descriptions of what happened, or my choices about what to describe. Anna and I spent quite large amounts of time—at least double-digit hours, I think probably triple-digit—searching for ways our cognition might be biased or motivated or PR-like, and trying to correct for that. We debated and introspected about it, ran drafts by friends of ours who seemed unusually likely to call us on bullshit, etc.
Looking back, my sense remains that we basically succeeded—i.e., that we described the situation about as accurately and neutrally as we could have. If I’m wrong about this… well, it wasn’t for lack of trying.
I think CFAR ultimately succeeded in providing a candid and good faith account of what went wrong, but the time it took to get there (i.e. 6 months between this and the initial update/apology) invites adverse inferences like those in the grandparent.
A lot of the information ultimately disclosed in March would definitely have been known to CFAR in September, such as Brent’s prior involvement as a volunteer/contractor for CFAR, his relationships/friendships with current staff, and the events as ESPR. The initial responses remained coy on these points, and seemed apt to give the misleading impression CFAR’s mistakes were (relatively) much milder than they in fact were. I (among many) contacted CFAR leadership to urge them to provide more candid and complete account when I discovered some of this further information independently.
I also think, similar to how it would be reasonable to doubt ‘utmost corporate candour’ back then given initial partial disclosure, it’s reasonable to doubt CFAR has addressed the shortcomings revealed given the lack of concrete follow-up. I also approached CFAR leadership when CFAR’s 2019 Progress Report and Future Plans initially made no mention of what happened with Brent, nor what CFAR intended to improve in response to it. What was added in is not greatly reassuring:
A cynic would note this is ‘marking your own homework’, but cynicism is unnecessary to recommend more self-scepticism. I don’t doubt the Brent situation indeed inspired a lot of soul searching and substantial, sincere efforts to improve. What is more doubtful (especially given the rest of the morass of comments) is whether these efforts actually worked. Although there is little prospect of satisfying me, more transparency over what exactly has changed—and perhaps third party oversight and review—may better reassure others.
It would help if they actually listed and gave examples of exactly what kind of mental manipulation they were doing to people other than telling them to take drugs. These comments seem to dance around the exactly details of what happened and only talk about the group dynamics between people as a result of these mysterious actions/events.