Thank you for writing this, Jessica. First, you’ve had some miserable experiences in the last several years, and regardless of everything else, those times sound terrifying and awful. You have my deep sympathy.
Regardless of my seeing a large distinction between the Leverage situation and MIRI/CFAR, I agree with Jessica that this is a good time to revisit the safety of various orgs in the rationality/EA space.
I almost perfectly overlapped with Jessica at MIRI from March 2015 to June 2017. (Yes, this uniquely identifies me. Don’t use my actual name here anyway, please.) So I think I can speak to a great deal of this.
I’ll run down a summary of the specifics first (or at least, the specifics I know enough about to speak meaningfully), and then at the end discuss what I see overall.
Claim: People in and adjacent to MIRI/CFAR manifest major mental health problems, significantly more often than the background rate.
I think this is true; I believe I know two of the first cases to which Jessica refers; and I’m probably not plugged-in enough socially to know the others. And then there’s the Ziz catastrophe.
Claim: Eliezer and Nate updated sharply toward shorter timelines, other MIRI researchers became similarly convinced, and they repeatedly tried to persuade Jessica and others.
This is true, but non-nefarious in my genuine opinion, because it’s a genuine belief and because given that belief, you’ll have better odds of success if the whole team at least takes the hypothesis quite seriously.
(As for me, I’ve stably been at a point where near-term AGI wouldn’t surprise me much, but the lack of it also wouldn’t surprise me much. That’s all it takes, really, to be worried about near-term AGI.)
Claim: MIRI started getting secretive about their research.
This is true, to some extent. Nate and Eliezer discussed with the team that some things might have to be kept secret, and applied some basic levels of it to things we thought at the time might be AGI-relevant instead of only FAI-relevant. I think that here, the concern was less about AGI timelines and more about the multipolar race caused by DeepMind vs OpenAI. Basically any new advance gets deployed immediately in our current world.
However, I don’t recall ever being told I’m not allowed to know what someone else is working on, at least in broad strokes. Maybe my memory is faulty here, but it diverges from Jessica’s.
(I was sometimes coy about whether I knew anything secret or not, in true glomarization fashion; I hope this didn’t contribute to that feeling.)
There are surely things that Eliezer and Nate only wanted to discuss with each other, or with a specific researcher or two.
Claim: MIRI had rarity narratives around itself and around Eliezer in particular.
This is true. It would be weird if, given MIRI’s reason for being, it didn’t at least have the institutional rarity narrative—if one believed somebody else were just as capable of causing AI to be Friendly, clearly one should join their project instead of starting one’s own.
About Eliezer, there was a large but not infinite rarity narrative. We sometimes joked about the “bus factor”: if researcher X were hit by a bus, how much would the chance of success drop? Setting aside that this is a ridiculous and somewhat mean thing to joke about, the usual consensus was that Eliezer’s bus quotient was the highest one but that a couple of MIRI’s researchers put together exceeded it. (Nate’s was also quite high.)
(My expectation is that the same would not have been said about Geoff within Leverage.)
Claim: Working at MIRI/CFAR made it harder to connect with people outside the community.
There’s an extent to which this is true of any community that includes an idealistic job (i.e. a paid political activist probably has likeminded friends and finds it a bit more difficult to connect outside that circle). Is it true beyond that?
Not for me, at least. I maintained my ties with the other community I’d been plugged into (social dancing) and kept in good touch with my family (it helps that I have a really good family). As with the above example, the social path of least resistance would have been to just be friends with the same network of people in one’s work orbit, but there wasn’t anything beyond that level of gravity in effect for me.
Claim: CFAR got way too far into Shiny-Woo-Adjacent-Flavor-Of-The-Week.
This is a unfair framing… because I agree with Jessica’s claim 100%. Besides Kegan Levels and the MAPLE dalliance, there was the Circling phase and probably much else I wasn’t around for.
As for causes, I’ve been of the opinion that Anna Salamon has a lot of strengths around communicating ideas, but that her hiring has had as many hits as misses. There’s massive churn, people come in with their Big Ideas and nobody to stop them, and also people come in who aren’t in a good emotional place for their responsibilities. I think CFAR would be better off if Anna delegated hiring to someone else. [EDIT: Vaniver corrects me to say that Pete Michaud has been mostly in charge of hiring for the past several years, in which case I’m criticizing him rather than Anna for any bad hiring decisions during that time.]
Overall Thoughts
Essentially, I think there’s one big difference between issues with MIRI/CFAR and issues at Leverage:
The actions of CFAR/MIRI harmed people unintentionally, as evidenced by the result that people burned out and left quickly and with high frequency. The churn, especially in CFAR, hurt the mission, so it was definitely not the successful result of any strategic process.
Geoff Anders and others at Leverage harmed people intentionally, in ways that were intended to maintain control over those people. And to a large extent, that seems to have succeeded until Leverage fell apart.
Specifically, [accidentally triggering psychotic mental states by conveying a strange but honestly held worldview without adding adequate safeties] is different from [intentionally triggering psychotic mental states in order to pull people closer and prevent them from leaving], which is Zoe’s accusation. Even if it’s possible for a mental breakdown to be benign under the right circumstances, and even if an unplanned one is more likely to result in very very wrong circumstances, I’m far more terrified of a group that strategically plans for its members to have psychosis with the intent of molding those members further toward the group’s mission.
Unintentional harm is still harm, of course! It might have even been greater harm in total! But it makes a big difference when it comes to assessing how realistic a project of reform might be.
There are surely some deep reforms along these lines that CFAR/MIRI must consider. For one thing: scrupulosity, in the context of AI safety, seems to be a common thread in several of these breakdowns. I’ve taken this seriously enough in the past to post extensively on it here. I’d like CFAR/MIRI leadership to carefully update on how scrupulosity hurts both their people and their mission, and think about changes beyond surface-level things like adding a curriculum on scrupulosity. The actual incentives ought to change.
Finally, a good amount of Jessica’s post (similarly to Zoe’s post) concerns her inner experiences, on which she is the undisputed expert. I’m not ignoring those parts above. I just can’t say anything about them, merely that as a third person observer it’s much easier to discuss the external realities than the internal ones. (Likewise with Zoe and Leverage.)
: People in and adjacent to MIRI/CFAR manifest major mental health problems, significantly more often than the background rate.
I think this is true
My main complaint about this and the Leverage post is the lack of base-rate data. How many people develop mental health problems in a) normal companies, b) startups, c) small non-profits, d) cults/sects? So far, all I have seen are two cases. And in the startups I have worked at, I would also have been able to find mental health cases that could be tied to the company narrative. Humans being human narratives get woven. And the internet being the internet, some will get blown out of proportion. That doesn’t diminish the personal experience at all. I am updating only slightly on CFAR or MIRI. And basically not at all on “things look better from the outside than from the inside.”
In particular, I want to remind people here that something like 30-40% of grad students at top universities have either clinically diagnosed depression or anxiety (link). I think given the kind of undirected, often low-paid, work that many have been doing for the last decade, I think that’s the right reference class to draw from, and my current guess is we are roughly at that same level, or slightly below it (which is a crazy high number, and I think should give us a lot of pause).
I want to remind people here that something like 30-40% of grad students at top universities have either clinically diagnosed [emphasis mine] depression or anxiety (link)
I’m confused about how you got to this conclusion, and think it is most likely false. Neither your link, the linked study, or the linked meta-analysis in the linked study of your link says this. Instead the abstract of the linked^3 meta-analysis says:
Among 16 studies reporting the prevalence of clinically significant symptoms of depression across 23,469 Ph.D. students, the pooled estimate of the proportion of students with depression was 0.24 (95% confidence interval [CI], 0.18-0.31; I2 = 98.75%). In a meta-analysis of the nine studies reporting the prevalence of clinically significant symptoms of anxiety across 15,626 students, the estimated proportion of students with anxiety was 0.17 (95% CI, 0.12-0.23; I2 = 98.05%).
Further, the discussion section of the linked^3 study emphasizes:
While validated screening instruments tend to over-identify cases of depression (relative to structured clinical interviews) by approximately a factor of two67,68, our findings nonetheless point to a major public health problem among Ph.D. students.
So I think there is at least two things going on here:
Most people with clinically significant significant symptoms do not go get diagnosed, so “clinically significant symptoms of” depression/anxiety is a noticeably lower bar than “actually clinically diagnosed”
As implied in the quoted discussion above, if everybody were to seek diagnosis, only ~half of the rate of symptomatic people would be clinically diagnosed as having depression/anxiety.
For those keeping score, this is ~12% for depression and 8.5% for anxiety, with some error bars.
Separately, I also think:
my current guess is we are roughly at that same level, or slightly below it
is wrong. My guess is that xrisk reducers have worse mental health on average compared to grad students. (I also believe this, with lower confidence, about people working in other EA cause areas like animal welfare, global poverty, or non-xrisk longtermism, as well as serious rationalists who aren’t professionally involved in EA cause areas).
Note that the pooled prevalence is 24% (CI 18-31). But it differs a lot across studies, symptoms, and location. In the individual studies, the range is really from zero to 50% (or rather to 38% if you exclude a study with only 6 participants). I think a suitable reference class would be the University of California which has 3,190 participants and a prevalence of 38%.
Sorry, am I misunderstanding something? I think taking “clinically significant symptoms”, specific to the UC system, as a given is wrong because it did not directly address either of my two criticisms:
1. Clinically significant symptoms =/= clinically diagnosed even in worlds where there is a 1:1 relationship between clinically significant symptoms and would have been clinically diagnosed, as many people do not get diagnosed
2. Clinically significant symptoms do not have a 1:1 relationship with would have been clinically diagnosed.
Well, I agree that the actual prevalence you have in mind would be roughly half of 38% i.e. ~20%. That is still much higher than the 12% you arrived at. And either value is so high that there is little surprise some severe episodes of some people happened in a 5-year frame.
The UC Berkeley study was the one that I had cached in my mind as generating this number. I will reread it later today to make sure that it’s right, but it sure seems like the most relevant reference class, given the same physical location.
I had a look at the situation in Germany and it doesn’t look much better. 17% of students are diagnosed with at least one psychical disorder. This is based on the health records of all students insured by one of the largest public health insurers in Germany (about ten percent of the population):
I feel like the paragraph you cited just seems like the straightforward explanation of where my belief comes from?
Among 16 studies reporting the prevalence of clinically significant symptoms of depression across 23,469 Ph.D. students, the pooled estimate of the proportion of students with depression was 0.24 (95% confidence interval [CI], 0.18–0.31; I2 = 98.75%). In a meta-analysis of the nine studies reporting the prevalence of clinically significant symptoms of anxiety across 15,626 students, the estimated proportion of students with anxiety was 0.17 (95% CI, 0.12–0.23; I2 = 98.05%)
24% of people have depression, 17% have anxiety, resulting in something like 30%-40% having one or the other.
I did not remember the section about the screening instruments over-identifying cases of depression/anxiety by approximately a factor of two, which definitely cuts down my number, and I should have adjusted it in my above comment. I do think that factor of ~2 does maybe make me think that we are doing a bit worse than grad students, though I am not super sure.
Sorry, maybe this is too nitpicky, but clinically significant symptoms =/= clinically diagnosed, even in worlds where the clinically significant symptoms are severe enough to be diagnosed as such.
If you instead said in “population studies 30-40% of graduate students have anxiety or depression severe enough to be clinically diagnosed as such were they to seek diagnosis” then I think this will be a normal misreading from not jumping through enough links.
Put another way, if someone in mid-2020 told me that they had symptomatic covid and was formally diagnosed with covid, I would expect that they had worse symptoms than someone who said they had covid symptoms and later tested for covid antibodies. This is because jumping through the hoops to get a clinical diagnosis is nontrivial Bayesian evidence of severity and not just certainty, under most circumstances, and especially when testing is limited and/or gatekeeped (which is true for many parts of the world for covid in 2020, and is usually true in the US for mental health).
I think CFAR would be better off if Anna delegated hiring to someone else.
I think Pete did (most of?) the hiring as soon as he became ED, so I think this has been the state of CFAR for a while (while I think Anna has also been able to hire people she wanted to hire).
It’s always been a somewhat group-involved process, but yes, I was primarily responsible for hiring for roughly 2016 through the end of 2017, then it would have been Tim. But again, it’s a small org and always involved some involvement of the whole group.
Without denying that it is a small org and staff usually have some input over hiring, that input is usually informal.
My understanding is that in the period when Anna was ED, there was an explicit all-staff discussion when they were considering a hire (after the person had done a trial?). In the Pete era, I’m sure Pete asked for staff members’ opinions, and if (for instance) I sent him an email with my thoughts on a potential hire, he would take that info into account, but there was not institutional group meeting.
if one believed somebody else were just as capable of causing AI to be Friendly, clearly one should join their project instead of starting one’s own.
Nitpicking: there are reasons to have multiple projects, for example it’s convenient to be in the same geographic location but not anyone can relocate to any place.
Sure—and MIRI/FHI are a decent complement to each other, the latter providing a respectable academic face to weird ideas.
Generally though, it’s far more productive to have ten top researchers in the same org rather than having five orgs each with two top researchers and a couple of others to round them out. Geography is a secondary concern to that.
A “secondary concern” in the sense that, we should work remotely? Or in the sense that everyone should relocate? Because the latter is unrealistic: people have families, friends, communities, not anyone can uproot themself.
A secondary concern in that it’s better to have one org that has some people in different locations, but everyone communicating heavily, than to have two separate organizations.
I think this is much more complex than you’re assuming. As a sketch of why, costs of communication scale poorly, and the benefits of being small and coordinating centrally often beats the costs imposed by needing to run everything as one organization. (This is why people advise startups to outsource non-central work.)
AFAICT, Anthropic is not an existential AI safety org per se, they’re just doing a very particular type of research which might help with existential safety. But also, why do you think they don’t require physical presence?
If you’re asking why I believe that they don’t require presence, I’ve been interviewing with them and that’s my understanding from talking with them. The first line of copy on their website is
Anthropic is an AI safety and research company that’s working to build reliable, interpretable, and steerable AI systems.
If you’re asking why I believe that they don’t require presence, I’ve been interviewing with them and that’s my understanding from talking with them.
Are you talking about “you can work from home and come to the office occasionally”, or “you can live on a different continent”?
Sounds pretty much like a safety org to me.
I found no mention of existential risk on their web page. They seem to be a commercial company, aiming at short-to-mid-term applications. I doubt they have any intention to do e.g. purely theoretical research, especially if it has no applications to modern systems. So, what they do can still be meritorious and relevant to reducing existential risk. But, the context of this discussion is: can we replace all AI safety orgs by just one org. And, Anthropic is too specialized to serve such a role.
I believe Anthropic doesn’t expect its employees to be in the office every day, but I think this is more pandemic-related than it is a deliberate organizational design choice; my guess is that most Anthropic employees will be in the office a year from now.
Thank you for writing this, Jessica. First, you’ve had some miserable experiences in the last several years, and regardless of everything else, those times sound terrifying and awful. You have my deep sympathy.
Regardless of my seeing a large distinction between the Leverage situation and MIRI/CFAR, I agree with Jessica that this is a good time to revisit the safety of various orgs in the rationality/EA space.
I almost perfectly overlapped with Jessica at MIRI from March 2015 to June 2017. (Yes, this uniquely identifies me. Don’t use my actual name here anyway, please.) So I think I can speak to a great deal of this.
I’ll run down a summary of the specifics first (or at least, the specifics I know enough about to speak meaningfully), and then at the end discuss what I see overall.
Claim: People in and adjacent to MIRI/CFAR manifest major mental health problems, significantly more often than the background rate.
I think this is true; I believe I know two of the first cases to which Jessica refers; and I’m probably not plugged-in enough socially to know the others. And then there’s the Ziz catastrophe.
Claim: Eliezer and Nate updated sharply toward shorter timelines, other MIRI researchers became similarly convinced, and they repeatedly tried to persuade Jessica and others.
This is true, but non-nefarious in my genuine opinion, because it’s a genuine belief and because given that belief, you’ll have better odds of success if the whole team at least takes the hypothesis quite seriously.
(As for me, I’ve stably been at a point where near-term AGI wouldn’t surprise me much, but the lack of it also wouldn’t surprise me much. That’s all it takes, really, to be worried about near-term AGI.)
Claim: MIRI started getting secretive about their research.
This is true, to some extent. Nate and Eliezer discussed with the team that some things might have to be kept secret, and applied some basic levels of it to things we thought at the time might be AGI-relevant instead of only FAI-relevant. I think that here, the concern was less about AGI timelines and more about the multipolar race caused by DeepMind vs OpenAI. Basically any new advance gets deployed immediately in our current world.
However, I don’t recall ever being told I’m not allowed to know what someone else is working on, at least in broad strokes. Maybe my memory is faulty here, but it diverges from Jessica’s.
(I was sometimes coy about whether I knew anything secret or not, in true glomarization fashion; I hope this didn’t contribute to that feeling.)
There are surely things that Eliezer and Nate only wanted to discuss with each other, or with a specific researcher or two.
Claim: MIRI had rarity narratives around itself and around Eliezer in particular.
This is true. It would be weird if, given MIRI’s reason for being, it didn’t at least have the institutional rarity narrative—if one believed somebody else were just as capable of causing AI to be Friendly, clearly one should join their project instead of starting one’s own.
About Eliezer, there was a large but not infinite rarity narrative. We sometimes joked about the “bus factor”: if researcher X were hit by a bus, how much would the chance of success drop? Setting aside that this is a ridiculous and somewhat mean thing to joke about, the usual consensus was that Eliezer’s bus quotient was the highest one but that a couple of MIRI’s researchers put together exceeded it. (Nate’s was also quite high.)
(My expectation is that the same would not have been said about Geoff within Leverage.)
Claim: Working at MIRI/CFAR made it harder to connect with people outside the community.
There’s an extent to which this is true of any community that includes an idealistic job (i.e. a paid political activist probably has likeminded friends and finds it a bit more difficult to connect outside that circle). Is it true beyond that?
Not for me, at least. I maintained my ties with the other community I’d been plugged into (social dancing) and kept in good touch with my family (it helps that I have a really good family). As with the above example, the social path of least resistance would have been to just be friends with the same network of people in one’s work orbit, but there wasn’t anything beyond that level of gravity in effect for me.
Claim: CFAR got way too far into Shiny-Woo-Adjacent-Flavor-Of-The-Week.
This is a unfair framing… because I agree with Jessica’s claim 100%. Besides Kegan Levels and the MAPLE dalliance, there was the Circling phase and probably much else I wasn’t around for.
As for causes, I’ve been of the opinion that Anna Salamon has a lot of strengths around communicating ideas, but that her hiring has had as many hits as misses. There’s massive churn, people come in with their Big Ideas and nobody to stop them, and also people come in who aren’t in a good emotional place for their responsibilities. I think CFAR would be better off if Anna delegated hiring to someone else. [EDIT: Vaniver corrects me to say that Pete Michaud has been mostly in charge of hiring for the past several years, in which case I’m criticizing him rather than Anna for any bad hiring decisions during that time.]
Overall Thoughts
Essentially, I think there’s one big difference between issues with MIRI/CFAR and issues at Leverage:
The actions of CFAR/MIRI harmed people unintentionally, as evidenced by the result that people burned out and left quickly and with high frequency. The churn, especially in CFAR, hurt the mission, so it was definitely not the successful result of any strategic process.
Geoff Anders and others at Leverage harmed people intentionally, in ways that were intended to maintain control over those people. And to a large extent, that seems to have succeeded until Leverage fell apart.
Specifically, [accidentally triggering psychotic mental states by conveying a strange but honestly held worldview without adding adequate safeties] is different from [intentionally triggering psychotic mental states in order to pull people closer and prevent them from leaving], which is Zoe’s accusation. Even if it’s possible for a mental breakdown to be benign under the right circumstances, and even if an unplanned one is more likely to result in very very wrong circumstances, I’m far more terrified of a group that strategically plans for its members to have psychosis with the intent of molding those members further toward the group’s mission.
Unintentional harm is still harm, of course! It might have even been greater harm in total! But it makes a big difference when it comes to assessing how realistic a project of reform might be.
There are surely some deep reforms along these lines that CFAR/MIRI must consider. For one thing: scrupulosity, in the context of AI safety, seems to be a common thread in several of these breakdowns. I’ve taken this seriously enough in the past to post extensively on it here. I’d like CFAR/MIRI leadership to carefully update on how scrupulosity hurts both their people and their mission, and think about changes beyond surface-level things like adding a curriculum on scrupulosity. The actual incentives ought to change.
Finally, a good amount of Jessica’s post (similarly to Zoe’s post) concerns her inner experiences, on which she is the undisputed expert. I’m not ignoring those parts above. I just can’t say anything about them, merely that as a third person observer it’s much easier to discuss the external realities than the internal ones. (Likewise with Zoe and Leverage.)
My main complaint about this and the Leverage post is the lack of base-rate data. How many people develop mental health problems in a) normal companies, b) startups, c) small non-profits, d) cults/sects? So far, all I have seen are two cases. And in the startups I have worked at, I would also have been able to find mental health cases that could be tied to the company narrative. Humans being human narratives get woven. And the internet being the internet, some will get blown out of proportion. That doesn’t diminish the personal experience at all. I am updating only slightly on CFAR or MIRI. And basically not at all on “things look better from the outside than from the inside.”
In particular, I want to remind people here that something like 30-40% of grad students at top universities have either clinically diagnosed depression or anxiety (link). I think given the kind of undirected, often low-paid, work that many have been doing for the last decade, I think that’s the right reference class to draw from, and my current guess is we are roughly at that same level, or slightly below it (which is a crazy high number, and I think should give us a lot of pause).
I’m confused about how you got to this conclusion, and think it is most likely false. Neither your link, the linked study, or the linked meta-analysis in the linked study of your link says this. Instead the abstract of the linked^3 meta-analysis says:
Further, the discussion section of the linked^3 study emphasizes:
So I think there is at least two things going on here:
Most people with clinically significant significant symptoms do not go get diagnosed, so “clinically significant symptoms of” depression/anxiety is a noticeably lower bar than “actually clinically diagnosed”
As implied in the quoted discussion above, if everybody were to seek diagnosis, only ~half of the rate of symptomatic people would be clinically diagnosed as having depression/anxiety.
For those keeping score, this is ~12% for depression and 8.5% for anxiety, with some error bars.
Separately, I also think:
is wrong. My guess is that xrisk reducers have worse mental health on average compared to grad students. (I also believe this, with lower confidence, about people working in other EA cause areas like animal welfare, global poverty, or non-xrisk longtermism, as well as serious rationalists who aren’t professionally involved in EA cause areas).
Note that the pooled prevalence is 24% (CI 18-31). But it differs a lot across studies, symptoms, and location. In the individual studies, the range is really from zero to 50% (or rather to 38% if you exclude a study with only 6 participants). I think a suitable reference class would be the University of California which has 3,190 participants and a prevalence of 38%.
Sorry, am I misunderstanding something? I think taking “clinically significant symptoms”, specific to the UC system, as a given is wrong because it did not directly address either of my two criticisms:
1. Clinically significant symptoms =/= clinically diagnosed even in worlds where there is a 1:1 relationship between clinically significant symptoms and would have been clinically diagnosed, as many people do not get diagnosed
2. Clinically significant symptoms do not have a 1:1 relationship with would have been clinically diagnosed.
Well, I agree that the actual prevalence you have in mind would be roughly half of 38% i.e. ~20%. That is still much higher than the 12% you arrived at. And either value is so high that there is little surprise some severe episodes of some people happened in a 5-year frame.
The UC Berkeley study was the one that I had cached in my mind as generating this number. I will reread it later today to make sure that it’s right, but it sure seems like the most relevant reference class, given the same physical location.
I had a look at the situation in Germany and it doesn’t look much better. 17% of students are diagnosed with at least one psychical disorder. This is based on the health records of all students insured by one of the largest public health insurers in Germany (about ten percent of the population):
https://www.barmer.de/blob/144368/08f7b513fdb6f06703c6e9765ee9375f/data/dl-barmer-arztreport-2018.pdf
I feel like the paragraph you cited just seems like the straightforward explanation of where my belief comes from?
24% of people have depression, 17% have anxiety, resulting in something like 30%-40% having one or the other.
I did not remember the section about the screening instruments over-identifying cases of depression/anxiety by approximately a factor of two, which definitely cuts down my number, and I should have adjusted it in my above comment. I do think that factor of ~2 does maybe make me think that we are doing a bit worse than grad students, though I am not super sure.
Sorry, maybe this is too nitpicky, but clinically significant symptoms =/= clinically diagnosed, even in worlds where the clinically significant symptoms are severe enough to be diagnosed as such.
If you instead said in “population studies 30-40% of graduate students have anxiety or depression severe enough to be clinically diagnosed as such were they to seek diagnosis” then I think this will be a normal misreading from not jumping through enough links.
Put another way, if someone in mid-2020 told me that they had symptomatic covid and was formally diagnosed with covid, I would expect that they had worse symptoms than someone who said they had covid symptoms and later tested for covid antibodies. This is because jumping through the hoops to get a clinical diagnosis is nontrivial Bayesian evidence of severity and not just certainty, under most circumstances, and especially when testing is limited and/or gatekeeped (which is true for many parts of the world for covid in 2020, and is usually true in the US for mental health).
Ah, sorry, yes. Me being unclear on that was also bad. The phrasing you give is the one I intended to convey, though I sure didn’t do it.
Thanks, appreciate the update!
Additionally, as a canary statement: I was also never asked to sign an NDA.
I think Pete did (most of?) the hiring as soon as he became ED, so I think this has been the state of CFAR for a while (while I think Anna has also been able to hire people she wanted to hire).
It’s always been a somewhat group-involved process, but yes, I was primarily responsible for hiring for roughly 2016 through the end of 2017, then it would have been Tim. But again, it’s a small org and always involved some involvement of the whole group.
Without denying that it is a small org and staff usually have some input over hiring, that input is usually informal.
My understanding is that in the period when Anna was ED, there was an explicit all-staff discussion when they were considering a hire (after the person had done a trial?). In the Pete era, I’m sure Pete asked for staff members’ opinions, and if (for instance) I sent him an email with my thoughts on a potential hire, he would take that info into account, but there was not institutional group meeting.
Nitpicking: there are reasons to have multiple projects, for example it’s convenient to be in the same geographic location but not anyone can relocate to any place.
Sure—and MIRI/FHI are a decent complement to each other, the latter providing a respectable academic face to weird ideas.
Generally though, it’s far more productive to have ten top researchers in the same org rather than having five orgs each with two top researchers and a couple of others to round them out. Geography is a secondary concern to that.
A “secondary concern” in the sense that, we should work remotely? Or in the sense that everyone should relocate? Because the latter is unrealistic: people have families, friends, communities, not anyone can uproot themself.
A secondary concern in that it’s better to have one org that has some people in different locations, but everyone communicating heavily, than to have two separate organizations.
I think this is much more complex than you’re assuming. As a sketch of why, costs of communication scale poorly, and the benefits of being small and coordinating centrally often beats the costs imposed by needing to run everything as one organization. (This is why people advise startups to outsource non-central work.)
This might be the right approach, but notice that no existing AI risk org does that. They all require physical presence.
Anthropic does not require consistent physical presence.
AFAICT, Anthropic is not an existential AI safety org per se, they’re just doing a very particular type of research which might help with existential safety. But also, why do you think they don’t require physical presence?
If you’re asking why I believe that they don’t require presence, I’ve been interviewing with them and that’s my understanding from talking with them. The first line of copy on their website is
Sounds pretty much like a safety org to me.
Are you talking about “you can work from home and come to the office occasionally”, or “you can live on a different continent”?
I found no mention of existential risk on their web page. They seem to be a commercial company, aiming at short-to-mid-term applications. I doubt they have any intention to do e.g. purely theoretical research, especially if it has no applications to modern systems. So, what they do can still be meritorious and relevant to reducing existential risk. But, the context of this discussion is: can we replace all AI safety orgs by just one org. And, Anthropic is too specialized to serve such a role.
I believe Anthropic doesn’t expect its employees to be in the office every day, but I think this is more pandemic-related than it is a deliberate organizational design choice; my guess is that most Anthropic employees will be in the office a year from now.