One thing that has been bothering me a lot is that it seems like it’s really likely that people don’t realize just how distinct CFAR and MIRI are.
I’ve worked at each org for about three years total.
Some things which make it reasonable to lump them together and use the label “CFAR/MIRI”:
They both descend from what was at one time a single organization.
They had side-by-side office spaces for many years, including a shared lunch table in the middle where people from both orgs would hang out and chat.
There are a lot of people common to both orgs (e.g. Anna does work for both orgs, I moved from CFAR to MIRI).
CFAR ran many explicit programs on MIRI’s behalf (e.g. MSFP, or less directly but still pretty clearly AIRCS).
Most MIRI staff have been to a CFAR workshop. Most MIRI staff have participated in at least one debugging session with a CFAR staff member (this was a service CFAR explicitly offered for a while).
Both orgs are explicitly concerned with navigating existential risk from unaligned artificial intelligence.
If MIRI “needed help,” CFAR would be there. If CFAR “needed help,” MIRI would be there. They are explicitly friendly, allied orgs.
The “most CFARish” MIRI employee has a lot in common, both in their own traits and in their historical experiences, with the “most MIRIish” CFAR employee.
That’s the motte. All of that is true, and reasonable, and relevant, and probably people could name a couple of other similarly true and reasonable and relevant points.
The bailey:
The overlap of people like Anna is representative of the overlap between the two orgs; MIRI people and CFAR people see themselves as being in a single bucket.
The median CFAR employee and the median MIRI employee interact frequently.
CFAR and MIRI share a single memetic pool; what’s being discussed in one org is being discussed in the other; what’s trendy in one org is what’s trendy in the other.
CFAR and MIRI coordinate on goals as a rule, as opposed to when it’s convenient; they’re working together on a single plan.
The day-to-day experience of a CFAR employee strongly resembles the day-to-day experience of a MIRI employee.
The experience of a CFAR employee is representative of the experience of a MIRI employee, and vice versa; one is a good proxy for the other.
The two orgs have identical, overlapping, or solidly compatible internal cultures. The rules and norms of one org are very similar to the rules and norms of the other.
Somebody who’s been to a CFAR workshop understands the vibe at MIRI; somebody who’s been to a MIRI-X understands the vibe at CFAR
Other relevant individuals and groups in the rationalist and EA communities think of CFAR and MIRI as being essentially the same thing
None of the above is even close to true, as far as I can see. And that seems really relevant—just as Eli was pointing out that it would be bad to wrongly bucket CFAR and Leverage together, so too do I claim that it would be bad to wrongly bucket MIRI and CFAR together.
I think that it’s entirely valid to point out commonalities between them, or ways in which the culture or norms of one might resemble or reinforce the culture or norms of the other.
I think it’s extremely invalid to assume those commonalities as a matter of course.
And I think it’s quite bad (though I don’t think anyone has done this intentionally) to cause people to start assuming those commonalities, or cause people to think that everyone knows that they exist.
I think that Alex Vermeer and Malo Bourgon are extremely unlike Tim Telleen-Lawton and Pete Michaud (even though they’ve all worked together a lot and have a lot of respect for each other); I think that the experience of Jack Carroll is extremely unlike the experience of Carson Jones; I think that a CFAR staff retreat is extremely unlike a MIRI research retreat; I think that what’s expected as a matter of course from a MIRI researcher is extremely unlike what’s expected as a matter of course from a CFAR theorist; I think that what constitutes proprietary information or informational security at each org is extremely different, etc. etc. etc.
I want to make space here for people to just plainly state the overlap; I don’t think this is offensive and I do think it’s relevant.
But until laying this groundwork, I had the feeling that people would feel quite scared or cautious or unlikely-to-be-believed or suspected-of-ulterior-motives if they dared point at the disoverlap, which in my experience is much larger.
I agree with all of the above. And yet a third thing, which Jessica also discusses in the OP, is the community near MIRI and/or CFAR, whose ideology has been somewhat shaped by the two organizations.
There are some good things to be gained from lumping things together (larger datasets on which to attempt inference) and some things that are confusing.
As far as I see it, nowadays CFAR is about 60% a hiring ground for MIRI and only 40% something else, though I could be wrong.
Actually, that was true for the last few years (with an ambiguous in-between time during covid), but it is not true now.
This thread is agreeing the orgs are completely different, but elsewhere you agreed that CFAR functions as a funnel into MIRI.
I ask this out of personal interest in CFAR and MIRI going forwards and because I’m currently much more confused about how the two work than I was a week ago.
In the era 2015 − 2018, CFAR served mostly as not a funnel into MIRI in terms of total effort, programs, the curriculum of those programs, etc., but also:
CFAR ran some specific programs intended to funnel promising people toward MIRI, such as MSFP
CFAR “kept its eyes out” during its regular programs for people who looked promising and might be interested in getting more involved with MIRI or MIRI-adjacent work
Toward the 2018 − 2020 era, some CFAR staff incubated the AIRCS program, which was a lot like CFAR workshops except geared toward bridging between the AI risk community and various computer scientist bubbles, with a strong eye toward finding people who might work on MIRI projects. AIRCS started as a more-or-less independent project that occasionally borrowed CFAR logistical support, but over time CFAR decided to contribute more explicit effort to it, until it eventually became (afaik) straightforwardly one of the two or three most important “things going on at CFAR,” according to CFAR.
Staff who were there at the time (this was as I was phasing out) might correct this summary, but I believe it’s right in its essentials.
In the last two years, CFAR hasn’t done much outward-facing work at all, due to COVID, and so has neither been a MIRI funnel nor definitively not a MIRI funnel.
In the last two years, CFAR hasn’t done much outward-facing work at all, due to COVID, and so has neither been a MIRI funnel nor definitively not a MIRI funnel.
Yes, but I would predict that we won’t be the same sort of MIRI funnel going forward. This is because MIRI used to have specific research programs that it needed to hire for, and it it was sponsoring AIRCS (covering direct expenses plus loaning us some researchers to help run the thing) in order to recruit for that, and those research programs have been discontinued and so AIRCS won’t be so much of a thing anymore.
This has been the main part of why no AIRCS post vaccines, not just COVID.
I, and I would guess some others at CFAR, am interested in running AIRCS-like programs going forward, especially if there are groups that want to help us pay the direct expenses for those programs and/or researchers that want to collaborate with us on such programs. (Message me if you’re reading this and in one of those categories.) But it’ll be less MIRI-specific this time, since there isn’t that recruiting angle.
Also, more broadly, CFAR has adopted different structures for organizing ourselves internally, and we are bigger now into “if you work for CFAR, or are a graduate of our instructor training program, and you have a ‘telos’ that you’re on fire to do, you can probably do it with CFAR’s venue/dollars/collaborations of some sorts” (we’re calling this “platform CFAR,” Elizabeth Garrett invented it and set it up maybe about a year ago, can’t remember), and also into doing hourly rather than salaried work in general (so we don’t feel an obligation to fill time with some imagined ’supposed to do CFAR-like activity” vagueness, so that we can be mentally free) and are also into taking more care not to have me or anyone speak for others at CFAR or organize people into a common imagined narrative one must pretend to believe, but rather into letting people do what we each believe in, and try to engage each other where sensible. Which makes it a bit harder to know what CFAR will be doing going forward, and also leaves me thinking it’ll have a bit more variety in it. Probably.
I think Anna was saying “it is true that in the 2018 − 2020 era, CFAR was about 60% a hiring ground and only 40% something else, but that is not true currently.”
If this is the case, I do understand now, but I think the comment claiming that it’s not true at the literal current moment of October 2021 is useless in a misleading (though probably not intentional way).
I think it is important to the CFAR-aligned folks that CFAR is not “bad” in the way noted in that comment, but to everyone else, the important thing is whether or not that criticism is true. It was the initial ignorance on my end that we were looking at the same fact from different angles that led to the confusion.
(Also, I’m not continuing this out of a desire to show that “I’m right” or something, but just to explain why I cared since I now understand the mistake and can explain it. I’m happy to flesh it out more if this wasn’t very clear)
TBC it easily may also be that CFAR made strategic shifts during COVID that make the statement true in a non-trivial way; I simply wouldn’t know that fact and so can’t speak to it.
Mostly agree. I especially agree about the organizational structure being very different.
I would not have said “”The median CFAR employee and the median MIRI employee interact frequently.” is not even close to true”, but it depends on the operationalization of frequently. But according to my operationalization, the lunch table alone makes it close to true.
I would also not have said “I think that a CFAR staff retreat is extremely unlike a MIRI research retreat.” (e.g. we have attempted to Circle at a research retreat more than once.) (I haven’t actually been to a CFAR staff retreat, but I have been to some things that I imagine are somewhat close, like workshops where a majority of attendees are CFAR staff).
I think “we’ve attempted to circle at a research retreat more than once” is only a little stronger evidence of overlap than “we also ate food at our retreat.”
Fair point about the lunch table, although it’s my sense that a strict majority of MIRI employees were almost never at the lunch table and for the first two years of my time at CFAR we didn’t share a lunch table.
Yeah. I am more pointing at “the very fact that Scott seems to think that ‘trying to circle more than once’ is sufficient to posit substantial resemblance between MIRI research retreats and CFAR staff retreats is strong evidence that Scott has no idea what the space of CFAR staff retreats is like.”
CFAR staff retreats often involve circling. Our last one, a couple weeks ago, had this, though as an optional evening thing that some but not most took part in.
I’m saying they involved circling often while I was there but that fact was something like 3-15% of their “character” (and probably closer to 3% imo) and so learning that some other thing also involves circling tells you very little about the overall resemblance of the two things.
One thing that has been bothering me a lot is that it seems like it’s really likely that people don’t realize just how distinct CFAR and MIRI are.
I’ve worked at each org for about three years total.
Some things which make it reasonable to lump them together and use the label “CFAR/MIRI”:
They both descend from what was at one time a single organization.
They had side-by-side office spaces for many years, including a shared lunch table in the middle where people from both orgs would hang out and chat.
There are a lot of people common to both orgs (e.g. Anna does work for both orgs, I moved from CFAR to MIRI).
CFAR ran many explicit programs on MIRI’s behalf (e.g. MSFP, or less directly but still pretty clearly AIRCS).
Most MIRI staff have been to a CFAR workshop. Most MIRI staff have participated in at least one debugging session with a CFAR staff member (this was a service CFAR explicitly offered for a while).
Both orgs are explicitly concerned with navigating existential risk from unaligned artificial intelligence.
If MIRI “needed help,” CFAR would be there. If CFAR “needed help,” MIRI would be there. They are explicitly friendly, allied orgs.
The “most CFARish” MIRI employee has a lot in common, both in their own traits and in their historical experiences, with the “most MIRIish” CFAR employee.
That’s the motte. All of that is true, and reasonable, and relevant, and probably people could name a couple of other similarly true and reasonable and relevant points.
The bailey:
The overlap of people like Anna is representative of the overlap between the two orgs; MIRI people and CFAR people see themselves as being in a single bucket.
The median CFAR employee and the median MIRI employee interact frequently.
CFAR and MIRI share a single memetic pool; what’s being discussed in one org is being discussed in the other; what’s trendy in one org is what’s trendy in the other.
CFAR and MIRI coordinate on goals as a rule, as opposed to when it’s convenient; they’re working together on a single plan.
The day-to-day experience of a CFAR employee strongly resembles the day-to-day experience of a MIRI employee.
The experience of a CFAR employee is representative of the experience of a MIRI employee, and vice versa; one is a good proxy for the other.
The two orgs have identical, overlapping, or solidly compatible internal cultures. The rules and norms of one org are very similar to the rules and norms of the other.
Somebody who’s been to a CFAR workshop understands the vibe at MIRI; somebody who’s been to a MIRI-X understands the vibe at CFAR
Other relevant individuals and groups in the rationalist and EA communities think of CFAR and MIRI as being essentially the same thing
None of the above is even close to true, as far as I can see. And that seems really relevant—just as Eli was pointing out that it would be bad to wrongly bucket CFAR and Leverage together, so too do I claim that it would be bad to wrongly bucket MIRI and CFAR together.
I think that it’s entirely valid to point out commonalities between them, or ways in which the culture or norms of one might resemble or reinforce the culture or norms of the other.
I think it’s extremely invalid to assume those commonalities as a matter of course.
And I think it’s quite bad (though I don’t think anyone has done this intentionally) to cause people to start assuming those commonalities, or cause people to think that everyone knows that they exist.
I think that Alex Vermeer and Malo Bourgon are extremely unlike Tim Telleen-Lawton and Pete Michaud (even though they’ve all worked together a lot and have a lot of respect for each other); I think that the experience of Jack Carroll is extremely unlike the experience of Carson Jones; I think that a CFAR staff retreat is extremely unlike a MIRI research retreat; I think that what’s expected as a matter of course from a MIRI researcher is extremely unlike what’s expected as a matter of course from a CFAR theorist; I think that what constitutes proprietary information or informational security at each org is extremely different, etc. etc. etc.
I want to make space here for people to just plainly state the overlap; I don’t think this is offensive and I do think it’s relevant.
But until laying this groundwork, I had the feeling that people would feel quite scared or cautious or unlikely-to-be-believed or suspected-of-ulterior-motives if they dared point at the disoverlap, which in my experience is much larger.
I agree with all of the above. And yet a third thing, which Jessica also discusses in the OP, is the community near MIRI and/or CFAR, whose ideology has been somewhat shaped by the two organizations.
There are some good things to be gained from lumping things together (larger datasets on which to attempt inference) and some things that are confusing.
I know you’re busy with all this and other things, but how is this statement
compatible with this statement?
This thread is agreeing the orgs are completely different, but elsewhere you agreed that CFAR functions as a funnel into MIRI. I ask this out of personal interest in CFAR and MIRI going forwards and because I’m currently much more confused about how the two work than I was a week ago.
In the era 2015 − 2018, CFAR served mostly as not a funnel into MIRI in terms of total effort, programs, the curriculum of those programs, etc., but also:
CFAR ran some specific programs intended to funnel promising people toward MIRI, such as MSFP
CFAR “kept its eyes out” during its regular programs for people who looked promising and might be interested in getting more involved with MIRI or MIRI-adjacent work
Toward the 2018 − 2020 era, some CFAR staff incubated the AIRCS program, which was a lot like CFAR workshops except geared toward bridging between the AI risk community and various computer scientist bubbles, with a strong eye toward finding people who might work on MIRI projects. AIRCS started as a more-or-less independent project that occasionally borrowed CFAR logistical support, but over time CFAR decided to contribute more explicit effort to it, until it eventually became (afaik) straightforwardly one of the two or three most important “things going on at CFAR,” according to CFAR.
Staff who were there at the time (this was as I was phasing out) might correct this summary, but I believe it’s right in its essentials.
In the last two years, CFAR hasn’t done much outward-facing work at all, due to COVID, and so has neither been a MIRI funnel nor definitively not a MIRI funnel.
Yes, but I would predict that we won’t be the same sort of MIRI funnel going forward. This is because MIRI used to have specific research programs that it needed to hire for, and it it was sponsoring AIRCS (covering direct expenses plus loaning us some researchers to help run the thing) in order to recruit for that, and those research programs have been discontinued and so AIRCS won’t be so much of a thing anymore.
This has been the main part of why no AIRCS post vaccines, not just COVID.
I, and I would guess some others at CFAR, am interested in running AIRCS-like programs going forward, especially if there are groups that want to help us pay the direct expenses for those programs and/or researchers that want to collaborate with us on such programs. (Message me if you’re reading this and in one of those categories.) But it’ll be less MIRI-specific this time, since there isn’t that recruiting angle.
Also, more broadly, CFAR has adopted different structures for organizing ourselves internally, and we are bigger now into “if you work for CFAR, or are a graduate of our instructor training program, and you have a ‘telos’ that you’re on fire to do, you can probably do it with CFAR’s venue/dollars/collaborations of some sorts” (we’re calling this “platform CFAR,” Elizabeth Garrett invented it and set it up maybe about a year ago, can’t remember), and also into doing hourly rather than salaried work in general (so we don’t feel an obligation to fill time with some imagined ’supposed to do CFAR-like activity” vagueness, so that we can be mentally free) and are also into taking more care not to have me or anyone speak for others at CFAR or organize people into a common imagined narrative one must pretend to believe, but rather into letting people do what we each believe in, and try to engage each other where sensible. Which makes it a bit harder to know what CFAR will be doing going forward, and also leaves me thinking it’ll have a bit more variety in it. Probably.
Ah, so I should take the first statement as being strictly NOW, like 2021? That clears things up a lot, thanks!
I think Anna was saying “it is true that in the 2018 − 2020 era, CFAR was about 60% a hiring ground and only 40% something else, but that is not true currently.”
If this is the case, I do understand now, but I think the comment claiming that it’s not true at the literal current moment of October 2021 is useless in a misleading (though probably not intentional way).
I think it is important to the CFAR-aligned folks that CFAR is not “bad” in the way noted in that comment, but to everyone else, the important thing is whether or not that criticism is true. It was the initial ignorance on my end that we were looking at the same fact from different angles that led to the confusion.
(Also, I’m not continuing this out of a desire to show that “I’m right” or something, but just to explain why I cared since I now understand the mistake and can explain it. I’m happy to flesh it out more if this wasn’t very clear)
TBC it easily may also be that CFAR made strategic shifts during COVID that make the statement true in a non-trivial way; I simply wouldn’t know that fact and so can’t speak to it.
Mostly agree. I especially agree about the organizational structure being very different.
I would not have said “”The median CFAR employee and the median MIRI employee interact frequently.” is not even close to true”, but it depends on the operationalization of frequently. But according to my operationalization, the lunch table alone makes it close to true.
I would also not have said “I think that a CFAR staff retreat is extremely unlike a MIRI research retreat.” (e.g. we have attempted to Circle at a research retreat more than once.) (I haven’t actually been to a CFAR staff retreat, but I have been to some things that I imagine are somewhat close, like workshops where a majority of attendees are CFAR staff).
I think “we’ve attempted to circle at a research retreat more than once” is only a little stronger evidence of overlap than “we also ate food at our retreat.”
Fair point about the lunch table, although it’s my sense that a strict majority of MIRI employees were almost never at the lunch table and for the first two years of my time at CFAR we didn’t share a lunch table.
If you pick a randomly selected academic or hobby conference, I will be much more surprised that they had circling than if they had food.
Yeah. I am more pointing at “the very fact that Scott seems to think that ‘trying to circle more than once’ is sufficient to posit substantial resemblance between MIRI research retreats and CFAR staff retreats is strong evidence that Scott has no idea what the space of CFAR staff retreats is like.”
To clarify, are you saying that CFAR staff retreats don’t involve circling?
CFAR staff retreats often involve circling. Our last one, a couple weeks ago, had this, though as an optional evening thing that some but not most took part in.
I’m saying they involved circling often while I was there but that fact was something like 3-15% of their “character” (and probably closer to 3% imo) and so learning that some other thing also involves circling tells you very little about the overall resemblance of the two things.
Surprised by the circling comment, but it doesn’t seem worth going deep on a nitpick.
All this sounds broadly correct to me, modulo some nitpicks that are on the whole smaller than Scott’s objections (for a sense of scale).