What do AI safety experts believe about the big picture of AI risk?
I would be careful not to implicitly claim that these 17 people are a “representative sample” of the AI safety community. Or, if you do want to make that claim, I think it’s important to say a lot more about how these particular participants were chosen and why you think they are represented.
At first glance, it seems to me like this pool of participants overrepresents some worldviews and under-represents others. For example, it seems like the vast majority of the participants either work for AGI labs, Open Philanthropy, and close allies/grantees of OP. OP undoubtedly funds a lot of AIS groups, but there are lots of experts who approach AIS from a different set of assumptions and worldviews.
More specifically, I’d say this list of 17 experts over-represents what I might refer to as the “Open Phil + AGI labs + people funded by or close to those entities” cluster of thinkers (who IMO generally are more optimistic than folks at groups like MIRI, Conjecture, CAIS, FLI, etc.) & over-represents people who are primarily focused on technical research (who IMO are generally most optimistic about technical alignment, more likely to believe empirical work is better than conceptual work, and more likely to believe in technical rather than socio-technical approaches.)
To be clear– I still think that work like this is & can be important. Also, there is some representation from people outside of the particular subculture I’m claiming is over-represented.
But I think it is very hard to do a survey that actually meaningfully represents the AI safety community, and I think there are a lot of subjective decisions that go into figuring out who counts as an “expert” in the field.
I think it probably doesn’t make sense to talk about “representative samples”.
Here are a bunch of different things this could mean:
A uniform sample from people who have done any work related to AI safety.
A sample from people weighted to their influence/power in the AI safety community.
A sample from people weighted by how much I personally respect their views about AI risk.
Maybe what you mean is: “I think this sample underrepresents a world view that I think this is promising. This world view is better represented by MIRI/Conjecture/CAIS/FLI/etc.”
I think programs like this one should probably just apply editorial discretion and note explicitly that they are doing so.
(This complaint is also a complaint about the post which does try to use a notion of “representative sample”.)
I would be careful not to implicitly claim that these 17 people are a “representative sample” of the AI safety community.
Worth noting that this is directly addressed in the post:
The sample of people I interviewed is not necessarily a representative sample of the AI safety movement as a whole. The sample was pseudo-randomly selected, optimizing for a) diversity of opinion, b) diversity of background, c) seniority, and d) who I could easily track down. Noticeably, there is an absence of individuals from MIRI, a historically influential AI safety organization, or those who subscribe to similar views. I approached some MIRI team members but no one was available for an interview. This is especially problematic since many respondents criticized MIRI for various reasons, and I didn’t get much of a chance to integrate MIRI’s side of the story into the project.
So, in this case, I would say this is explicitly disclaimed let alone implicitly claimed.
Which of the institutions would you count as AGI labs? (genuinely curious– usually I don’t think about academic labs [relative to like ODA + Meta + Microsoft] but perhaps there are some that I should be counting.)
And yeah, OP funding is a weird metric because there’s a spectrum of how much grantees are closely tied to OP. Like, there’s a wide spectrum from “I have an independent research group and got 5% of my total funding from OP” all the way to like “I get ~all my funding from OP and work in the same office as OP and other OP allies and many of my friends/colleagues are OP etc.”
That’s why I tried to use the phrase “close allies/grantees”, to convey more of this implicit cultural stuff than merely “have you ever received OP $.” My strong impression is that the authors of the paper are much more intellectually/ideologically/culturally independent from OP, relative to the list of 17 interviewees presented above.
Anca Dragan, who currently leads an alignment team at DeepMind, is the one I saw (I then mistakenly assumed there were others). And fair point re: academic OpenPhil grantees.
I would be careful not to implicitly claim that these 17 people are a “representative sample” of the AI safety community. Or, if you do want to make that claim, I think it’s important to say a lot more about how these particular participants were chosen and why you think they are represented.
At first glance, it seems to me like this pool of participants overrepresents some worldviews and under-represents others. For example, it seems like the vast majority of the participants either work for AGI labs, Open Philanthropy, and close allies/grantees of OP. OP undoubtedly funds a lot of AIS groups, but there are lots of experts who approach AIS from a different set of assumptions and worldviews.
More specifically, I’d say this list of 17 experts over-represents what I might refer to as the “Open Phil + AGI labs + people funded by or close to those entities” cluster of thinkers (who IMO generally are more optimistic than folks at groups like MIRI, Conjecture, CAIS, FLI, etc.) & over-represents people who are primarily focused on technical research (who IMO are generally most optimistic about technical alignment, more likely to believe empirical work is better than conceptual work, and more likely to believe in technical rather than socio-technical approaches.)
To be clear– I still think that work like this is & can be important. Also, there is some representation from people outside of the particular subculture I’m claiming is over-represented.
But I think it is very hard to do a survey that actually meaningfully represents the AI safety community, and I think there are a lot of subjective decisions that go into figuring out who counts as an “expert” in the field.
I think it probably doesn’t make sense to talk about “representative samples”.
Here are a bunch of different things this could mean:
A uniform sample from people who have done any work related to AI safety.
A sample from people weighted to their influence/power in the AI safety community.
A sample from people weighted by how much I personally respect their views about AI risk.
Maybe what you mean is: “I think this sample underrepresents a world view that I think this is promising. This world view is better represented by MIRI/Conjecture/CAIS/FLI/etc.”
I think programs like this one should probably just apply editorial discretion and note explicitly that they are doing so.
(This complaint is also a complaint about the post which does try to use a notion of “representative sample”.)
Worth noting that this is directly addressed in the post:
So, in this case, I would say this is explicitly disclaimed let alone implicitly claimed.
Note that the linked paper includes a bunch of authors from AGI labs or who have received OpenPhil funding.
Which of the institutions would you count as AGI labs? (genuinely curious– usually I don’t think about academic labs [relative to like ODA + Meta + Microsoft] but perhaps there are some that I should be counting.)
And yeah, OP funding is a weird metric because there’s a spectrum of how much grantees are closely tied to OP. Like, there’s a wide spectrum from “I have an independent research group and got 5% of my total funding from OP” all the way to like “I get ~all my funding from OP and work in the same office as OP and other OP allies and many of my friends/colleagues are OP etc.”
That’s why I tried to use the phrase “close allies/grantees”, to convey more of this implicit cultural stuff than merely “have you ever received OP $.” My strong impression is that the authors of the paper are much more intellectually/ideologically/culturally independent from OP, relative to the list of 17 interviewees presented above.
Anca Dragan, who currently leads an alignment team at DeepMind, is the one I saw (I then mistakenly assumed there were others). And fair point re: academic OpenPhil grantees.