Thanks for writing this, Ryan! I’d be excited to see more lists like this. A few ideas I’ll add:
An organization that runs high-quality AI safety workshops every month. Target audience = Top people who have finished AGISF or exceptionally-talented people who haven’t yet been exposed to AI safety. Somewhat similar to Impact Generator workshops, the Bright Futures workshop, and these workshops.
MATS adding mentors for non-technical research (e.g., people like Ajeya Cotra or Daniel Kokotajlo)
An organization that produces research designed to make alignment problems more concrete & unlocks new ways for people to contribute to them. Stuff like the specification gaming post & the goal misgeneralization papers.
Writing an intro to AIS piece that is (a) short, (b) technically compelling, and (c) emotionally compelling.
An organization that regularly organizes talks at big tech companies, prestigious academic labs, and AI labs. (The talks are given by senior AIS researchers who are good at presenting ideas to new audiences, but the organization does a lot of the ops/organizing).
I’m skeptical of the usefulness of such an organization. I think we have a plethora of motivated talent passing through AGISF currently that doesn’t need another short workshop or a bunch of low-context 1-1s with researchers who probably have better-vetted people to spend their scarce time on (I think vetting is very hard). I think the AI safety talent development pipeline needs less centralized, short-duration, broad-funnel workshops and more programs that allow for longer-term talent incubation into specific niches (e.g., programs/low-stakes jobs that build critical skills in research vs. management vs. operations) that don’t eat up valuable researcher time unnecessarily and encourage decentralization of alignment research hubs. Sorry if this sounds like bad-faith criticism; it’s not intended to be.
Seems like a great idea. I think I’d strictly prefer “a combined research mentorship and seminar program that aims to do for AI governance research what MATS is trying to do for technical AI alignment research,” because it feels like the optimal program for Cotra or Kokotajlo is a bit different from MATS and likely includes other great governance/worldview/macrostrategy researchers. However, I’ll probably talk to Cotra and Kokotajlo to see if we can add value to them!
I think this is rather the domain of existing academic and industry research groups (e.g., Krueger’s lab, Anthropic, CHAI, OpenAI safety team, DeepMind safety team, etc.), as these groups have the necessary talent and, presumably, motivation. I’d also be excited for MATS scholars and alumni and LTFF-funded independent researchers to work on this.
Seems good, if hard (and not something I’d expect a competition to help with, on priors, if the most capable/aligned people already are not working on this). In particular, I’d be excited to see something that discusses, from first principles, whether solving alignment is significantly harder than similar scientific problems (credit to Caleb for often talking about this).
I’m skeptical that this would be low-risk (in regards to making researchers more skeptical of alignment and less likely to listen to AI safety spokespeople at a critical date) or a counterfactually valuable use of senior AI safety researcher time.
Regarding (2), I think the best AI gov research training org is substantially different from MATS’ idealized form because:
To the extent possible, I think MATS should be trying to solve a compartmentalization of the alignment problem (i.e., the technical part, to the extent that’s separable from governance) because I think people probably grow more as deep researchers from academic cohorts with a dominant focus;
The Bay Area, where MATS is based, is not the governance hub of the US;
One should market differently to pol. sci., international relations, and complex systems researchers compared to ML, maths, physics, and neuroscience researchers;
The MATS seminar program is geared towards technical alignment research and an ideal AI governance seminar program would look substantially different;
I don’t understand the governance literature as well as I understand the technical safety literature and there are probably unknown unknowns;
MATS is quite large (55 in-person scholars for winter);
There are good alignment mentors MATS could support that might get bumped out if we add governance mentors;
At some point, more MATS team members (to support a larger cohort) might make it hard for us to work effectively (as per Lightcone’s model);
A small, cohesive MAGS team could work in parallel with the MATS team for greater impact than a unified team;
GovAI are already doing something like the hypothetical MAGS, and the OpenAI governance team might want to do the same, which would mean a lot of potential mentors (maybe enough for a MATS-sized program, funders + mentors willing).
The Bay Area, where MATS is based, is not the governance hub of the US;
The Bay is an AI hub, home to OpenAI, Google, Meta, etc., and therefore an AI governance hub. Governance is not governments. Important decisions are being made there—maybe more important decisions than in DC. To quote Allan Dafoe:
AI governance concerns how humanity can best navigate the transition to a world with advanced AI systems[1]. It relates to how decisions are made about AI[2], and what institutions and arrangements would help those decisions to be made well.
Also, many, many AI governance projects go hand-in-hand with technical expertise.
Regarding point 5: AI safety researchers are already taking the time to write talks and present them (e.g., Rohin Shah’s introduction to AI alignment, though I think he has a more ML-oriented version). If we work off of an existing talk or delegate the preparation of the talk, then it wouldn’t take much time for a researcher to present it.
Thanks for writing this, Ryan! I’d be excited to see more lists like this. A few ideas I’ll add:
An organization that runs high-quality AI safety workshops every month. Target audience = Top people who have finished AGISF or exceptionally-talented people who haven’t yet been exposed to AI safety. Somewhat similar to Impact Generator workshops, the Bright Futures workshop, and these workshops.
MATS adding mentors for non-technical research (e.g., people like Ajeya Cotra or Daniel Kokotajlo)
An organization that produces research designed to make alignment problems more concrete & unlocks new ways for people to contribute to them. Stuff like the specification gaming post & the goal misgeneralization papers.
Writing an intro to AIS piece that is (a) short, (b) technically compelling, and (c) emotionally compelling.
An organization that regularly organizes talks at big tech companies, prestigious academic labs, and AI labs. (The talks are given by senior AIS researchers who are good at presenting ideas to new audiences, but the organization does a lot of the ops/organizing).
I’m skeptical of the usefulness of such an organization. I think we have a plethora of motivated talent passing through AGISF currently that doesn’t need another short workshop or a bunch of low-context 1-1s with researchers who probably have better-vetted people to spend their scarce time on (I think vetting is very hard). I think the AI safety talent development pipeline needs less centralized, short-duration, broad-funnel workshops and more programs that allow for longer-term talent incubation into specific niches (e.g., programs/low-stakes jobs that build critical skills in research vs. management vs. operations) that don’t eat up valuable researcher time unnecessarily and encourage decentralization of alignment research hubs. Sorry if this sounds like bad-faith criticism; it’s not intended to be.
Seems like a great idea. I think I’d strictly prefer “a combined research mentorship and seminar program that aims to do for AI governance research what MATS is trying to do for technical AI alignment research,” because it feels like the optimal program for Cotra or Kokotajlo is a bit different from MATS and likely includes other great governance/worldview/macrostrategy researchers. However, I’ll probably talk to Cotra and Kokotajlo to see if we can add value to them!
I think this is rather the domain of existing academic and industry research groups (e.g., Krueger’s lab, Anthropic, CHAI, OpenAI safety team, DeepMind safety team, etc.), as these groups have the necessary talent and, presumably, motivation. I’d also be excited for MATS scholars and alumni and LTFF-funded independent researchers to work on this.
Seems good, if hard (and not something I’d expect a competition to help with, on priors, if the most capable/aligned people already are not working on this). In particular, I’d be excited to see something that discusses, from first principles, whether solving alignment is significantly harder than similar scientific problems (credit to Caleb for often talking about this).
I’m skeptical that this would be low-risk (in regards to making researchers more skeptical of alignment and less likely to listen to AI safety spokespeople at a critical date) or a counterfactually valuable use of senior AI safety researcher time.
Regarding (2), I think the best AI gov research training org is substantially different from MATS’ idealized form because:
To the extent possible, I think MATS should be trying to solve a compartmentalization of the alignment problem (i.e., the technical part, to the extent that’s separable from governance) because I think people probably grow more as deep researchers from academic cohorts with a dominant focus;
The Bay Area, where MATS is based, is not the governance hub of the US;
One should market differently to pol. sci., international relations, and complex systems researchers compared to ML, maths, physics, and neuroscience researchers;
The MATS seminar program is geared towards technical alignment research and an ideal AI governance seminar program would look substantially different;
I don’t understand the governance literature as well as I understand the technical safety literature and there are probably unknown unknowns;
Currently MATS expects applicants to have background knowledge at the level of the AGI Safety Fundamentals AI Alignment Curriculum and not the AI Governance Curriculum;
I’d rather do one thing well than two things poorly.
Thoughts, Mauricio?
Additional reasons why MATS + MAGS > MATS + governance patch:
MATS is quite large (55 in-person scholars for winter);
There are good alignment mentors MATS could support that might get bumped out if we add governance mentors;
At some point, more MATS team members (to support a larger cohort) might make it hard for us to work effectively (as per Lightcone’s model);
A small, cohesive MAGS team could work in parallel with the MATS team for greater impact than a unified team;
GovAI are already doing something like the hypothetical MAGS, and the OpenAI governance team might want to do the same, which would mean a lot of potential mentors (maybe enough for a MATS-sized program, funders + mentors willing).
The Bay is an AI hub, home to OpenAI, Google, Meta, etc., and therefore an AI governance hub. Governance is not governments. Important decisions are being made there—maybe more important decisions than in DC. To quote Allan Dafoe:
Also, many, many AI governance projects go hand-in-hand with technical expertise.
Maybe more broadly, AI strategy is part of AI governance.
Regarding point 5: AI safety researchers are already taking the time to write talks and present them (e.g., Rohin Shah’s introduction to AI alignment, though I think he has a more ML-oriented version). If we work off of an existing talk or delegate the preparation of the talk, then it wouldn’t take much time for a researcher to present it.