I would amend it to say “sometimes struggles to find meaningful employment despite having the requisite talent to further impactful research directions (which I believe are plentiful)”
This still reads to me as advocating for a jobs program for the benefit of MATS grads, not safety. My guess is you’re aiming for something more like “there is talent that could do useful work under someone else’s direction, but not on their own, and we can increase safety by utilizing it”.
I expect that Ryan means to say one of the these things:
There isn’t enough funding for MATS grads to do useful work in the research directions they are working on, that have already been vouched for by senior alignment researchers (especially their mentors) to be valuable. (Potential examples: infrabayesianism)
There isn’t (yet) institutional infrastructure to support MATS grads to do useful work together as part of a team focused on the same (or very similar) research agendas, and that this is the case for multiple nascent and established research agendas. They are forced to go to academia and disperse across the world instead of being able to work together in one location. (Potential examples: selection theorems, multi-agent alignment (of the sort that Caspar Oesterheld and company work on))
There aren’t enough research managers in existing established alignment research organizations or frontier labs to enable MATS grads to work on the research directions they consider extremely high value, and would benefit from multiple people working together on (Potential examples: activation steering)
I’m pretty sure that Ryan does not mean to say that MATS grads cannot do useful work on their own. The point is that we don’t yet have the institutional infrastructure to absorb, enable, and scale new researchers the way our civilization has for existing STEM fields via, say, PhD programs or yearlong fellowships at OpenAI/MSR/DeepMind (which are also pretty rare). AFAICT, the most valuable part of such infrastructure in general is the ability to co-locate researchers working on the same or similar research problems—this is standard for academic and industry research groups, for example, and from experience I know that being able to do so is invaluable. Another extremely valuable facet of institutional infrastructure that enables researchers is the ability to delegate operations and logistics problems—particularly the difficulty of finding grant funding, interfacing with other organizations, getting paperwork handled, etc.
I keep getting more and more convinced, as time passes, that it would be more valuable for me to work on building the infrastructure to enable valuable teams and projects, than to simply do alignment research while disregarding such bottlenecks to this research ecosystem.
@Elizabeth, Mesa nails it above. I would also add that I am conceptualizing impactful AI safety research as the product of multiple reagents, including talent, ideas, infrastructure, and funding. In my bullet point, I was pointing to an abundance of talent and ideas relative to infrastructure and funding. I’m still mostly working on talent development at MATS, but I’m also helping with infrastructure and funding (e.g., founding LISA, advising Catalyze Impact, regranting via Manifund) and I want to do much more for these limiting reagents.
Also note that historically many individuals entering AI safety seem to have been pursuing the “Connector” path, when most jobs now (and probably in the future) are “Iterator”-shaped, and larger AI safety projects are also principally bottlenecked by “Amplifiers”. The historical focus on recruiting and training Connectors to the detriment of Iterators and Amplifiers has likely contributed to this relative talent shortage. A caveat: Connectors are also critical for founding new research agendas and organizations, though many self-styled Connectors would likely substantially benefit as founders by improving some Amplifier-shaped soft skills, including leadership, collaboration, networking, and fundraising.
I interpret your comment as assuming that new researchers with good ideas produce more impact on their own than in teams working towards a shared goal; this seems false to me. I think that independent research is usually a bad bet in general and that most new AI safety researchers should be working on relatively few impactful research directions, most of which are best pursued within a team due to the nature of the research (though some investment in other directions seems good for the portfolio).
I’ve addressed this a bit in thread, but here are some more thoughts:
New AI safety researchers seem to face mundane barriers to reducing AI catastrophic risk, including funding, infrastructure, and general executive function.
MATS alumni are generally doing great stuff (~46% currently work in AI safety/control, ~1.4% work on AI capabilities), but we can do even better.
Like any other nascent scientific/engineering discipline, AI safety will produce more impactful research with scale, albeit with some diminishing returns on impact eventually (I think we are far from the inflection point, however).
MATS alumni, as a large swathe of the most talented new AI safety researchers in my (possibly biased) opinion, should ideally not experience mundane barriers to reducing AI catastrophic risk.
Independent research seems worse than team-based research for most research that aims to reduce AI catastrophic risk:
“Pair-programming”, builder-breaker, rubber-ducking, etc. are valuable parts of the research process and are benefited by working in a team.
Funding insecurity and grantwriting responsibilities are larger for independent researchers and obstruct research.
Orgs with larger teams and discretionary funding can take on interns to help scale projects and provide mentorship.
Good prosaic AI safety research largely looks more like large teams doing engineering and less like lone geniuses doing maths. Obviously, some lone genius researchers (especially on mathsy non-prosaic agendas) seem great for the portfolio too, but these people seem hard to find/train anyways (so there is often more alpha in the former by my lights). Also, I have doubts that the optimal mechanism to incentivize “lone genius research” is via small independent grants instead of large bounties and academic nerdsniping.
Therefore, more infrastructure and funding for MATS alumni, who are generally value-aligned and competent, is good for reducing AI catastrophic risk in expectation.
I interpret your comment as assuming that new researchers with good ideas produce more impact on their own than in teams working towards a shared goal
I don’t believe that, although I see how my summary could be interpreted that way. I agree with basically all the reasons in your recent comment and most in the original comment. I could add a few reasons of my own doing independent grant-funded work sucks. But I think it’s really important to track how founding projects tracks to increased potential safety instead of intermediates, and push hard against potential tail wagging the dog scenarios.
I was trying to figure out why this was important to me, given how many of your points I agree with. I think it’s a few things:
Alignment work seems to be prone to wagging the dog, and is harder to correct, due to poor feedback loops.
The consequences of this can be dire
making it harder to identify and support the best projects.
making it harder to identify and stop harmful projects
making it harder to identify when a decent idea isn’t panning out, leading to people and money getting stuck in the mediocre project instead of moving on.
One of the general concerns about MATS is it spins up potential capabilities researchers. If the market can’t absorb the talent, that suggests maybe MATS should shrink.
OTOH if you told me that for every 10 entrants MATS spins up 1 amazing safety researcher and 9 people who need makework to prevent going into capabilities, I’d be open to arguments that that was a good trade.
I’m surprised this one was included, it feels tail-wagging-the-dog to me.
I would amend it to say “sometimes struggles to find meaningful employment despite having the requisite talent to further impactful research directions (which I believe are plentiful)”
This still reads to me as advocating for a jobs program for the benefit of MATS grads, not safety. My guess is you’re aiming for something more like “there is talent that could do useful work under someone else’s direction, but not on their own, and we can increase safety by utilizing it”.
I expect that Ryan means to say one of the these things:
There isn’t enough funding for MATS grads to do useful work in the research directions they are working on, that have already been vouched for by senior alignment researchers (especially their mentors) to be valuable. (Potential examples: infrabayesianism)
There isn’t (yet) institutional infrastructure to support MATS grads to do useful work together as part of a team focused on the same (or very similar) research agendas, and that this is the case for multiple nascent and established research agendas. They are forced to go to academia and disperse across the world instead of being able to work together in one location. (Potential examples: selection theorems, multi-agent alignment (of the sort that Caspar Oesterheld and company work on))
There aren’t enough research managers in existing established alignment research organizations or frontier labs to enable MATS grads to work on the research directions they consider extremely high value, and would benefit from multiple people working together on (Potential examples: activation steering)
I’m pretty sure that Ryan does not mean to say that MATS grads cannot do useful work on their own. The point is that we don’t yet have the institutional infrastructure to absorb, enable, and scale new researchers the way our civilization has for existing STEM fields via, say, PhD programs or yearlong fellowships at OpenAI/MSR/DeepMind (which are also pretty rare). AFAICT, the most valuable part of such infrastructure in general is the ability to co-locate researchers working on the same or similar research problems—this is standard for academic and industry research groups, for example, and from experience I know that being able to do so is invaluable. Another extremely valuable facet of institutional infrastructure that enables researchers is the ability to delegate operations and logistics problems—particularly the difficulty of finding grant funding, interfacing with other organizations, getting paperwork handled, etc.
I keep getting more and more convinced, as time passes, that it would be more valuable for me to work on building the infrastructure to enable valuable teams and projects, than to simply do alignment research while disregarding such bottlenecks to this research ecosystem.
@Elizabeth, Mesa nails it above. I would also add that I am conceptualizing impactful AI safety research as the product of multiple reagents, including talent, ideas, infrastructure, and funding. In my bullet point, I was pointing to an abundance of talent and ideas relative to infrastructure and funding. I’m still mostly working on talent development at MATS, but I’m also helping with infrastructure and funding (e.g., founding LISA, advising Catalyze Impact, regranting via Manifund) and I want to do much more for these limiting reagents.
Also note that historically many individuals entering AI safety seem to have been pursuing the “Connector” path, when most jobs now (and probably in the future) are “Iterator”-shaped, and larger AI safety projects are also principally bottlenecked by “Amplifiers”. The historical focus on recruiting and training Connectors to the detriment of Iterators and Amplifiers has likely contributed to this relative talent shortage. A caveat: Connectors are also critical for founding new research agendas and organizations, though many self-styled Connectors would likely substantially benefit as founders by improving some Amplifier-shaped soft skills, including leadership, collaboration, networking, and fundraising.
I interpret your comment as assuming that new researchers with good ideas produce more impact on their own than in teams working towards a shared goal; this seems false to me. I think that independent research is usually a bad bet in general and that most new AI safety researchers should be working on relatively few impactful research directions, most of which are best pursued within a team due to the nature of the research (though some investment in other directions seems good for the portfolio).
I’ve addressed this a bit in thread, but here are some more thoughts:
New AI safety researchers seem to face mundane barriers to reducing AI catastrophic risk, including funding, infrastructure, and general executive function.
MATS alumni are generally doing great stuff (~46% currently work in AI safety/control, ~1.4% work on AI capabilities), but we can do even better.
Like any other nascent scientific/engineering discipline, AI safety will produce more impactful research with scale, albeit with some diminishing returns on impact eventually (I think we are far from the inflection point, however).
MATS alumni, as a large swathe of the most talented new AI safety researchers in my (possibly biased) opinion, should ideally not experience mundane barriers to reducing AI catastrophic risk.
Independent research seems worse than team-based research for most research that aims to reduce AI catastrophic risk:
“Pair-programming”, builder-breaker, rubber-ducking, etc. are valuable parts of the research process and are benefited by working in a team.
Funding insecurity and grantwriting responsibilities are larger for independent researchers and obstruct research.
Orgs with larger teams and discretionary funding can take on interns to help scale projects and provide mentorship.
Good prosaic AI safety research largely looks more like large teams doing engineering and less like lone geniuses doing maths. Obviously, some lone genius researchers (especially on mathsy non-prosaic agendas) seem great for the portfolio too, but these people seem hard to find/train anyways (so there is often more alpha in the former by my lights). Also, I have doubts that the optimal mechanism to incentivize “lone genius research” is via small independent grants instead of large bounties and academic nerdsniping.
Therefore, more infrastructure and funding for MATS alumni, who are generally value-aligned and competent, is good for reducing AI catastrophic risk in expectation.
I don’t believe that, although I see how my summary could be interpreted that way. I agree with basically all the reasons in your recent comment and most in the original comment. I could add a few reasons of my own doing independent grant-funded work sucks. But I think it’s really important to track how founding projects tracks to increased potential safety instead of intermediates, and push hard against potential tail wagging the dog scenarios.
I was trying to figure out why this was important to me, given how many of your points I agree with. I think it’s a few things:
Alignment work seems to be prone to wagging the dog, and is harder to correct, due to poor feedback loops.
The consequences of this can be dire
making it harder to identify and support the best projects.
making it harder to identify and stop harmful projects
making it harder to identify when a decent idea isn’t panning out, leading to people and money getting stuck in the mediocre project instead of moving on.
One of the general concerns about MATS is it spins up potential capabilities researchers. If the market can’t absorb the talent, that suggests maybe MATS should shrink.
OTOH if you told me that for every 10 entrants MATS spins up 1 amazing safety researcher and 9 people who need makework to prevent going into capabilities, I’d be open to arguments that that was a good trade.