Thanks for your comments Akash. I think I have two main points I want to address.
I agree that it’s very good that the field of AI Alignment is very competitive! I did not want to imply that this is a bad thing. I was mainly trying to point out that from my point of view, it seems like overall there are more qualified and experienced people than there are jobs at large organizations. And in order to fill that gap we would need more senior researchers, who then can follow their research agendas and hire people (and fund orgs), which is however hard to achieve. One disclaimer I want to note is that I do not work at a large org, and I do not precisely know what kinds of hiring criteria they have, i.e. it is possible that in their view we still lack talented enough people. However, from the outside, it definitely does look like there are many experienced researchers.
It is possible that my previous statement may have been misinterpreted. I wish to clarify that my concerns do not pertain to funding being a challenge. I did not want to make an assertion about funding in general, and if my words gave that impression, I apologize. I do not know enough about the funding landscape to know whether there is a lot or not enough funding (especially in recent months).
I agree with you that, for all I know, it’s feasible to get funding for independent researchers (and definitely easier than doing a Ph.D. or getting a full-time position). I also agree that independent research seems to be more heavily funded than in other fields.
My point was mainly the following:
Many people have joined the field (which is great!), or at least it looks like it from the outside. 80000 hours etc. still recommend switching to AI Alignment, so it seems likely that more people will join.
I believe that there are many opportunities for people to up-skill to a certain level if they want to join the field (Seri Mats, AI safety camp, etc.).
However full-time positions (for example at big labs) are very limited. This also makes sense, since they can only hire so many people a year.
It seems like the most obvious option for people who want to stay in the field is to do independent research (and apply for grants). I think it’s great that people do independent research and that one has the opportunity to get grants.
However, doing independent research is not always ideal for many reasons (as outlined in my main comment). Note I’m not saying it doesn’t make sense at all, it definitely has its merits.
In order to have more full-time positions we need more senior people, who can then fund their organizations, or independently hire people, etc. Independent research does not seem like a promising avenue to me, to groom senior researchers. It’s essential that you can learn from people that are better than you and be in a good environment (yes there are exceptions like Einstein, but I think most researchers I know would agree with that statement).
So to me, the biggest bottleneck of all is how can we get many great researchers and groom them to be senior researchers who can lead their own orgs. I think that so far we have really optimized for getting people into the field (which is great). But we haven’t really found a solution to grooming senior researchers (again, some programs try to do that and I’m aware that this takes time). Overall I believe that this is a hard problem and probably others have already thought about it. I’m just trying to make that point in case nobody has written it up yet. Especially if people are trying to do AI safety field building it seems to me that, coming up with ways to groom senior researchers is a top priority.
Ultimately I’m not even sure whether there is a clear solution to this problem. The field is still very new and it’s amazing what has already happened. It’s probable that it just takes time for the field to mature and people getting more experience. I think I mostly wanted to point this out, even if it is maybe obvious.
Overall I believe that this is a hard problem and probably others have already thought about it.
I’m not sure people seriously thought about this before, your perspective seems rather novel.
I think existing labs themselves are the best vehicle to groom new senior researchers. Anthropic, Redwood Research, ARC, and probably other labs were all found by ex-staff of existing labs at the time (except that maybe one shouldn’t credit OpenAI for “grooming” Paul Cristiano to senior level, but anyways).
It’s unclear what field-building projects could incentivise labs to part with their senior researchers and let them spin off their own labs. Or to groom senior researchers “faster”, so to speak.
If the theory that AI alignment is extremely competitive is right, then logically both the labs shouldn’t cling to their senior people too much (because it will be relatively easy to replace them), and senior researchers shouldn’t worry about starting their own projects too much because they know they can assemble a very competent team very quickly.
It seems that it’s only the funding for these new labs and their organisational strategy which could be a point of uncertainty for senior researchers that could deter them from starting their own projects (apart from, of course, just being content with the project they are involved in at their current jobs, and their level of influence on research agendas).
So, maybe the best field-building project that could be done in this area is someone offering knowledge about and support through founding, funding, and setting a strategy for new labs (which may range from brief informal consultation to more structured support, a-la “incubator for AI safety labs”) and advertise this offering among the staff of existing AI labs.
Thanks for your comments Akash. I think I have two main points I want to address.
I agree that it’s very good that the field of AI Alignment is very competitive! I did not want to imply that this is a bad thing. I was mainly trying to point out that from my point of view, it seems like overall there are more qualified and experienced people than there are jobs at large organizations. And in order to fill that gap we would need more senior researchers, who then can follow their research agendas and hire people (and fund orgs), which is however hard to achieve. One disclaimer I want to note is that I do not work at a large org, and I do not precisely know what kinds of hiring criteria they have, i.e. it is possible that in their view we still lack talented enough people. However, from the outside, it definitely does look like there are many experienced researchers.
It is possible that my previous statement may have been misinterpreted. I wish to clarify that my concerns do not pertain to funding being a challenge. I did not want to make an assertion about funding in general, and if my words gave that impression, I apologize. I do not know enough about the funding landscape to know whether there is a lot or not enough funding (especially in recent months).
I agree with you that, for all I know, it’s feasible to get funding for independent researchers (and definitely easier than doing a Ph.D. or getting a full-time position). I also agree that independent research seems to be more heavily funded than in other fields.
My point was mainly the following:
Many people have joined the field (which is great!), or at least it looks like it from the outside. 80000 hours etc. still recommend switching to AI Alignment, so it seems likely that more people will join.
I believe that there are many opportunities for people to up-skill to a certain level if they want to join the field (Seri Mats, AI safety camp, etc.).
However full-time positions (for example at big labs) are very limited. This also makes sense, since they can only hire so many people a year.
It seems like the most obvious option for people who want to stay in the field is to do independent research (and apply for grants). I think it’s great that people do independent research and that one has the opportunity to get grants.
However, doing independent research is not always ideal for many reasons (as outlined in my main comment). Note I’m not saying it doesn’t make sense at all, it definitely has its merits.
In order to have more full-time positions we need more senior people, who can then fund their organizations, or independently hire people, etc. Independent research does not seem like a promising avenue to me, to groom senior researchers. It’s essential that you can learn from people that are better than you and be in a good environment (yes there are exceptions like Einstein, but I think most researchers I know would agree with that statement).
So to me, the biggest bottleneck of all is how can we get many great researchers and groom them to be senior researchers who can lead their own orgs. I think that so far we have really optimized for getting people into the field (which is great). But we haven’t really found a solution to grooming senior researchers (again, some programs try to do that and I’m aware that this takes time). Overall I believe that this is a hard problem and probably others have already thought about it. I’m just trying to make that point in case nobody has written it up yet. Especially if people are trying to do AI safety field building it seems to me that, coming up with ways to groom senior researchers is a top priority.
Ultimately I’m not even sure whether there is a clear solution to this problem. The field is still very new and it’s amazing what has already happened. It’s probable that it just takes time for the field to mature and people getting more experience. I think I mostly wanted to point this out, even if it is maybe obvious.
I’m not sure people seriously thought about this before, your perspective seems rather novel.
I think existing labs themselves are the best vehicle to groom new senior researchers. Anthropic, Redwood Research, ARC, and probably other labs were all found by ex-staff of existing labs at the time (except that maybe one shouldn’t credit OpenAI for “grooming” Paul Cristiano to senior level, but anyways).
It’s unclear what field-building projects could incentivise labs to part with their senior researchers and let them spin off their own labs. Or to groom senior researchers “faster”, so to speak.
If the theory that AI alignment is extremely competitive is right, then logically both the labs shouldn’t cling to their senior people too much (because it will be relatively easy to replace them), and senior researchers shouldn’t worry about starting their own projects too much because they know they can assemble a very competent team very quickly.
It seems that it’s only the funding for these new labs and their organisational strategy which could be a point of uncertainty for senior researchers that could deter them from starting their own projects (apart from, of course, just being content with the project they are involved in at their current jobs, and their level of influence on research agendas).
So, maybe the best field-building project that could be done in this area is someone offering knowledge about and support through founding, funding, and setting a strategy for new labs (which may range from brief informal consultation to more structured support, a-la “incubator for AI safety labs”) and advertise this offering among the staff of existing AI labs.