Trying to break into MIRI-style[1] research seems to be much, much harder than trying to break into ML-style safety research. This is worrying if you believe this research to be important[2]. I’ll examine two kinds of causes: those which come from MIRI-style research being a niche area and those which go beyond this:
Challenges beyond MIRI-style research being a niche area:
MIRI doesn’t seem to be running internships[3] or running their AI safety for computer scientists workshops
If you try to break into ML-style Safety and fail, you can always be reuse at least part of what you’ve learned to obtain a highly-compensated role in industry. Agent foundations knowledge is highly niche and unlikely to be used elsewhere.
You can park in a standard industry job for a while in order to earn career capital for ML-style safety. Not so for MIRI-style research.
MIRI publishes a lot less material these days. I support this decision I support as infohazards deserve to be taken seriously, but it also makes it harder to contribute.
There are well-crafted materials for learning a lot of the prerequisites for ML-style safety.
There seems to be a natural pathway of studying a masters then pursuing a PhD to break into ML-style safety. There are a large number of scholarships available and many countries offer loans or income support.
The above opportunities mean that there are more ways to gauge fit for ML-style safety research.
There’s no equivalent to submitting a paper[4]. If a paper passes review, then it gains a certain level of credibility. There are upvotes, but this signaling mechanism is more distorted by popularity or accessibility. Further, unlike writing an academic paper, writing alignment forum posts won’t provide credibility outside of the field.
Challenges that come from being a niche area
I think this probably should be a niche area. It would be a bit strange if foundations work were the majority of the research. Nonetheless, it’s worth highlighting some of the implications:
It is much harder to find people with similar interests to collaborate with or mentor you. Compare to how easy it is to meet a bunch of people interested in ML-style research by attending EA meetups or EAGx.
If you want to feel part of the AI safety community and join in the conversations people are having, you will have to spend time learning about ML-style research. While is likely valuable to broaden your scope as this can lead to cross-pollination, it also sucks up time when you could be learning about MIRI-style research.
Further Thoughts
I think it’s worth thinking about what this looks like overall. If you want to try breaking into MIRI-style research, the most likely path looks like saving up 3-12 months runway[5]. 3 months might be possible if you’ve been consistently working on things in your free time and you’ve already read a lot of the material that you need to read + made substantial research progress. That said, even if you’re able to produce material to prove yourself in 3 months, you’d probably need an extra month or two to obtain funding and you always need more runway than the minimum possible time. It would be possible to apply for an LTFF grant to support this research, but it’s probably easier to build up the credibility for ML-style research. Further, if you fail, then you haven’t learned skills nor gained credibility that would assist you for any other paths.
I suspect that these considerations not only significantly curtail the number of people who pursue this path, but also ensures that those who do pursue it will often only do so after significant delay.
I guess it’s particularly interesting that these difficulties exist in light of the large amount of funding that now appears to be available for AI Safety. In fact, MIRI is now so well-funded that they didn’t even bother with a fundraiser this year. I’m not saying that it’s easy to resolve these problems by throwing money at them, merely that the availability of funds opens up a lot more options for mitigation.
I received a comment suggesting that most of what I’ve written in this article would hold if MIRI-style research” were replaced everywhere with “preparadigmatic research”.
The inability to submit papers also reduces the ability to obtain review and feedback. Less Wrong does have a feedback option, but you can’t expect the same level of expertise and engagement that you would if you would receive from submitting a journal article.
Challenges with Breaking into MIRI-Style Research
Trying to break into MIRI-style[1] research seems to be much, much harder than trying to break into ML-style safety research. This is worrying if you believe this research to be important[2]. I’ll examine two kinds of causes: those which come from MIRI-style research being a niche area and those which go beyond this:
Challenges beyond MIRI-style research being a niche area:
MIRI doesn’t seem to be running internships[3] or running their AI safety for computer scientists workshops
If you try to break into ML-style Safety and fail, you can always be reuse at least part of what you’ve learned to obtain a highly-compensated role in industry. Agent foundations knowledge is highly niche and unlikely to be used elsewhere.
You can park in a standard industry job for a while in order to earn career capital for ML-style safety. Not so for MIRI-style research.
MIRI publishes a lot less material these days. I support this decision I support as infohazards deserve to be taken seriously, but it also makes it harder to contribute.
There are well-crafted materials for learning a lot of the prerequisites for ML-style safety.
There seems to be a natural pathway of studying a masters then pursuing a PhD to break into ML-style safety. There are a large number of scholarships available and many countries offer loans or income support.
The above opportunities mean that there are more ways to gauge fit for ML-style safety research.
There’s no equivalent to submitting a paper[4]. If a paper passes review, then it gains a certain level of credibility. There are upvotes, but this signaling mechanism is more distorted by popularity or accessibility. Further, unlike writing an academic paper, writing alignment forum posts won’t provide credibility outside of the field.
Challenges that come from being a niche area
I think this probably should be a niche area. It would be a bit strange if foundations work were the majority of the research. Nonetheless, it’s worth highlighting some of the implications:
General AI safety programs and support—ie. AI Safety Fundamentals Course, AI Safety Support, AI Safety Camp, Alignment Newsletter, ect. are naturally going to strongly focus on ML-style research and might not even have the capability to vet MIRI-style research.
It is much harder to find people with similar interests to collaborate with or mentor you. Compare to how easy it is to meet a bunch of people interested in ML-style research by attending EA meetups or EAGx.
If you want to feel part of the AI safety community and join in the conversations people are having, you will have to spend time learning about ML-style research. While is likely valuable to broaden your scope as this can lead to cross-pollination, it also sucks up time when you could be learning about MIRI-style research.
Further Thoughts
I think it’s worth thinking about what this looks like overall. If you want to try breaking into MIRI-style research, the most likely path looks like saving up 3-12 months runway[5]. 3 months might be possible if you’ve been consistently working on things in your free time and you’ve already read a lot of the material that you need to read + made substantial research progress. That said, even if you’re able to produce material to prove yourself in 3 months, you’d probably need an extra month or two to obtain funding and you always need more runway than the minimum possible time. It would be possible to apply for an LTFF grant to support this research, but it’s probably easier to build up the credibility for ML-style research. Further, if you fail, then you haven’t learned skills nor gained credibility that would assist you for any other paths.
I suspect that these considerations not only significantly curtail the number of people who pursue this path, but also ensures that those who do pursue it will often only do so after significant delay.
I guess it’s particularly interesting that these difficulties exist in light of the large amount of funding that now appears to be available for AI Safety. In fact, MIRI is now so well-funded that they didn’t even bother with a fundraiser this year. I’m not saying that it’s easy to resolve these problems by throwing money at them, merely that the availability of funds opens up a lot more options for mitigation.
I received a comment suggesting that most of what I’ve written in this article would hold if MIRI-style research” were replaced everywhere with “preparadigmatic research”.
I’m still largely undecided as to whether Agent Foundations research is important. I’m mostly pursuing it due to comparative advantage.
Although Evan Hubringer, a MIRI researcher, is involved in running internships separately.
The inability to submit papers also reduces the ability to obtain review and feedback. Less Wrong does have a feedback option, but you can’t expect the same level of expertise and engagement that you would if you would receive from submitting a journal article.
Admittedly, applying to the EA hotel might also be an option depending on your life circumstances.