My interpretation of those at fault are the field builders and funders. That is part of the reason I quit doing alignment. The entire funding landscape feels incredibly bait and switch: come work for us! We are desperate for talent! The alignment problem is the hardest issue of the century! (Cue 2 years and an SBF later) Erm, no, we don’t fund AI safety startups or interp, and we want to see tangible results in a few narrow domains...
In particular, I advocate for the concept of endorsing work with a nonlinear apparent progress rate. Call it ‘slow work’ or something. Often a lot of hard things look like they’re getting nowhere but all the small failures add up to something big. This is also why I do not recommend MATS as a one size fits all solution for people joining the alignment field: some people do better with slow work, and with carefully thinking about where they are heading with direction and intent, not just putting their heads down to ‘get something done’. In fact, this mindset gave me burnout earlier this year.
The people not at fault are those who are middle of the pack undergrads or not Physicists Doing Real Things. This is a system wide problem.
I agree with ‘don’t streetlight’ and ‘we should move the field towards riskier and harder projects’. For me, this means bets on indviduals, not tangible projects. What I mean by this, is similar to how Entrepreneur First makes bets on founders, not products, funders should make bets on people, who they believe with enough time, can make meaningful progress that’s orthogonal to existing directions.
I don’t agree at all with the elitist take of ‘and that is why only physics postdocs (or those of similar capability—whatever that means) are only capable of doing real work’, at all. This take is quite absurd to me, and frankly a little angering (because I have the impression it’s quietly shared among certain circles the same way some STEM majors look down on non-STEM majors), but I think that was the goal, so achieved. In particular, here are the reasons why I disagree:
It isn’t true that this skill can only be found in ‘physics postdocs’. The ability to push through hard things and technical fluency can be gained by a good chunk of STEM degrees. Anyone can open a mathematics textbook and read.
Critical thinking is more important in my opinion than technical fluency, for avoiding falling into streetlights and deferring opinions to others.
On a meta level, beliefs like this lead to segregation of the research community in a way that is unhealthy. Promoting further segregration is not ideal. There must be balance. I think any sensible human is capable of maintaining their own opinions while taking in those of others.
Agree with the comments about the car and driver. My current opinion is physicists need to work with non-physicists. There is a risk otherwise of working only on interesting problems that lead to us maybe ‘solving alignment’ (with 139 caveats) by 2178.
In fact, this mindset gave me burnout earlier this year.
I relate pretty strongly to this. I think almost all junior researchers are incentivised to ‘paper grind’ for longer than is correct. I do think there are pretty strong returns to having one good paper for credibility reasons; it signals that you are capable of doing AI safety research, and thus makes it easier to apply for subsequent opportunities.
Over the past 6 months I’ve dropped the paper grind mindset and am much happier for this. Notably, were it not for short term grants where needing to visibly make progress is important, I would have made this update sooner. Another take that I have is that if you have the flexibility to do so (e.g. by already having stable funding, perhaps via being a PhD student), front-loading learning seems good. See here for a related take by Rohin. Making progress on hard problems requires understanding things deeply, in a way which making progress on easy problems that you could complete during e.g. MATS might not.
Some broad points:
My interpretation of those at fault are the field builders and funders. That is part of the reason I quit doing alignment. The entire funding landscape feels incredibly bait and switch: come work for us! We are desperate for talent! The alignment problem is the hardest issue of the century! (Cue 2 years and an SBF later) Erm, no, we don’t fund AI safety startups or interp, and we want to see tangible results in a few narrow domains...
In particular, I advocate for the concept of endorsing work with a nonlinear apparent progress rate. Call it ‘slow work’ or something. Often a lot of hard things look like they’re getting nowhere but all the small failures add up to something big. This is also why I do not recommend MATS as a one size fits all solution for people joining the alignment field: some people do better with slow work, and with carefully thinking about where they are heading with direction and intent, not just putting their heads down to ‘get something done’. In fact, this mindset gave me burnout earlier this year.
The people not at fault are those who are middle of the pack undergrads or not Physicists Doing Real Things. This is a system wide problem.
I agree with ‘don’t streetlight’ and ‘we should move the field towards riskier and harder projects’. For me, this means bets on indviduals, not tangible projects. What I mean by this, is similar to how Entrepreneur First makes bets on founders, not products, funders should make bets on people, who they believe with enough time, can make meaningful progress that’s orthogonal to existing directions.
I don’t agree at all with the elitist take of ‘and that is why only physics postdocs (or those of similar capability—whatever that means) are only capable of doing real work’, at all. This take is quite absurd to me, and frankly a little angering (because I have the impression it’s quietly shared among certain circles the same way some STEM majors look down on non-STEM majors), but I think that was the goal, so achieved. In particular, here are the reasons why I disagree:
It isn’t true that this skill can only be found in ‘physics postdocs’. The ability to push through hard things and technical fluency can be gained by a good chunk of STEM degrees. Anyone can open a mathematics textbook and read.
Critical thinking is more important in my opinion than technical fluency, for avoiding falling into streetlights and deferring opinions to others.
On a meta level, beliefs like this lead to segregation of the research community in a way that is unhealthy. Promoting further segregration is not ideal. There must be balance. I think any sensible human is capable of maintaining their own opinions while taking in those of others.
Agree with the comments about the car and driver. My current opinion is physicists need to work with non-physicists. There is a risk otherwise of working only on interesting problems that lead to us maybe ‘solving alignment’ (with 139 caveats) by 2178.
I relate pretty strongly to this. I think almost all junior researchers are incentivised to ‘paper grind’ for longer than is correct. I do think there are pretty strong returns to having one good paper for credibility reasons; it signals that you are capable of doing AI safety research, and thus makes it easier to apply for subsequent opportunities.
Over the past 6 months I’ve dropped the paper grind mindset and am much happier for this. Notably, were it not for short term grants where needing to visibly make progress is important, I would have made this update sooner. Another take that I have is that if you have the flexibility to do so (e.g. by already having stable funding, perhaps via being a PhD student), front-loading learning seems good. See here for a related take by Rohin. Making progress on hard problems requires understanding things deeply, in a way which making progress on easy problems that you could complete during e.g. MATS might not.