Relatedly, rereading this post was what prompted me to write this stub post:
I’m fairly concerned with the practice of telling people who “really care about AI safety” to go into AI capabilities research, unless they are very junior researchers who are using general AI research as a place to improve their skills until they’re able to contribute to AI safety later. (See Leveraging Academia).
The reason is not a fear that they will contribute to AI capabilities advancement in some manner that will be marginally detrimental to the future. It’s also not a fear that they’ll fail to change the company’s culture in the ways they’d hope, and end up feeling discouraged. What I’m afraid of is that they’ll feel pressure to start pretending to themselves, or to others, that their work is “relevant to safety”. Then what we end up with are companies and departments filled with people who are “concerned about safety”, creating a false sense of security that something relevant is being done, when all we have are a bunch of simmering concerns and concomitant rationalizations.
This fear of mine requires some context from my background as a researcher. I see this problem with environmentalists who “really care about climate change”, who tell themselves they’re “working on it” by studying the roots of a fairly arbitrary species of tree in a fairly arbitrary ecosystem that won’t generalize to anything likely to help with climate change.
My assessment that their work won’t generalize is mostly not from my own outside view; it comes from asking the researcher about how their work is likely to have an impact, and getting a response that either says nothing more than “I’m not sure, but it seems relevant somehow”, or an argument with a lot of caveats like “X might help with Y, which might help with Z, which might help with climate change, but we really can’t be sure, and it’s not my job to defend the relevance of my work. It’s intrinsically interesting to me, and you never know if something could turn out to be useful that seemed useless at first.”
At the same time, I know other climate scientists who seem to have actually done an explicit or implicit Fermi estimate for the probability that they will personally soon discover a species of bacteria that could safely scrub the Earth’s atmosphere of excess carbon. That’s much better.
Relatedly, rereading this post was what prompted me to write this stub post: