My main answer is capacity constrains at central places. I think you are not considering how small the community was.
One somewhat representative anecdote: sometime in ~2019, at FHI, there was a discussion that the “AI ethics” and “AI safety” research communities seem to be victims of unfortunate polarization dynamics, where even while in the Platonic realm of ideas concerns tracked by the people are compatible, there is somewhat unfortunate social dynamic, where loud voices on both sides are extremely dismissive of the other community. My guess at that time was the divide has decent chance of exploding when AI worries go mainstream (like, arguments about AI risk facing vociferous opposition from part of academia entrenched under the “ethics” flag), and my proposal was to do something about it, as there were some opportunities to pre-empt/heal this, e.g. by supporting people from both camps to visit each others conferences, or writing papers explaining the concerns in a language of the other camp. Overall this was often specific and actionable. The only problem was … “who has time to work on this”, and the answer was “no one”.
If you looked at what senior staff at FHI was working on, the counterfactuals were e.g. Toby Ord writing The Precipice. I think even with the benefit of hindsight, that was clearly more valuable—if today you see UN Security Council discussing AI risk and at least some people in the room have somewhat sane models, it’s also because a bunch of people at UN read The Precipice and started to think about xrisk and AI risk.
If you looked at junior people, I was juggling already quite high number of balls, including research on active inference minds and implications for value learning, research on technical problems in comprehensive AI services, organizing academic-friendly Human-aligned AI summer school, organizing Epistea summer experiment, organizing ESPR, participating in a bunch of CFAR things. Even in retrospect, I think all of these bets were better than me trying to do something about the expected harmful AI ethics vs AI safety flamewar.
Similarly, we had an early-stage effort on “robust communication”, attempting to design a system for testing robustly good public communication about xrisk and similar sensitive topics (including e.g. developing good shareable models of future problems fitting in the Overton window). It went nowhere because … there just weren’t any people. FHI had dozens of topic like that where a whole org should work on them, but the actual attention was about 0.2FTE of someone junior.
Overall I think with the benefit of hindsight, a lot of what FHI worked on was more or less what you suggest should have been done. It’s true that this was never in the spotlight on LessWrong—I guess in 2019 the prevailing LW sentiment would be that Toby Ord engaging with UN is most likely useless waste of time.
This is fantastic information, thank you for taking the time.
One of my big takeaways from all of the comments on this post is a big update to my understanding of the “AI Risk” community and that LW was not actually the epicenter and there were significant efforts being made elsewhere that didn’t necessarily circle back to LW.
That is very encouraging actually!
The other big update is what you say: There were just so few people with the time and ability to work on these things.
My main answer is capacity constrains at central places. I think you are not considering how small the community was.
One somewhat representative anecdote: sometime in ~2019, at FHI, there was a discussion that the “AI ethics” and “AI safety” research communities seem to be victims of unfortunate polarization dynamics, where even while in the Platonic realm of ideas concerns tracked by the people are compatible, there is somewhat unfortunate social dynamic, where loud voices on both sides are extremely dismissive of the other community. My guess at that time was the divide has decent chance of exploding when AI worries go mainstream (like, arguments about AI risk facing vociferous opposition from part of academia entrenched under the “ethics” flag), and my proposal was to do something about it, as there were some opportunities to pre-empt/heal this, e.g. by supporting people from both camps to visit each others conferences, or writing papers explaining the concerns in a language of the other camp. Overall this was often specific and actionable. The only problem was … “who has time to work on this”, and the answer was “no one”.
If you looked at what senior staff at FHI was working on, the counterfactuals were e.g. Toby Ord writing The Precipice. I think even with the benefit of hindsight, that was clearly more valuable—if today you see UN Security Council discussing AI risk and at least some people in the room have somewhat sane models, it’s also because a bunch of people at UN read The Precipice and started to think about xrisk and AI risk.
If you looked at junior people, I was juggling already quite high number of balls, including research on active inference minds and implications for value learning, research on technical problems in comprehensive AI services, organizing academic-friendly Human-aligned AI summer school, organizing Epistea summer experiment, organizing ESPR, participating in a bunch of CFAR things. Even in retrospect, I think all of these bets were better than me trying to do something about the expected harmful AI ethics vs AI safety flamewar.
Similarly, we had an early-stage effort on “robust communication”, attempting to design a system for testing robustly good public communication about xrisk and similar sensitive topics (including e.g. developing good shareable models of future problems fitting in the Overton window). It went nowhere because … there just weren’t any people. FHI had dozens of topic like that where a whole org should work on them, but the actual attention was about 0.2FTE of someone junior.
Overall I think with the benefit of hindsight, a lot of what FHI worked on was more or less what you suggest should have been done. It’s true that this was never in the spotlight on LessWrong—I guess in 2019 the prevailing LW sentiment would be that Toby Ord engaging with UN is most likely useless waste of time.
This is fantastic information, thank you for taking the time.
One of my big takeaways from all of the comments on this post is a big update to my understanding of the “AI Risk” community and that LW was not actually the epicenter and there were significant efforts being made elsewhere that didn’t necessarily circle back to LW.
That is very encouraging actually!
The other big update is what you say: There were just so few people with the time and ability to work on these things.