because LW/AF do not have established standards of rigor like ML, they end up operating more like a less-functional social science field, where (I’ve heard) trends, personality, and celebrity play an outsized role in determining which research is valorized by the field.
In addition, the AI x-safety field is now rapidly expanding. There is a huge amount of status to be collected by publishing quickly and claiming large contributions.
In the absence of rigor and metrics, the incentives are towards: - setting new research directions, and inventing new cool terminology; - using mathematics in a way that impresses, but is too low-level to yield a useful claim; - and vice versa, relying too much on complex philosophical insights without empirical work; - getting approval from alignment research insiders.
On the other hand, the current community believes that getting AI x-safety right is the most important research question of all time. Most people would not publish something just for their career advancement, if it meant sucking oxygen from more promising research directions.
This might be a mitigating factor for my comment above. I am curious about what happened research fields which had “change/save the world’ vibes. Was environmental science immune to similar issues?
I actually agree that empirical work generally outperforms theoretical work or philosophical work, but in that tweet thread I question why he suggests the Turing Test as relating anything to x-risk.
In addition, the AI x-safety field is now rapidly expanding.
There is a huge amount of status to be collected by publishing quickly and claiming large contributions.
In the absence of rigor and metrics, the incentives are towards:
- setting new research directions, and inventing new cool terminology;
- using mathematics in a way that impresses, but is too low-level to yield a useful claim;
- and vice versa, relying too much on complex philosophical insights without empirical work;
- getting approval from alignment research insiders.
See also the now ancient Troubling Trends in Machine Learning Scholarship.
I expect the LW/AF community microcosm will soon reproduce many of of those failures.
On the other hand, the current community believes that getting AI x-safety right is the most important research question of all time. Most people would not publish something just for their career advancement, if it meant sucking oxygen from more promising research directions.
This might be a mitigating factor for my comment above. I am curious about what happened research fields which had “change/save the world’ vibes. Was environmental science immune to similar issues?
I actually agree that empirical work generally outperforms theoretical work or philosophical work, but in that tweet thread I question why he suggests the Turing Test as relating anything to x-risk.