I’m still not sure I buy the examples. In the early parts of the post you seem to contrast ‘machine learning research agendas’ with ‘foundational and mathematical’/‘agent foundations’ type stuff. Mechanistic interpretability can be quite mathematical but surely it falls into the former category? i.e. it is essentially ML work as opposed to constituting an example of people doing “mathematical and foundational” work.
I can’t say much about the Goodhart’s Law comment but it seems at best unclear that its link to goal misgeneralization is an example of the kind you are looking for (i.e. in the absence of much more concrete examples, I have approximately no reason to believe it has anything to do with what one would call mathematical work).
I’m not really clear what you mean by not buying the example. You certainly seem to understand the distinction I’m drawing—mechanistic interpretability is definitely not what I mean by “mathematical AI safety,” though I agree there is math involved.
And I think the work on goal misgeneralization was conceptualized in ways directly related to Goodhart, and this type of problem inspired a number of research projects, including quantilizers, which is certainly agent-foundations work. I’ll point here for more places the agents foundations people think it is relevant.
Ah OK, I think I’ve worked out where some of my confusion is coming from: I don’t really see any argument for why mathematical work may be useful, relative to other kinds of foundational conceptual work. e.g. you write (with my emphasis): “Current mathematical research could play a similar role in the coming years...” But why might it? Isn’t that where you need to be arguing?
The examples seem to be of cases where people have done some kind of conceptual foundational work which has later gone on to influence/inspire ML work. But early work on deception or goodhart was not mathematical work, that’s why I don’t understand how these are examples.
I think th dispute here is that you’re interpreting mathematical too narrowly, and almost all of the work happening in agent foundations and similar is exactly what was being worked on by “mathematical AI research” 5-7 years ago. The argument was that those approaches have been fruitful, and we should expect them to continue to be so—if you want to call that “foundational conceptual research” instead of “Mathematical AI research,” that’s fine..
I’m still not sure I buy the examples. In the early parts of the post you seem to contrast ‘machine learning research agendas’ with ‘foundational and mathematical’/‘agent foundations’ type stuff. Mechanistic interpretability can be quite mathematical but surely it falls into the former category? i.e. it is essentially ML work as opposed to constituting an example of people doing “mathematical and foundational” work.
I can’t say much about the Goodhart’s Law comment but it seems at best unclear that its link to goal misgeneralization is an example of the kind you are looking for (i.e. in the absence of much more concrete examples, I have approximately no reason to believe it has anything to do with what one would call mathematical work).
I’m not really clear what you mean by not buying the example. You certainly seem to understand the distinction I’m drawing—mechanistic interpretability is definitely not what I mean by “mathematical AI safety,” though I agree there is math involved.
And I think the work on goal misgeneralization was conceptualized in ways directly related to Goodhart, and this type of problem inspired a number of research projects, including quantilizers, which is certainly agent-foundations work. I’ll point here for more places the agents foundations people think it is relevant.
Ah OK, I think I’ve worked out where some of my confusion is coming from: I don’t really see any argument for why mathematical work may be useful, relative to other kinds of foundational conceptual work. e.g. you write (with my emphasis): “Current mathematical research could play a similar role in the coming years...” But why might it? Isn’t that where you need to be arguing?
The examples seem to be of cases where people have done some kind of conceptual foundational work which has later gone on to influence/inspire ML work. But early work on deception or goodhart was not mathematical work, that’s why I don’t understand how these are examples.
I think th dispute here is that you’re interpreting mathematical too narrowly, and almost all of the work happening in agent foundations and similar is exactly what was being worked on by “mathematical AI research” 5-7 years ago. The argument was that those approaches have been fruitful, and we should expect them to continue to be so—if you want to call that “foundational conceptual research” instead of “Mathematical AI research,” that’s fine..