Mesa-optimizers are in the search space and would achieve high scores in the training set, so why wouldn’t we expect to see them?
I like this as a statement of the core concern (modulo some worries about the concept of mesa-optimisation, which I’ll save for another time).
With respect to formalization, I did say up front that less-formal work, and empirical work, is still valuable.
I missed this disclaimer, sorry. So that assuages some of my concerns about balancing types of work. I’m still not sure what intuitions or arguments underlie your optimism about formal work, though. I assume that this would be fairly time-consuming to spell out in detail—but given that the core point of this post is to encourage such work, it seems worth at least gesturing towards those intuitions, so that it’s easier to tell where any disagreement lies.
To me, the post as written seems like enough to spell out my optimism… there multiple directions for formal work which seem under-explored to me. Well, I suppose I didn’t focus on explaining why things seem under-explored. Hopefully the writeup-to-come will make that clear.
I like this as a statement of the core concern (modulo some worries about the concept of mesa-optimisation, which I’ll save for another time).
I missed this disclaimer, sorry. So that assuages some of my concerns about balancing types of work. I’m still not sure what intuitions or arguments underlie your optimism about formal work, though. I assume that this would be fairly time-consuming to spell out in detail—but given that the core point of this post is to encourage such work, it seems worth at least gesturing towards those intuitions, so that it’s easier to tell where any disagreement lies.
To me, the post as written seems like enough to spell out my optimism… there multiple directions for formal work which seem under-explored to me. Well, I suppose I didn’t focus on explaining why things seem under-explored. Hopefully the writeup-to-come will make that clear.