I agree that it seems possible. I have doubts that predicting the results of editing weights is a more compute-efficient way of causing a model to exhibit the desired behavior than giving it the obvious tools and using fine-tuning / RL to make it able to use those tools though, or alternatively just don’t the RL/finetune directly. That’s basically the heart of how I interpret the bitter lesson—it’s not that you can’t find more efficient ways to do what DL can do, it’s that when you have a task that humans can do and computers can’t, the approach of “introspect and think really hard about how to approach task the right way” is outperformed by the approach of “lol more layers go brrrrr”.
I agree that it seems possible. I have doubts that predicting the results of editing weights is a more compute-efficient way of causing a model to exhibit the desired behavior than giving it the obvious tools and using fine-tuning / RL to make it able to use those tools though, or alternatively just don’t the RL/finetune directly. That’s basically the heart of how I interpret the bitter lesson—it’s not that you can’t find more efficient ways to do what DL can do, it’s that when you have a task that humans can do and computers can’t, the approach of “introspect and think really hard about how to approach task the right way” is outperformed by the approach of “lol more layers go brrrrr”.