The argument in this post is just that it might help prevent mesa-optimization from happening at all, not that it would make it more aligned. The next post will be about how to align mesa-optimizers.
Is it a requirement of mesa-optimization that the optimization algorithm must be learned? I would have expected that it would only be a requirement that the objective be learned. Are there considerations that apply to learned optimization algorithms that don’t apply to hardcoded optimization algorithms?
The main benefit I see of hardcoding optimisation is that, assuming the system’s pieces learn as intended (without any mesa-optimisation happening in addition to the hardcoded optimisation) you get more access and control as a programmer over what the learned objective actually is. You could attempt to regress the learned objective directly to a goal you want, or attempt to enforce a certain form on it, etc. When the optimisation itself is learned*, the optimiser is more opaque, and you have fewer ways to affect what goal is learned: which weights of your enormous LSTM-based mesa-optimiser represent the objective?
This doesn’t solve the problem completely (you might still learn an objective that is very incorrect off-distribution, etc.), but could offer more control and insight into the system to the programmer.
*Of course, you can have learned optimisation where you keep track of the objective which is being optimised (like in Learning to Learn by Gradient Descent), but I’d class that more under hard-coded optimisation for the purposes of this discussion. Here I mean the kind of learned optimisation that happens where you’re not building the architecture explicitly around optimising or learning to optimise.
The argument in this post is just that it might help prevent mesa-optimization from happening at all, not that it would make it more aligned. The next post will be about how to align mesa-optimizers.
Is it a requirement of mesa-optimization that the optimization algorithm must be learned? I would have expected that it would only be a requirement that the objective be learned. Are there considerations that apply to learned optimization algorithms that don’t apply to hardcoded optimization algorithms?
The main benefit I see of hardcoding optimisation is that, assuming the system’s pieces learn as intended (without any mesa-optimisation happening in addition to the hardcoded optimisation) you get more access and control as a programmer over what the learned objective actually is. You could attempt to regress the learned objective directly to a goal you want, or attempt to enforce a certain form on it, etc. When the optimisation itself is learned*, the optimiser is more opaque, and you have fewer ways to affect what goal is learned: which weights of your enormous LSTM-based mesa-optimiser represent the objective?
This doesn’t solve the problem completely (you might still learn an objective that is very incorrect off-distribution, etc.), but could offer more control and insight into the system to the programmer.
*Of course, you can have learned optimisation where you keep track of the objective which is being optimised (like in Learning to Learn by Gradient Descent), but I’d class that more under hard-coded optimisation for the purposes of this discussion. Here I mean the kind of learned optimisation that happens where you’re not building the architecture explicitly around optimising or learning to optimise.
That makes sense, thanks.