Disagree that it’s obvious, depends a lot on how efficient (large-scale) RL (post-training) is at very significantly changing model internals, rather than just ‘wrapping it around’, making the model more reliable, etc. In the past, post-training (including RL) has been really bad at this.
If that’s true, perhaps the performance penalty for pinning/freezing weights in the ‘internals’, prior to the post-training, would be low. That could ease interpretability a lot, if you didn’t need to worry so much about those internals which weren’t affected by post-training?
Yes. Also, if the LMs after pretraining as Simulators model is right (I think it is) it should also help a lot with safety in general, because the simulator should be quite malleable, even if some of the simulacra might be malign. As long as you can elicit the malign simulacra, you can also apply interp to them or do things in the style of Interpreting the Learning of Deceit for post-training. This chould also help a lot with e.g. coup probes and other similar probes for monitoring.
Disagree that it’s obvious, depends a lot on how efficient (large-scale) RL (post-training) is at very significantly changing model internals, rather than just ‘wrapping it around’, making the model more reliable, etc. In the past, post-training (including RL) has been really bad at this.
If that’s true, perhaps the performance penalty for pinning/freezing weights in the ‘internals’, prior to the post-training, would be low. That could ease interpretability a lot, if you didn’t need to worry so much about those internals which weren’t affected by post-training?
Yes. Also, if the LMs after pretraining as Simulators model is right (I think it is) it should also help a lot with safety in general, because the simulator should be quite malleable, even if some of the simulacra might be malign. As long as you can elicit the malign simulacra, you can also apply interp to them or do things in the style of Interpreting the Learning of Deceit for post-training. This chould also help a lot with e.g. coup probes and other similar probes for monitoring.