I wonder if a similar technique could form the foundation for a fully general solution to the alignment problem. Like mathematically speaking all this technique needs is a vector-to-vector function, and it’s not just layer-to-layer relationships that can be understood as a vector-valued function; the world as a function of the policy is also vector-valued.
I.e. rather than running a search to maximize some utility function, a model-based agent could run a search for small changes in policy that have a large impact on the world. If one can then taxonomize, constrain and select between these impacts, one might be able to get a highly controllable AI.
Obviously there’s some difficulties here because the activations are easier to search over since we have an exact way to calculate them. But that’s a capabilities question rather than an alignment question.
I wonder if a similar technique could form the foundation for a fully general solution to the alignment problem. Like mathematically speaking all this technique needs is a vector-to-vector function, and it’s not just layer-to-layer relationships that can be understood as a vector-valued function; the world as a function of the policy is also vector-valued.
I.e. rather than running a search to maximize some utility function, a model-based agent could run a search for small changes in policy that have a large impact on the world. If one can then taxonomize, constrain and select between these impacts, one might be able to get a highly controllable AI.
Obviously there’s some difficulties here because the activations are easier to search over since we have an exact way to calculate them. But that’s a capabilities question rather than an alignment question.