I’m skeptical of this approach. Mostly this is because I’m generally skeptical that an intelligent agent will consist of a separate “planning” part and “reward” part. However, if that were true, then I’d think that this approach could plausibly give us some additional alignment, but can’t solve the entire problem of inner alignment. Specifically, the reward function encodes a _huge_ amount of information: it specifies the optimal behavior in all possible situations you could be in. The “intelligent” part of the net is only ever going to get a subset of this information from the reward function, and so its plans can never be perfectly optimized for that reward function, but instead could be compatible with any reward function that would provide the same information on the “queries” that the intelligent part has produced.
For a slightly-more-concrete example, for any “normal” utility function U, there is a utility function U’ that is “like U, but also the best outcomes are ones in which you hack the memory so that the ‘reward’ variable is set to infinity”. To me, wireheading is possible because the “intelligent” part doesn’t get enough information about U to distinguish U from U’, and so its plans could very well be optimized for U’ instead of U.
This is rather abstract / complex so I’d be interested in suggestions for how to make it more understandable.
My opinion (also going in the newsletter):
This is rather abstract / complex so I’d be interested in suggestions for how to make it more understandable.