It seems like this won’t happen with the value learning method that seems most natural to me (and consistent with IRL/CIRL): have the true utility function, definition of chocolate, etc be “historical” facts that are not in the AI’s future. In this case, there is no incentive to manipulate the definition of chocolate, since according to the AI’s model, this definition has already been decided.
So I’m curious about what model you’re using; it seems like in your model, it is natural to place the definition of chocolate in the AI’s future.
have the true utility function, definition of chocolate, etc be “historical” facts that are not in the AI’s future.
The whole point of stratification (which is a kind of counterfactual reasoning) is to achieve this. Most value learning suggestions that I’ve seen do not.
I think the other natural approach is to simply make decisions based on the current estimated preferences, but to learn instrumental preferences of the user (including desire for the agent to learn more), as described here. Of course this also doesn’t have the problem from the OP.
Yeah, this seems like the most natural way to deal with things like “chocolate” that aren’t yet well-defined. In this case, the instrumental preferences themselves will be treated as historical facts (it’s assumed that they’re already well-defined enough to learn).
It seems like this won’t happen with the value learning method that seems most natural to me (and consistent with IRL/CIRL): have the true utility function, definition of chocolate, etc be “historical” facts that are not in the AI’s future. In this case, there is no incentive to manipulate the definition of chocolate, since according to the AI’s model, this definition has already been decided.
So I’m curious about what model you’re using; it seems like in your model, it is natural to place the definition of chocolate in the AI’s future.
The whole point of stratification (which is a kind of counterfactual reasoning) is to achieve this. Most value learning suggestions that I’ve seen do not.
What are you thinking of here? Could you point to an example?
I think the other natural approach is to simply make decisions based on the current estimated preferences, but to learn instrumental preferences of the user (including desire for the agent to learn more), as described here. Of course this also doesn’t have the problem from the OP.
Yeah, this seems like the most natural way to deal with things like “chocolate” that aren’t yet well-defined. In this case, the instrumental preferences themselves will be treated as historical facts (it’s assumed that they’re already well-defined enough to learn).