I’d summarise the results of your post, just to check my understanding and to allow myself to be corrected.
The utility function of a value-learning agent will change as it encounters evidence, and depending on its actions.
This is a good ideal yet we should note two traps with this approach:
1) if an agent uses its current utility function to evaluate future actions, then it will avoid learning a different utility function—because a change in utility function will cause the agent to behave poorly based on its current standard (i.e. goal preservation from Omohundro’s AI Drives)
2) So suppose that the agent uses its future utility function to evaluate its future decisions. Then, in a bad case, it will choose to learn the utility function that is the easiest. The bad case is that the agent thinks that certain actions will change its utility function in a predictable way. i.e. if E[p(U=u_1 | a)] != p(U=u_1). So we must enforce this axiom of probability E[p(U=u_1 | a)] == p(U=u_1) to prevent the agent from assigning itself an easy utility function.
Nothing new here, just carrying on explaining my understanding in case it helps others:
Following on from (2): in the simple case where the AI can ask the advisor or not, we want the expected utility after asking to also be used to evaluate the case where the AI doesn’t ask. i.e.
E[p(C=u1 | A=”don’t ask”)] := E[p(C=u_1 | A=”ask”] (:= is assignment; C is the correct utility function)
So we’ll renormalise the probability of each utility function in the “don’t ask” scenario.
A more complex case arises where there multiple actions cause changes in the utility function, e.g. if there are a bunch of different advisors. In these more complex cases, it’s not so useful to think about a direction of assignment. The more useful model for what’s going on is that the agent must have a distribution over C that is updated when it gets a different model of what the advisors will say.
Basically, requiring the agent to update its distribution over utility functions in a way that obeys the axioms of probability will prevent the agent from sliding toward the utility functions that are easiest to fulfil.
I’d summarise the results of your post, just to check my understanding and to allow myself to be corrected.
The utility function of a value-learning agent will change as it encounters evidence, and depending on its actions.
This is a good ideal yet we should note two traps with this approach:
1) if an agent uses its current utility function to evaluate future actions, then it will avoid learning a different utility function—because a change in utility function will cause the agent to behave poorly based on its current standard (i.e. goal preservation from Omohundro’s AI Drives)
2) So suppose that the agent uses its future utility function to evaluate its future decisions. Then, in a bad case, it will choose to learn the utility function that is the easiest. The bad case is that the agent thinks that certain actions will change its utility function in a predictable way. i.e. if E[p(U=u_1 | a)] != p(U=u_1). So we must enforce this axiom of probability E[p(U=u_1 | a)] == p(U=u_1) to prevent the agent from assigning itself an easy utility function.
Nothing new here, just carrying on explaining my understanding in case it helps others:
Following on from (2): in the simple case where the AI can ask the advisor or not, we want the expected utility after asking to also be used to evaluate the case where the AI doesn’t ask. i.e.
E[p(C=u1 | A=”don’t ask”)] := E[p(C=u_1 | A=”ask”] (:= is assignment; C is the correct utility function)
So we’ll renormalise the probability of each utility function in the “don’t ask” scenario.
A more complex case arises where there multiple actions cause changes in the utility function, e.g. if there are a bunch of different advisors. In these more complex cases, it’s not so useful to think about a direction of assignment. The more useful model for what’s going on is that the agent must have a distribution over C that is updated when it gets a different model of what the advisors will say.
Basically, requiring the agent to update its distribution over utility functions in a way that obeys the axioms of probability will prevent the agent from sliding toward the utility functions that are easiest to fulfil.