Humans can and do change their goals—e.g. religious conversions.
However, I expect to see less of that in more advanced agents.
If we build an AI to perform some task, we will want it to do what we tell it—not decide to go off and do something else.
An AI that forgets what it was built to do is normally broken. We could build such systems—but why would we want to?
As Omohundro says: expected utility maximisers can be expected to back-up and defend their goals. Changing your goals is normally a serious hit to future utility, from the current perspective. Something clearly to be avoided at all costs.
FWIW, Omohundro claims his results are pretty general—and I tend to agree with him. I don’t see the use of an economic framework as a problem—microeconomics itself is pretty general and broadly applicable.
Re: AIs not “wanting” to change their goals
Humans can and do change their goals—e.g. religious conversions.
However, I expect to see less of that in more advanced agents.
If we build an AI to perform some task, we will want it to do what we tell it—not decide to go off and do something else.
An AI that forgets what it was built to do is normally broken. We could build such systems—but why would we want to?
As Omohundro says: expected utility maximisers can be expected to back-up and defend their goals. Changing your goals is normally a serious hit to future utility, from the current perspective. Something clearly to be avoided at all costs.
FWIW, Omohundro claims his results are pretty general—and I tend to agree with him. I don’t see the use of an economic framework as a problem—microeconomics itself is pretty general and broadly applicable.