Yes, agreed, building a system that can reliably predict the consequences of its actions… for example, that can recognize that hypothetically making X change to its utility function results in its children hypothetically not being fed… is a hard engineering problem.
That said, calling something an AGI with a utility function at all, let alone a superhuman one, seems to presuppose that this problem has been solved. If it can’t do that, we have bigger problems than the stability of its utility function.
(If my actions aren’t conditioned on reliable judgments about likely consequences in the first place, you have no grounds for taking an intentional stance with respect to me at all… knowing what I want does not let you predict what I’ll do. I’m not sure on what grounds you’re calling me intelligent, at that point.)
Distinct from that is the system being built such that, before making a change to its utility function, it considers the likely consequences of that change, and such that, if it considers the likely consequences bad ones, it doesn’t make the change.
But that part seems relatively simple by comparison.
Yes, agreed, building a system that can reliably predict the consequences of its actions… for example, that can recognize that hypothetically making X change to its utility function results in its children hypothetically not being fed… is a hard engineering problem.
That said, calling something an AGI with a utility function at all, let alone a superhuman one, seems to presuppose that this problem has been solved. If it can’t do that, we have bigger problems than the stability of its utility function.
(If my actions aren’t conditioned on reliable judgments about likely consequences in the first place, you have no grounds for taking an intentional stance with respect to me at all… knowing what I want does not let you predict what I’ll do. I’m not sure on what grounds you’re calling me intelligent, at that point.)
Distinct from that is the system being built such that, before making a change to its utility function, it considers the likely consequences of that change, and such that, if it considers the likely consequences bad ones, it doesn’t make the change.
But that part seems relatively simple by comparison.