The Oracle AI, realizing this, breaks out of its box and carefully destroys Florida in the fashion most closely resembling a hurricane that it can manage.
Seems like “minimize impact” is being applied at the wrong granularity, if a large deliberate impact is required to cancel out a large incidental one. If we break open the “utility-function maximizing agent” black box, and apply the minimum-impact rule to subgoals instead of actions, it might work better. (This does, however, require an internal architecture that supports a coherent notion of “subgoal”, and maintains it in spite of suboptimality through self modifications—both large cans of worms.)
Seems like “minimize impact” is being applied at the wrong granularity, if a large deliberate impact is required to cancel out a large incidental one. If we break open the “utility-function maximizing agent” black box, and apply the minimum-impact rule to subgoals instead of actions, it might work better. (This does, however, require an internal architecture that supports a coherent notion of “subgoal”, and maintains it in spite of suboptimality through self modifications—both large cans of worms.)
What “minimum impact rule”? How is “impact” computed so that applying it to “subgoals” changes anything?