Suppose an agent has this altruistic empowerment objective, and the problem of getting an objectiveI(humanactions,state) into the agent has been solved. I think this formalization of empowerment is fatally flawed.
Wouldn’t it be maximized by forcing the human in front of a box that encrypts its actions and uses the resulting stream to determine the fate of the universe? Then the human would be maximally “in control” of the universe but unlikely to create a universe that’s good by human preferences.
I think this reflects two problems:
Most injective functions from human actions to world-states are not “human decides the future based on its values”
The channel capacity is higher with the evil box than where the agent is aligned with human preferences, because the human might prefer to limit its own power. For example, the human could want others to have power, know that power corrupts its values, want itself to be unable to destroy the sun with a typo, or have constructed the optimal hedonium farms and not want to change them.
Suppose an agent has this altruistic empowerment objective, and the problem of getting an objectiveI(human actions,state) into the agent has been solved. I think this formalization of empowerment is fatally flawed.
Wouldn’t it be maximized by forcing the human in front of a box that encrypts its actions and uses the resulting stream to determine the fate of the universe? Then the human would be maximally “in control” of the universe but unlikely to create a universe that’s good by human preferences.
I think this reflects two problems:
Most injective functions from human actions to world-states are not “human decides the future based on its values”
The channel capacity is higher with the evil box than where the agent is aligned with human preferences, because the human might prefer to limit its own power. For example, the human could want others to have power, know that power corrupts its values, want itself to be unable to destroy the sun with a typo, or have constructed the optimal hedonium farms and not want to change them.