I think there’s another important reason: a powerful agent might sometimes need to “burn” its own utility to get a bargaining edge, and we don’t have a full theory for that yet.
What does “burn it’s own utility” mean? Spending resources in bargains (or spending resources to get better at bargaining) makes perfect sense, if that’s the best path to utility. But that’s still optimizing and maximizing utility, not burning it for some non-utility-related purpose.
I think there’s another important reason: a powerful agent might sometimes need to “burn” its own utility to get a bargaining edge, and we don’t have a full theory for that yet.
What does “burn it’s own utility” mean? Spending resources in bargains (or spending resources to get better at bargaining) makes perfect sense, if that’s the best path to utility. But that’s still optimizing and maximizing utility, not burning it for some non-utility-related purpose.