Good point. Here’s the raw beginnings of a response:
The idea here would be to resolve such questions “democratically” in some sense. I’m intentionally leaving unspecified what I mean by that because I don’t want to give the impression that I think I could ever tie up all the loose ends with this proposal. In other words, this is a toy example to suggest that there’s useful space to explore in between “fully utilitarian agents” and “non-agents”, that agents with weakly-ordered and/or intransitive preferences may in some senses be superior to fully-utilitarian ones.
I realize that “democratic” answers to the issue you raise will tend to be susceptible to the “Omelas problem” (a majority that gets small benefits by imposing large costs on a minority) and/or the repugnant conclusion (“cram the world with people until barely-over-half of their lives are barely-better-than-death”). Thus, I do not think that “majority rules” should actually be a foundational principle. But I do think that when you encounter intransitivity in collective preferences, it may in some cases be better to live with that than to try to subtract it out by converting everything into comparable-and-summable utility functions.
The costs [of consistency] seem small to me, because consistency requires nothing more than having an ordering on possible worlds. For example, if some possible world seems ok to you, you can put it at the top of the ordering. So assuming infinite power, any ok outcome that can be achieved by any other system can be achieved by a consistent system.
With that in mind, can you try spelling out in what sense an inconsistent system could be better?
Consistency is the opposite of humility. Instead of saying “sometimes, I don’t and can’t know”, it says “I will definitively answer any question (of the correct form)”.
Let’s assume that there is some consistent utility function that we’re using as a basis for comparison. This could be the “correct” utility function (eg, God’s); it could be a given individual’s extrapolated consistent utility; or it could be some well-defined function of many people’s utility.
So, given that we’ve assumed that this function exists, obviously if there’s a quasi-omnipotent agent rationally maximizing it, it will be maximized. This outcome will be at least as good as if the agent is “humble”, with a weakly-ordered objective function; and, in many cases, it will be better. So, you’re right, under this metric, the best utility function is equal-or-better to any humble objective.
But if you get the utility function wrong, it could be much worse than a humble objective. For instance, consider adding some small amount of Gaussian noise to the utility. The probability that the “optimized” outcome will have a utility arbitrarily close to the lower bound could, depending on various things, be arbitrarily high; while I think you can argue that a “humble” deus ex machina, by allowing other agents to have more power to choose between world-states over which the machina has no strict preference, would be less likely to end up in such an arbitrarily bad “Goodhart” outcome.
This response is a bit sketchy, but does it answer your question?
Good point. Here’s the raw beginnings of a response:
The idea here would be to resolve such questions “democratically” in some sense. I’m intentionally leaving unspecified what I mean by that because I don’t want to give the impression that I think I could ever tie up all the loose ends with this proposal. In other words, this is a toy example to suggest that there’s useful space to explore in between “fully utilitarian agents” and “non-agents”, that agents with weakly-ordered and/or intransitive preferences may in some senses be superior to fully-utilitarian ones.
I realize that “democratic” answers to the issue you raise will tend to be susceptible to the “Omelas problem” (a majority that gets small benefits by imposing large costs on a minority) and/or the repugnant conclusion (“cram the world with people until barely-over-half of their lives are barely-better-than-death”). Thus, I do not think that “majority rules” should actually be a foundational principle. But I do think that when you encounter intransitivity in collective preferences, it may in some cases be better to live with that than to try to subtract it out by converting everything into comparable-and-summable utility functions.
Sometime ago I made this argument to Said Achmiz:
With that in mind, can you try spelling out in what sense an inconsistent system could be better?
Consistency is the opposite of humility. Instead of saying “sometimes, I don’t and can’t know”, it says “I will definitively answer any question (of the correct form)”.
Let’s assume that there is some consistent utility function that we’re using as a basis for comparison. This could be the “correct” utility function (eg, God’s); it could be a given individual’s extrapolated consistent utility; or it could be some well-defined function of many people’s utility.
So, given that we’ve assumed that this function exists, obviously if there’s a quasi-omnipotent agent rationally maximizing it, it will be maximized. This outcome will be at least as good as if the agent is “humble”, with a weakly-ordered objective function; and, in many cases, it will be better. So, you’re right, under this metric, the best utility function is equal-or-better to any humble objective.
But if you get the utility function wrong, it could be much worse than a humble objective. For instance, consider adding some small amount of Gaussian noise to the utility. The probability that the “optimized” outcome will have a utility arbitrarily close to the lower bound could, depending on various things, be arbitrarily high; while I think you can argue that a “humble” deus ex machina, by allowing other agents to have more power to choose between world-states over which the machina has no strict preference, would be less likely to end up in such an arbitrarily bad “Goodhart” outcome.
This response is a bit sketchy, but does it answer your question?
It makes sense to value other agents having power, but are you sure that value can’t be encoded consistently?