1) A single utility function? What would it mean to have multiple?
It would mean there is a difficult-to-characterize ecosystem of competing/cooperating agents. Does this sort of cognitive architecture seem familiar at all? :)
My general problem with “utilitarianism” is that it’s sort of like Douglas Adams’ “42.” An answer of the wrong type to a difficult question. Of course we should maximize, that is a useful ingredient of the answer, but is not the only (or the most interesting) ingredient.
My general problem with “utilitarianism” is that it’s sort of like Douglas Adams’ “42.” An answer of the wrong type to a difficult question. Of course we should maximize, that is a useful ingredient of the answer, but is not the only (or the most interesting) ingredient.
Taking off from the end of that point, I might add (but I think this was probably part of your total point, here, about “the most interesting” ingredient) that people sometimes forget that utilitarianism is not a theory itself about what is normatively desirable, and least not much of one.
For Bentham-style “greatest good for the greatest number” to have any meaning, it has to be supplemented with a view of what property, state of being, action type, etc, counts as a “good” thing, to begin with.
Once this is defined, we can then go on to maximize that—seeking to achieve the most of that, for the most people (or relevant entities.)
But greatest good for the greatest number means nothing until we figure out a theory of normativity, or meta-normativity that can be instantiated across specific, varying situations and scenarios.
IF the “good” is maximizing simple total body weight, then adding up the body weight of all people in possible world A, vs in possible world B, etc, will allow us a utilitarian decision among possible worlds.
IF the “good” were fitness, or mental healty, or educational achievement… we use the same calculus, but the target property is obviously different.
Utilitarianism is sometimes a person’s default answer, until you remind them that this is not an answer at all about what is good. It is just an implementation standard for how that good is to be devided up. Kind of a trivial point, I guess, but worth reminding ourselves from time to time that utilitarianism is not a theory of what is actually good, but how that might be distributed, if that admits of scarcity.
In the same way people’s minds do. They are inconsistent but will notice the setup very quickly and stop. (I don’t find Dutch book arguments very convincing, really).
(a) In what settings do you want an architecture like that, and
(b) Ethics dictate we don’t just want to replace entities for the sake of efficiency even if they disagree. This leads to KILL ALL HUMANS. So, we might get an architecture like that due to how history played out. And then it’s just a brute fact.
I am guessing (a) has to do with “robustness” (I am not prepared to mathematise what I mean yet, but I am thinking about it).
People that think about UDT/blackmail are thinking precisely about how to win in settings I am talking about.
Pick a side of this fence. Will AI resist running-in-circles trivially, or is its running in circles all that’s saving us from KILL ALL HUMANS objectives like you say in part b?
It would mean there is a difficult-to-characterize ecosystem of competing/cooperating agents. Does this sort of cognitive architecture seem familiar at all? :)
My general problem with “utilitarianism” is that it’s sort of like Douglas Adams’ “42.” An answer of the wrong type to a difficult question. Of course we should maximize, that is a useful ingredient of the answer, but is not the only (or the most interesting) ingredient.
Taking off from the end of that point, I might add (but I think this was probably part of your total point, here, about “the most interesting” ingredient) that people sometimes forget that utilitarianism is not a theory itself about what is normatively desirable, and least not much of one. For Bentham-style “greatest good for the greatest number” to have any meaning, it has to be supplemented with a view of what property, state of being, action type, etc, counts as a “good” thing, to begin with. Once this is defined, we can then go on to maximize that—seeking to achieve the most of that, for the most people (or relevant entities.)
But greatest good for the greatest number means nothing until we figure out a theory of normativity, or meta-normativity that can be instantiated across specific, varying situations and scenarios.
IF the “good” is maximizing simple total body weight, then adding up the body weight of all people in possible world A, vs in possible world B, etc, will allow us a utilitarian decision among possible worlds.
IF the “good” were fitness, or mental healty, or educational achievement… we use the same calculus, but the target property is obviously different.
Utilitarianism is sometimes a person’s default answer, until you remind them that this is not an answer at all about what is good. It is just an implementation standard for how that good is to be devided up. Kind of a trivial point, I guess, but worth reminding ourselves from time to time that utilitarianism is not a theory of what is actually good, but how that might be distributed, if that admits of scarcity.
How would they interact such that it’s not simply adding over them, and they don’t end up being predictably Dutch-bookable?
In the same way people’s minds do. They are inconsistent but will notice the setup very quickly and stop. (I don’t find Dutch book arguments very convincing, really).
Seems like a layer of inefficiency to have to resist temptation to run in circles rather than just want to go uphill.
There are two issues:
(a) In what settings do you want an architecture like that, and
(b) Ethics dictate we don’t just want to replace entities for the sake of efficiency even if they disagree. This leads to KILL ALL HUMANS. So, we might get an architecture like that due to how history played out. And then it’s just a brute fact.
I am guessing (a) has to do with “robustness” (I am not prepared to mathematise what I mean yet, but I am thinking about it).
People that think about UDT/blackmail are thinking precisely about how to win in settings I am talking about.
Pick a side of this fence. Will AI resist running-in-circles trivially, or is its running in circles all that’s saving us from KILL ALL HUMANS objectives like you say in part b?
If the latter, we are so utterly screwed.