This post has a lot of implications. But I feel like one of them might be underrated: Utilitarianism is wrong? I’m not sure if I’m reading too much into it, but I feel like implicitly this post demolishes utilitarianism as usually understood.
We might think of utilitarianism as relying on a few simple intuitive moral arguments like “the point of morality is to make people better off”… as well as relying on the assumption that morality is path-independent. This is the key assumption that this post brings to life. And I feel like if you challenged people on whether morality is path-independent, then the assumption would be “obviously not”.
Whenever I’ve seen people people discussing whether utilitarianism is correct, I’ve been linking them to this post. So I would be curious if you agree with the assessment that this is a strong challenge to utilitarianism.
I think it’s also interesting, because it suggests that we should think of morality in terms of subagents. Obviously there’s a few different ways of going about that, but one thing I’ve been playing with recently is to think of morality as a sort of renormalization of individual preferences in large-scale groups. That is, you can take as input the preferences of all individual people or something, and then you can try and derive a small set of subagents that approximately represent their preference ordering.
This post is describing a style of agent that can exist. Whether or not they are coherent is arguable. You could say “sure, that type of agent is coherent. But I happen to be an agent with totally ordered preferences. ” Or you could say, “I as a whole act according to a committee of utilitarianism and personal hedonism”.
Utilitarianism, on some abstract level, can only ever be a fact about the sort of agent you (or other people) are or want to be. There are arguments of “if you aren’t utilitarian, then …” but some agents are fine with ”...”. It is a specific human trait to see this argument, decide that ”...” is bad and act utilitarian.
I think this is probably right, but I also think the sequences generally made a solid case for that from a few different angles, so it was already pretty established on LW by the time this post came out.
Here are some ones that feel relevant. (This was me searching for posts where Eliezer is criticizing having “one great moral principle” that you could give an AI, which is a major theme. One major point of the sequences is that morality is quite complicated)
The arguments here sound like “morality is actually complex, and you shouldn’t oversimplify it”. But utilitarianism is pretty complex, in the relevant sense, so this kind of fails to land for me.
Hmm. What do you mean by “complex in the relevant sense?”. The two obvious things you might call complex are “the part where you figure out to estimate a person’s utility in the first place, and aggregate that across people”, and “the part where in practice you need all kinds of complex rules of thumb or brute force evaluation of second-order consequences.”
The former seems legit “hard”, I guess, but sorta seems like a one-shot upfront scientific/philosophical problem that isn’t that hard. (I realize it’s, like, unsolved after decades of relevant work, but, idk, still, doesn’t seem fundamentally confusing to me?). Is this what you meant by “relevant sense?”
The second seems complex in some sense, but seems sorta like how AlphaGo can figure out complex Go strategy given the simple task of “play go against yourself a bunch”. And it seemed like the sequences were arguing against this sort of thing being that easy.
Also I guess depends what sort of utilitarianism you mean, but note:
I dunno if there’s a “not for the sake of aggregate preference utility (alone)”, but I felt like the sequences were arguing (albeit indirectly) that this was still more complex than you (generic you) were probably imagining.
I mean the former: like, whatever “utility” is is not a simple thing to define in terms of things we have a handle on (“pleasurable mental states” does not count as a simple definition), and even if you allow yourself access to standard language about mental states I don’t think it’s so easy (e.g. there are a bunch of different sorts of mental states that might fall under the broad umbrella of “pleasure”).
I do agree that “not for the sake of happiness alone” argues against utilitarianism.
This post has a lot of implications. But I feel like one of them might be underrated: Utilitarianism is wrong? I’m not sure if I’m reading too much into it, but I feel like implicitly this post demolishes utilitarianism as usually understood.
We might think of utilitarianism as relying on a few simple intuitive moral arguments like “the point of morality is to make people better off”… as well as relying on the assumption that morality is path-independent. This is the key assumption that this post brings to life. And I feel like if you challenged people on whether morality is path-independent, then the assumption would be “obviously not”.
Whenever I’ve seen people people discussing whether utilitarianism is correct, I’ve been linking them to this post. So I would be curious if you agree with the assessment that this is a strong challenge to utilitarianism.
I think it’s also interesting, because it suggests that we should think of morality in terms of subagents. Obviously there’s a few different ways of going about that, but one thing I’ve been playing with recently is to think of morality as a sort of renormalization of individual preferences in large-scale groups. That is, you can take as input the preferences of all individual people or something, and then you can try and derive a small set of subagents that approximately represent their preference ordering.
This post is describing a style of agent that can exist. Whether or not they are coherent is arguable. You could say “sure, that type of agent is coherent. But I happen to be an agent with totally ordered preferences. ” Or you could say, “I as a whole act according to a committee of utilitarianism and personal hedonism”.
Utilitarianism, on some abstract level, can only ever be a fact about the sort of agent you (or other people) are or want to be. There are arguments of “if you aren’t utilitarian, then …” but some agents are fine with ”...”. It is a specific human trait to see this argument, decide that ”...” is bad and act utilitarian.
I think this is probably right, but I also think the sequences generally made a solid case for that from a few different angles, so it was already pretty established on LW by the time this post came out.
Huh, I don’t remember that. What’s an example post that makes such a case?
Here are some ones that feel relevant. (This was me searching for posts where Eliezer is criticizing having “one great moral principle” that you could give an AI, which is a major theme. One major point of the sequences is that morality is quite complicated)
https://www.lesswrong.com/s/fqh9TLuoquxpducDb/p/NnohDYHNnKDtbiMyp
https://www.lesswrong.com/posts/4ARaTpNX62uaL86j6/the-hidden-complexity-of-wishes
https://www.lesswrong.com/posts/RFnkagDaJSBLDXEHs/heading-toward-morality
The arguments here sound like “morality is actually complex, and you shouldn’t oversimplify it”. But utilitarianism is pretty complex, in the relevant sense, so this kind of fails to land for me.
Hmm. What do you mean by “complex in the relevant sense?”. The two obvious things you might call complex are “the part where you figure out to estimate a person’s utility in the first place, and aggregate that across people”, and “the part where in practice you need all kinds of complex rules of thumb or brute force evaluation of second-order consequences.”
The former seems legit “hard”, I guess, but sorta seems like a one-shot upfront scientific/philosophical problem that isn’t that hard. (I realize it’s, like, unsolved after decades of relevant work, but, idk, still, doesn’t seem fundamentally confusing to me?). Is this what you meant by “relevant sense?”
The second seems complex in some sense, but seems sorta like how AlphaGo can figure out complex Go strategy given the simple task of “play go against yourself a bunch”. And it seemed like the sequences were arguing against this sort of thing being that easy.
Also I guess depends what sort of utilitarianism you mean, but note:
https://www.lesswrong.com/posts/synsRtBKDeAFuo7e3/not-for-the-sake-of-happiness-alone
I dunno if there’s a “not for the sake of aggregate preference utility (alone)”, but I felt like the sequences were arguing (albeit indirectly) that this was still more complex than you (generic you) were probably imagining.
I mean the former: like, whatever “utility” is is not a simple thing to define in terms of things we have a handle on (“pleasurable mental states” does not count as a simple definition), and even if you allow yourself access to standard language about mental states I don’t think it’s so easy (e.g. there are a bunch of different sorts of mental states that might fall under the broad umbrella of “pleasure”).
I do agree that “not for the sake of happiness alone” argues against utilitarianism.
In my ontology, there is no such thing as morality being right or wrong in any sense, so I don’t understand why this is an implication of this post.
Well I guess you should stay out of moral discussions then.