Thanks for the well-thought out comment. It helps me think through the issue of suffering a lot more.
~
If you took a human and replaced its brain with a search algorithm which found the motor output solutions which maximized the original human’s preferences, I’d consider this search algorithm to fit the definition of a person (though not necessarily the same person). [...] Violation of this being’s preferences may or may not be called “suffering” depending on how you define “suffering”...but either way, I think this being’s preferences are just as morally relevant as a humans. [...]
The question is, regardless of the label, what is the underlying morally relevant feature?
I think this is a good thought experiment and it does push me more toward preference satisfaction theories of well-being, which I have long been sympathetic to. I still don’t know much myself about what I view as suffering. I’d like to read and think more on the issue—I have bookmarked some of Brian Tomasik’s essays to read (he’s become more preference-focused recently) as well as an interview with Peter Singer where he explains why he’s abandoned preference utilitarianism for something else. So I’m not sure I can answer your question yet.
There are interesting problems with desires, such as formalizing it (what is a desire and what makes a desire stronger or weaker, etc.), population ethics (do we care about creating new beings with preferences, etc.) and others that we would have to deal with as well.
~
Paperclippers, molecular modelers, search engines, seek to maximize a simple set of preferences (number of paperclips, best fit model, best search). They have “preferences”, but not morally relevant ones. A human (or, hopefully one day a friendly AI) seeks to fulfill an extremely complex set of preference...as does a fish. They have preferences which carry moral weight.
So it seems like, to you, an entity’s welfare matters when it has preferences, weighted based on the complexity of those preferences, with a certain zero threshold somewhere (so thermostat preferences don’t count).
I don’t think complexity is the key driver for me, but I can’t tell you what is.
~
I haven’t seen any behavioral evidence of fish doing problem solving, being empathetic towards each other, exhibiting cognitive capacities beyond very basic associative learning & memory, or that sort of thing.
Likewise, I don’t think this is much of a concern for me, and it seems inconsistent with the rest of what you’ve been saying.
Why are problem solving and empathy important? Surely I could imagine a non-empathetic program without the ability to solve most problems, that still has the kind of robust preferences you’ve been talking about.
And what level of empathy and problem solving are you looking for? Notably, fish engage in cleaning symbiosis (which seems to be in the lower-tier of the empathy skill tree) and Wikipedia seems to indicate (though perhaps unreliably) that fish have pretty good learning capabilities.
~
I don’t think so, but I might be wrong...Is risk aversion in the face of uncertainty actually rational in this scenario? Seems to me that there are certain scenarios where risk aversion makes sense (personal finance, for example) and scenarios where it doesn’t (effective altruism, for example) and this decision seems to fall in the latter camp.
an entity’s welfare matters when it has preferences, weighted based on the complexity of those preferences
No, it’s not complexity, but content of the preferences that make a difference. Sorry for mentioning the complexity—i didn’t mean to imply that it was the morally relevant feature.
I’m not yet sure what sort of preferences give an agent morally weighty status...the only thing I’m pretty sure about is that the morally relevant component is contained somewhere within the preferences, with intelligence as a possible mediating or enabling factor.
Here’s one pattern I think I’ve identified:
I belong within reference Class X.
All beings in Reference Class X care about other beings in Reference Class X, when you extrapolate their volition.
When I hear about altruistic mice, it is evidence that the mouse’s extrapolated volition would cause it to care about Class X-being’s preferences to the extent that it can comprehend them. The cross-species altruism of dogs and dolphins and elephants is an especially strong indicator of Class X membership.
On the other hand, the within-colony altruism of bees (basically identical to Reference Class X except it only applies to members of the colony and I do not belong in it), or the swarms and symbiosis of fishes or bacterial gut flora, wouldn’t count...being in Reference Class X is clearly not the factor behind the altruism in those cases.
...which sounds awfully like reciprocal altruism in practice, doesn’t it? Except that, rather than looking at the actual act of reciprocation of altruism, I’d be extrapolating the agent’s preferences for altruism. Perhaps Class X would be better named “Friendly”, in the “Friendly AI” sense—all beings within the class are to some extent Friendly towards each other.
This is at the rough edge of my thinking though—the ideas as just stated are experimental and I don’t have well defined notions about which preferences matter yet.
Edit: Another (very poorly thought out) trend which seems to emerge is that agents which have a certain sort of awareness are entitled to a sort of bodily autonomy … because it seems immoral to sit around torturing insects if one has no instrumental reason to do so. (But is it immoral in the sense that there are a certain number of insects which morally outweigh a a human? Or is it immoral in a virtue ethic-y, “this behavior signals sadism” sort of way?)
My main point is that I’m mildly guessing that it’s probably safe to narrow down the problem to some combination of preference functions and level of awareness. In any case, I’m almost certain that there exist preference functions are sufficient (but maybe not necessary?) to confer moral weight onto an agent...and though there may be other factors unrelated to preference or intelligence that play a role, preference function is the only thing with a concrete definition that I’ve identified so far.
...which sounds awfully like reciprocal altruism in practice, doesn’t it? Except that, rather than looking at the actual act of reciprocation of altruism, I’d be extrapolating the agent’s preferences for altruism. Perhaps Class X would be better named “Friendly”, in the “Friendly AI” sense—all beings within the class are to some extent Friendly towards each other.
Just so I understand you better, how would you compare and contrast this kind of pro-X “kin” altruism with utilitarianism?
Utilitarianism has never made much sense to me except as a handy way to talk about things abstractly when precision isn’t important
...but I suppose X would be a class of agents who consider each other’s preferences when they make utilitarian calculations? I pretty much came up with the pro-X idea less than a month ago, and haven’t thought it through very carefully.
Oh, here’s a good example of where preference utilitarianism fails which illustrates it:
10^100 intelligent people terminally prefer that 1 person is tortured. Preference utilitarianism says “do the torture”. My moral instinct says “no, it’s still wrong, no matter how many people prefer it”.
Perhaps under the pro-X system, the reason we can ignore the preferences of 10^100 people is that the preference which they have expressed lies strictly outside category X and therefore that preference can be ignored?
Whereas, if you have a Friendly Paperclipper (cares about X-agents and paperclips with some weight on each), the Friendly moral values put it within X...which means that we should now be willing to cater to its morally neutral paper-clip preferences as well.
(If this reads sloppy, it’s because my thoughts on the matter currently are sloppy)
So...I guess there’s sort of a taxonomy of moral-good, neutral-selfish, and evil preferences...and part of being good means caring about other people’s selfish preferences? And part of being evil means valuing the violation of other’s preferences? And, good agents can simply ignore evil preferences.
And (under the pro-X system), good agents can also ignore the preferences of agents that aren’t in any way good...which seems like it might not be correct, which is why I say that there might be other factors in addition to pro-X that make an agent worth caring about for my moral instincts, but if they exist I don’t know what they are.
Are you perhaps confusing ‘morally wrong’ with ‘a sucky tradeoff that I would prefer not to be bound by’?
Just because torturing one person sucks, just because we find it abhorrent, does not mean that it isn’t the best outcome in various situations. If your definition of ‘moral’ is “best outcome when all things are considered, even though aspects of it suck a lot and are far from ideal”, then yes, torturing someone can in fact be moral. If your definition of ‘moral’ is “those things which I find reprehensible”, then quite probably you can never find torturing someone to be moral. However, there are scenarios where it may still be necessary, or the best option.
Are you perhaps confusing ‘morally wrong’ with ‘a sucky tradeoff that I would prefer not to be bound by’?
Nope...because ..
quite probably you can never find torturing someone to be moral. However, there are scenarios where it may still be necessary, or the best option.
...because I believe that torturing someone could still instrumentally be the right thing to do on a consequential grounds.
In this scenario, 10^100 people terminally value torturing one person, but I do not care about their preferences, because it is an evil preference.
However, in an alternate scenario, if I had to choose between 10^100 people getting mildly hurt or 1 person getting tortured, I’d choose the one person getting tortured.
In these two scenarios, the preference weights are identical, but in the first scenario the preference of the 10^100 people is evil and therefore irrelevant in my calculations, whereas in the second scenario the needs of 10^100 outweigh the needs of the one.
This is less a discussion about torture, and more a discussion about whose/which preferences matter. Sadistic preferences (involving real harm, not the consensual kink), for example, don’t matter morally—there’s no moral imperative to fulfill those preferences, no “good” done when those preferences are fulfilled and no “evil” resulting from thwarting those preferences.
I think you should temporarily taboo ‘moral’, ‘morality’, and ‘evil’, and simply look at the utility calculations. 10^100 people terminally value something that you ascribe zero or negative value to; therefore, their preferences do not matter to you or will make your universe worse from the standpoint of your utility function.
Which preferences matter? Yours matter to you, and thiers matter to them. There’s no ‘good’ or ‘evil’ in any absolute sense, merely different utility functions that happen to conflict. There’s no utility function which is ‘correct’, except by some arbitrary metric, of which there are many.
Consider another hypothetical utility function: The needs of the 10^100 don’t outweigh the needs of the one, so we let the entire 10^100 suffer when we could eliminate it by inconveniencing one single entity. Neither you nor the 10^100 are happy with this one, but the person about to be tortured may think it’s just fine and dandy...
...I don’t denotatively disagree with anything you’ve said, but I also think you’re sort of missing the point and forgetting the context of the conversation as it was in the preceding comments.
We all have preferences, but we do not always know what our own preferences are. A subset of our preferences (generally those which do not directly reference ourselves) are termed “moral preferences”. The preceding discussion between me and Peter Hurford is an attempt to figure out what our preferences are.
In the above conversation, words like “matter”, “should” and “moral” is understood to mean “the shared preferences of Ishaan, Dentin, and Peter_Hurford which they agree to define as moral”. Since we are all human (and similar in many other ways beyond that), we probably have very similar moral preferences...so any disagreement that arises between us is usually due to one or both of us inaccurately understanding our own preferences.
There’s no ‘good’ or ‘evil’ in any absolute sense
This is technically true, but it’s also often a semantic stopsign which derails discussions of morality. The fact is that the three of us humans have a very similar notion of “good”, and can speak meaningfully about what it is...the implicitly understood background truths of moral nihilism notwithstanding.
It doesn’t do to exclaim “but wait! good and evil are relative!” during every moral discussion...because here, between us three humans, our moral preferences are pretty much in agreement and we’d all be well served by figuring out exactly those preferences are. It’s not like we’re negotiating morality with aliens.
Which preferences matter? Yours matter to you
Precisely...my preferences are all that matter to me, and our preferences are all that matter to us. So if 10^100 sadistic aliens want to torture...so what? We don’t care if they like torture, because we dislike torture and our preferences are all that matter. Who cares about overall utility? “Morality”, for all practical purposes, means shared human morality...or, at least, the shared morality of the humans who are having the discussion.
“Utility” is kind of like “paperclips”...yes, I understand that in the best case scenario it might be possible to create some sort of construct which measures how much “utility” various agent-like objects get from various real world outcomes, but maximizing utility for all agents within this framework is not necessarily my goal...just like maximizing paperclips is not my goal.
For the purposes of this conversation at least. I’ve largely got them taboo’d in general because I find them confusing and full of political connotations; I suspect at least some of that is the problem here as well.
10^100 intelligent people terminally prefer that 1 person is tortured. Preference utilitarianism says “do the torture”. My moral instinct says “no, it’s still wrong, no matter how many people prefer it”.
Yet your moral instinct is perfectly fine with having a justice system that puts innocent people in jail with a greater than 1 in 10^100 error rate.
Thanks for the well-thought out comment. It helps me think through the issue of suffering a lot more.
~
I think this is a good thought experiment and it does push me more toward preference satisfaction theories of well-being, which I have long been sympathetic to. I still don’t know much myself about what I view as suffering. I’d like to read and think more on the issue—I have bookmarked some of Brian Tomasik’s essays to read (he’s become more preference-focused recently) as well as an interview with Peter Singer where he explains why he’s abandoned preference utilitarianism for something else. So I’m not sure I can answer your question yet.
There are interesting problems with desires, such as formalizing it (what is a desire and what makes a desire stronger or weaker, etc.), population ethics (do we care about creating new beings with preferences, etc.) and others that we would have to deal with as well.
~
So it seems like, to you, an entity’s welfare matters when it has preferences, weighted based on the complexity of those preferences, with a certain zero threshold somewhere (so thermostat preferences don’t count).
I don’t think complexity is the key driver for me, but I can’t tell you what is.
~
Likewise, I don’t think this is much of a concern for me, and it seems inconsistent with the rest of what you’ve been saying.
Why are problem solving and empathy important? Surely I could imagine a non-empathetic program without the ability to solve most problems, that still has the kind of robust preferences you’ve been talking about.
And what level of empathy and problem solving are you looking for? Notably, fish engage in cleaning symbiosis (which seems to be in the lower-tier of the empathy skill tree) and Wikipedia seems to indicate (though perhaps unreliably) that fish have pretty good learning capabilities.
~
That makes sense to me.
No, it’s not complexity, but content of the preferences that make a difference. Sorry for mentioning the complexity—i didn’t mean to imply that it was the morally relevant feature.
I’m not yet sure what sort of preferences give an agent morally weighty status...the only thing I’m pretty sure about is that the morally relevant component is contained somewhere within the preferences, with intelligence as a possible mediating or enabling factor.
Here’s one pattern I think I’ve identified:
I belong within reference Class X.
All beings in Reference Class X care about other beings in Reference Class X, when you extrapolate their volition.
When I hear about altruistic mice, it is evidence that the mouse’s extrapolated volition would cause it to care about Class X-being’s preferences to the extent that it can comprehend them. The cross-species altruism of dogs and dolphins and elephants is an especially strong indicator of Class X membership.
On the other hand, the within-colony altruism of bees (basically identical to Reference Class X except it only applies to members of the colony and I do not belong in it), or the swarms and symbiosis of fishes or bacterial gut flora, wouldn’t count...being in Reference Class X is clearly not the factor behind the altruism in those cases.
...which sounds awfully like reciprocal altruism in practice, doesn’t it? Except that, rather than looking at the actual act of reciprocation of altruism, I’d be extrapolating the agent’s preferences for altruism. Perhaps Class X would be better named “Friendly”, in the “Friendly AI” sense—all beings within the class are to some extent Friendly towards each other.
This is at the rough edge of my thinking though—the ideas as just stated are experimental and I don’t have well defined notions about which preferences matter yet.
Edit: Another (very poorly thought out) trend which seems to emerge is that agents which have a certain sort of awareness are entitled to a sort of bodily autonomy … because it seems immoral to sit around torturing insects if one has no instrumental reason to do so. (But is it immoral in the sense that there are a certain number of insects which morally outweigh a a human? Or is it immoral in a virtue ethic-y, “this behavior signals sadism” sort of way?)
My main point is that I’m mildly guessing that it’s probably safe to narrow down the problem to some combination of preference functions and level of awareness. In any case, I’m almost certain that there exist preference functions are sufficient (but maybe not necessary?) to confer moral weight onto an agent...and though there may be other factors unrelated to preference or intelligence that play a role, preference function is the only thing with a concrete definition that I’ve identified so far.
Just so I understand you better, how would you compare and contrast this kind of pro-X “kin” altruism with utilitarianism?
Utilitarianism has never made much sense to me except as a handy way to talk about things abstractly when precision isn’t important
...but I suppose X would be a class of agents who consider each other’s preferences when they make utilitarian calculations? I pretty much came up with the pro-X idea less than a month ago, and haven’t thought it through very carefully.
Oh, here’s a good example of where preference utilitarianism fails which illustrates it:
10^100 intelligent people terminally prefer that 1 person is tortured. Preference utilitarianism says “do the torture”. My moral instinct says “no, it’s still wrong, no matter how many people prefer it”.
Perhaps under the pro-X system, the reason we can ignore the preferences of 10^100 people is that the preference which they have expressed lies strictly outside category X and therefore that preference can be ignored?
Whereas, if you have a Friendly Paperclipper (cares about X-agents and paperclips with some weight on each), the Friendly moral values put it within X...which means that we should now be willing to cater to its morally neutral paper-clip preferences as well.
(If this reads sloppy, it’s because my thoughts on the matter currently are sloppy)
So...I guess there’s sort of a taxonomy of moral-good, neutral-selfish, and evil preferences...and part of being good means caring about other people’s selfish preferences? And part of being evil means valuing the violation of other’s preferences? And, good agents can simply ignore evil preferences.
And (under the pro-X system), good agents can also ignore the preferences of agents that aren’t in any way good...which seems like it might not be correct, which is why I say that there might be other factors in addition to pro-X that make an agent worth caring about for my moral instincts, but if they exist I don’t know what they are.
Are you perhaps confusing ‘morally wrong’ with ‘a sucky tradeoff that I would prefer not to be bound by’?
Just because torturing one person sucks, just because we find it abhorrent, does not mean that it isn’t the best outcome in various situations. If your definition of ‘moral’ is “best outcome when all things are considered, even though aspects of it suck a lot and are far from ideal”, then yes, torturing someone can in fact be moral. If your definition of ‘moral’ is “those things which I find reprehensible”, then quite probably you can never find torturing someone to be moral. However, there are scenarios where it may still be necessary, or the best option.
Nope...because ..
...because I believe that torturing someone could still instrumentally be the right thing to do on a consequential grounds.
In this scenario, 10^100 people terminally value torturing one person, but I do not care about their preferences, because it is an evil preference.
However, in an alternate scenario, if I had to choose between 10^100 people getting mildly hurt or 1 person getting tortured, I’d choose the one person getting tortured.
In these two scenarios, the preference weights are identical, but in the first scenario the preference of the 10^100 people is evil and therefore irrelevant in my calculations, whereas in the second scenario the needs of 10^100 outweigh the needs of the one.
This is less a discussion about torture, and more a discussion about whose/which preferences matter. Sadistic preferences (involving real harm, not the consensual kink), for example, don’t matter morally—there’s no moral imperative to fulfill those preferences, no “good” done when those preferences are fulfilled and no “evil” resulting from thwarting those preferences.
I think you should temporarily taboo ‘moral’, ‘morality’, and ‘evil’, and simply look at the utility calculations. 10^100 people terminally value something that you ascribe zero or negative value to; therefore, their preferences do not matter to you or will make your universe worse from the standpoint of your utility function.
Which preferences matter? Yours matter to you, and thiers matter to them. There’s no ‘good’ or ‘evil’ in any absolute sense, merely different utility functions that happen to conflict. There’s no utility function which is ‘correct’, except by some arbitrary metric, of which there are many.
Consider another hypothetical utility function: The needs of the 10^100 don’t outweigh the needs of the one, so we let the entire 10^100 suffer when we could eliminate it by inconveniencing one single entity. Neither you nor the 10^100 are happy with this one, but the person about to be tortured may think it’s just fine and dandy...
...I don’t denotatively disagree with anything you’ve said, but I also think you’re sort of missing the point and forgetting the context of the conversation as it was in the preceding comments.
We all have preferences, but we do not always know what our own preferences are. A subset of our preferences (generally those which do not directly reference ourselves) are termed “moral preferences”. The preceding discussion between me and Peter Hurford is an attempt to figure out what our preferences are.
In the above conversation, words like “matter”, “should” and “moral” is understood to mean “the shared preferences of Ishaan, Dentin, and Peter_Hurford which they agree to define as moral”. Since we are all human (and similar in many other ways beyond that), we probably have very similar moral preferences...so any disagreement that arises between us is usually due to one or both of us inaccurately understanding our own preferences.
This is technically true, but it’s also often a semantic stopsign which derails discussions of morality. The fact is that the three of us humans have a very similar notion of “good”, and can speak meaningfully about what it is...the implicitly understood background truths of moral nihilism notwithstanding.
It doesn’t do to exclaim “but wait! good and evil are relative!” during every moral discussion...because here, between us three humans, our moral preferences are pretty much in agreement and we’d all be well served by figuring out exactly those preferences are. It’s not like we’re negotiating morality with aliens.
Precisely...my preferences are all that matter to me, and our preferences are all that matter to us. So if 10^100 sadistic aliens want to torture...so what? We don’t care if they like torture, because we dislike torture and our preferences are all that matter. Who cares about overall utility? “Morality”, for all practical purposes, means shared human morality...or, at least, the shared morality of the humans who are having the discussion.
“Utility” is kind of like “paperclips”...yes, I understand that in the best case scenario it might be possible to create some sort of construct which measures how much “utility” various agent-like objects get from various real world outcomes, but maximizing utility for all agents within this framework is not necessarily my goal...just like maximizing paperclips is not my goal.
So, I’m curious… can you unpack what you mean by “temporarily” in this comment?
For the purposes of this conversation at least. I’ve largely got them taboo’d in general because I find them confusing and full of political connotations; I suspect at least some of that is the problem here as well.
Yet your moral instinct is perfectly fine with having a justice system that puts innocent people in jail with a greater than 1 in 10^100 error rate.
Sure, on instrumental grounds for consequentialist reasons. Not a terminal preference.