There exists a set of maxims which all intelligent and social agents find it in their long-term interest to adhere to, even if adherence is not always in the agent’s short-term interest. These maxims are called a “moral code”, particularly when the long-term incentive to adhere to the code arises by way of the approval or disapproval of other agents who themselves adhere to the code.
This view of morality is a ‘moral realist’ rather than ‘moral conventionalist’ position when it is claimed that there is a theoretically best moral code, and that it may be morally permissible to flout the locally conventional moral code if that code is different from the best code.
I think this provides a variant of moral realism without invoking a Supreme Lawgiver. Kant’s categorical imperative was another (seriously flawed, IMO) attempt to derive some moral rules using no other assumptions beyond human rationality and social lifestyle. The moral realist position may well be incorrect, but it is not a nonsensical position for an atheist to take.
The position you describe is sensical, but it’s not what people (at least on LW) who think “moral realism” is nonsensical mean by “morality”. You’re not saying anything about ultimate ends (which I’m pretty sure is what NMJablonski, e.g., means by “preferences”); the version of “moral realism” that gets rejected is about certain terminal values being spookily metaphysically privileged.
Actually, I am saying something about ultimate ends, at least indirectly. My position only makes sense if long-term ultimate ends become somehow ‘spookily metaphysically privileged’ over short-term ultimate ends.
My position still contains ‘spookiness’ but it is perhaps a less arbitrary kind of ‘magic’ - I’m talking about time-scales rather than laws inscribed on tablets.
Well, I am attempting to claim here that there exists an objective moral code (moral realism) which applies to all agents—both those who care about the long term and those who don’t. Agents who mostly care about the short term will probably be more ethically challenged than agents who easily and naturally defer their gratification. But, in this thread at least, I’m arguing that both short-sighted and long-sighted agents confront the same objective moral code. So, I apparently need to appeal to some irreducible spookiness to justify that long-term bias.
One possible attack: take the moral code and add “Notwithstanding all this, kill the humans.” to the end. This should be superior for all the remaining agents, since humans won’t be using up any resources after that’s accomplished (assuming we can’t put up enough of a fight).
A practical vulnerability to (perhaps unconsciously biased) self-interested gamers: untestable claims that although the moral code is at a local optimum, everyone needs to switch to a far away alternative, and then, after the new equilibrium, things will really be better. Sci-fi rejoinder: this explains why there are so many simulations :)
Yes, in a society with both human and non-human agents, if the humans contribute nothing at all to the non-humans, and only consume resources that might have been otherwise used, then the non-humans will judge the humans to be worthless. Worse than worthless. A drain to be eliminated.
But there is nothing special about my version of ethics in this regard. It is a problem that must be faced in any system of ethics. It is the FAI problem. Eliezer’s solution is apparently to tell the non-human agents as they are created “Humans are to be valued. And don’t you forget it when you self-modify.” I think that a better approach is to make sure that the humans actually contribute something tangible to the well-being of the non-humans.
Perhaps neither approach is totally safe in the long run.
Interestingly, it seems to me that this definition would produce meta-maxims more readily than normal ones—“find out what your local society’s norms are and stick to those” rather than “don’t eat babies”, for example.
Eliezer’s Babyeaters and Superhappies both seemed to follow that meta-rule. (It was less obvious for the latter, but the fact that they had complex societies with different people holding different set roles strongly implies it.)
Interestingly, it seems to me that this definition would produce meta-maxims more readily than normal ones—“find out what your local society’s norms are and stick to those” rather than “don’t eat babies”, for example.
Yes, it does produce meta-maxims in some sense. But meta-maxims can provide useful guidance. “Do unto others …” is a meta-maxim. So is “Don’t steal (anything from anybody)” or “Don’t lie (about anything to anyone)”
As to whether this definition leads to the meta-maxim “find out what your local society’s norms are and stick to those”, that is explicitly not a consequence of this definition. Instead, the meta-maxim would be “work out what your local society’s norms should be and set an example by doing that”.
Thanks for this. Your first paragraph looks very clean to me—not tendentious and even almost falsifiable. But can you tell me how I should understand “best” and “morally permissible” in the second paragraph? (Morally permissible according to the best maxims only, or morally permissible according to all maxims indicated in the first paragraph?)
‘Best’ here means most advantageous to the agents involved—both the focal agent, and the other agents who join in the convention. What I have in mind here is something like the Nash Bargaining Solution, which is both unique and (in some sense) optimal.
“Morally permissible” by the best code, which is also the code mentioned in the first paragraph. What I am claiming is that the ‘moral should’ can be reduced to the ‘practical self-interested should’ if (and it is a big if) it is reasonable to treat the other agents as rational and well-informed.
The “intelligent” qualification could mean just about anything, and sort functions as an applause light on LW, and the “social” part sounds kind of collectivist. Besides that, this is just called having a low time preference, and is a common self-help concept. I don’t see any need to label this morality.
You’re right, I did stop reading carefully toward the end because the Kantian stuff looked like it was just a side note. I’ve now read it carefully, but I still don’t see the significance.
There exists a set of maxims which all intelligent and social agents find it in their long-term interest
I can see how with that definition of morality it could be sensibly theorized as objective. I don’t think that sentence is true, as there are many people (e.g. suicide bombers) whose evaluations of their long-term interest are significant outliers from other agents.
I don’t think that sentence is true, as there are many people (e.g. suicide bombers) whose evaluations of their long-term interest are significant outliers from other agents.
That’s right, but this exception (people whose interests are served by violating the moral norm) itself has a large exception, which is that throughout most of the suicide bomber’s life, he (rightly) respects the moral norm. Bad people can’t be bad every second of their lives—they have to behave themselves the vast majority of the time if for no other reason than to survive until the next opportunity to be bad. The suicide bomber has no interest in surviving once he presses the button, but for every second of his life prior to that, he has an interest in surviving.
And the would-be eventual suicide bomber also, through most of his life, has no choice but to enforce moral behavior in others if he wants to make it to his self-chosen appointment with death.
If we try to imagine someone who never respects recognizable norms—well, it’s hard to imagine, but for one thing they would probably make most “criminally insane” look perfectly normal and safe to be around by contrast.
Upvoted. The events you describe makes sense and your reasoning seems valid. Do you think, based upon any of our discussion, that we disagree on the substance of the issue in any way?
That one agent’s preferences differ greatly from the norm does not automatically make cooperation impossible. In a non-zero-sum game of perfect information, there is always a gain to be made by cooperating. Furthermore, it is usually possible to restructure the game so that it is no longer zero-sum.
For example, a society confronting a would-be suicide bomber will (morally and practically) incarcerate him, if it has the information and the power to do so. And, once thwarted from his primary goal, the would-be bomber may find that he now has some common interests with his captors. The game is no longer zero-sum.
So I don’t think that divergent interests are a fatal objection to my scheme. What may be fatal is that real-world games are not typically games with perfect information. Sometimes, in the real world, it is advantageous to lie about your capabilities, values, and intentions. At least advantageous in the short term. Maybe not in the long term.
That is a zero-sum game. (Linear transformations of the payoff matrices don’t change the game.)
It is also a game with only one player. Not really a game at all.
ETA: If you want to allow ‘games’ where only one ‘agent’ can act, then you can probably construct a non-zero-sum example by offering the active player three choices (A, B, and C). If the active player prefers A to B and B to C, and the passive player prefers B to C and C to A, then the game is non-zero-sum since they both prefer B to C.
I suppose there are cases like this in which what I would call the ‘cooperative’ solution can be reached without any cooperation—it is simply the dominant strategy for each active player. (A in the example above). But excluding that possibility, I don’t believe there are counterexamples.
Rather than telling me how my counterexample violates the spirit of what you meant, can you say what you mean more precisely? What you’re saying in 1. and 2. are literally false, even if I kind of (only kind of) see what you’re getting at.
When I make it precise, it is a tautology. Define a “strictly competitive game” as one in which all ‘pure outcomes’ (i.e. results of pure strategies by all players) are Pareto optimal. Then, in any game which is not ‘strictly competitive’, cooperation can result in an outcome that is Pareto optimal—i.e. better for both players than any outcome that can be achieved without cooperation.
The “counter-example” you supplied is ‘strictly competitive’. Some game theory authors take ‘strictly competitive’ to be synonymous with ‘zero sum’. Some, I now learn, do not.
That one agent’s preferences differ greatly from the norm does not automatically make cooperation impossible.
I wasn’t arguing that cooperation is impossible. From everything you said there it looks like your understanding of morality is similar to mine:
Agents each judging possible outcomes based upon subjective values and taking actions to try to maximize those values, where the ideal strategy can vary between cooperation, competition, etc.
This makes sense I think when you say:
For example, a society confronting a would-be suicide bomber will (morally and practically) incarcerate him
The members of that society do that because they prefer the outcome in which he does not suicide attack them, to one where he does.
once thwarted from his primary goal, the would-be bomber may find that he now has some common interests with his captors
This phrasing seems exactly right to me. The would-be bomber may elect to cooperate, but only if he feels that his long-term values are best fulfilled in that manor. It is also possible that the bomber will resent his captivity, and if released will try again to attack.
If his utility function assigns (carry out martyrdom operation against the great enemy) an astronomically higher value than his own survival or material comfort, it may be impossible for society to contrive circumstances in which he would agree to long term cooperation.
This sort of morality, where agents negotiate their actions based upon their self-interest and the impact of others actions, until they reach an equilibrium, makes perfect sense to me.
Try this:
There exists a set of maxims which all intelligent and social agents find it in their long-term interest to adhere to, even if adherence is not always in the agent’s short-term interest. These maxims are called a “moral code”, particularly when the long-term incentive to adhere to the code arises by way of the approval or disapproval of other agents who themselves adhere to the code.
This view of morality is a ‘moral realist’ rather than ‘moral conventionalist’ position when it is claimed that there is a theoretically best moral code, and that it may be morally permissible to flout the locally conventional moral code if that code is different from the best code.
I think this provides a variant of moral realism without invoking a Supreme Lawgiver. Kant’s categorical imperative was another (seriously flawed, IMO) attempt to derive some moral rules using no other assumptions beyond human rationality and social lifestyle. The moral realist position may well be incorrect, but it is not a nonsensical position for an atheist to take.
The position you describe is sensical, but it’s not what people (at least on LW) who think “moral realism” is nonsensical mean by “morality”. You’re not saying anything about ultimate ends (which I’m pretty sure is what NMJablonski, e.g., means by “preferences”); the version of “moral realism” that gets rejected is about certain terminal values being spookily metaphysically privileged.
Actually, I am saying something about ultimate ends, at least indirectly. My position only makes sense if long-term ultimate ends become somehow ‘spookily metaphysically privileged’ over short-term ultimate ends.
My position still contains ‘spookiness’ but it is perhaps a less arbitrary kind of ‘magic’ - I’m talking about time-scales rather than laws inscribed on tablets.
How is this different from “if the agents under consideration care about the long run more than the short run”?
Well, I am attempting to claim here that there exists an objective moral code (moral realism) which applies to all agents—both those who care about the long term and those who don’t. Agents who mostly care about the short term will probably be more ethically challenged than agents who easily and naturally defer their gratification. But, in this thread at least, I’m arguing that both short-sighted and long-sighted agents confront the same objective moral code. So, I apparently need to appeal to some irreducible spookiness to justify that long-term bias.
Interesting.
One possible attack: take the moral code and add “Notwithstanding all this, kill the humans.” to the end. This should be superior for all the remaining agents, since humans won’t be using up any resources after that’s accomplished (assuming we can’t put up enough of a fight).
A practical vulnerability to (perhaps unconsciously biased) self-interested gamers: untestable claims that although the moral code is at a local optimum, everyone needs to switch to a far away alternative, and then, after the new equilibrium, things will really be better. Sci-fi rejoinder: this explains why there are so many simulations :)
Yes, in a society with both human and non-human agents, if the humans contribute nothing at all to the non-humans, and only consume resources that might have been otherwise used, then the non-humans will judge the humans to be worthless. Worse than worthless. A drain to be eliminated.
But there is nothing special about my version of ethics in this regard. It is a problem that must be faced in any system of ethics. It is the FAI problem. Eliezer’s solution is apparently to tell the non-human agents as they are created “Humans are to be valued. And don’t you forget it when you self-modify.” I think that a better approach is to make sure that the humans actually contribute something tangible to the well-being of the non-humans.
Perhaps neither approach is totally safe in the long run.
Interestingly, it seems to me that this definition would produce meta-maxims more readily than normal ones—“find out what your local society’s norms are and stick to those” rather than “don’t eat babies”, for example.
Eliezer’s Babyeaters and Superhappies both seemed to follow that meta-rule. (It was less obvious for the latter, but the fact that they had complex societies with different people holding different set roles strongly implies it.)
Yes, it does produce meta-maxims in some sense. But meta-maxims can provide useful guidance. “Do unto others …” is a meta-maxim. So is “Don’t steal (anything from anybody)” or “Don’t lie (about anything to anyone)”
As to whether this definition leads to the meta-maxim “find out what your local society’s norms are and stick to those”, that is explicitly not a consequence of this definition. Instead, the meta-maxim would be “work out what your local society’s norms should be and set an example by doing that”.
Thanks for this. Your first paragraph looks very clean to me—not tendentious and even almost falsifiable. But can you tell me how I should understand “best” and “morally permissible” in the second paragraph? (Morally permissible according to the best maxims only, or morally permissible according to all maxims indicated in the first paragraph?)
‘Best’ here means most advantageous to the agents involved—both the focal agent, and the other agents who join in the convention. What I have in mind here is something like the Nash Bargaining Solution, which is both unique and (in some sense) optimal.
“Morally permissible” by the best code, which is also the code mentioned in the first paragraph. What I am claiming is that the ‘moral should’ can be reduced to the ‘practical self-interested should’ if (and it is a big if) it is reasonable to treat the other agents as rational and well-informed.
The “intelligent” qualification could mean just about anything, and sort functions as an applause light on LW, and the “social” part sounds kind of collectivist. Besides that, this is just called having a low time preference, and is a common self-help concept. I don’t see any need to label this morality.
I didn’t downvote, but you didn’t address the crypto-Kantian angle at all. So it seems like you stopped reading too soon.
You’re right, I did stop reading carefully toward the end because the Kantian stuff looked like it was just a side note. I’ve now read it carefully, but I still don’t see the significance.
I can see how with that definition of morality it could be sensibly theorized as objective. I don’t think that sentence is true, as there are many people (e.g. suicide bombers) whose evaluations of their long-term interest are significant outliers from other agents.
That’s right, but this exception (people whose interests are served by violating the moral norm) itself has a large exception, which is that throughout most of the suicide bomber’s life, he (rightly) respects the moral norm. Bad people can’t be bad every second of their lives—they have to behave themselves the vast majority of the time if for no other reason than to survive until the next opportunity to be bad. The suicide bomber has no interest in surviving once he presses the button, but for every second of his life prior to that, he has an interest in surviving.
And the would-be eventual suicide bomber also, through most of his life, has no choice but to enforce moral behavior in others if he wants to make it to his self-chosen appointment with death.
If we try to imagine someone who never respects recognizable norms—well, it’s hard to imagine, but for one thing they would probably make most “criminally insane” look perfectly normal and safe to be around by contrast.
Upvoted. The events you describe makes sense and your reasoning seems valid. Do you think, based upon any of our discussion, that we disagree on the substance of the issue in any way?
If so, what part of my map differs from yours?
I’m withholding judgment for now because I’m not sure if or where we differ on any specifics.
That one agent’s preferences differ greatly from the norm does not automatically make cooperation impossible. In a non-zero-sum game of perfect information, there is always a gain to be made by cooperating. Furthermore, it is usually possible to restructure the game so that it is no longer zero-sum.
For example, a society confronting a would-be suicide bomber will (morally and practically) incarcerate him, if it has the information and the power to do so. And, once thwarted from his primary goal, the would-be bomber may find that he now has some common interests with his captors. The game is no longer zero-sum.
So I don’t think that divergent interests are a fatal objection to my scheme. What may be fatal is that real-world games are not typically games with perfect information. Sometimes, in the real world, it is advantageous to lie about your capabilities, values, and intentions. At least advantageous in the short term. Maybe not in the long term.
Can’t I construct trivial examples where this is false? E.g. the one-by-two payoff matrices (0,100) and (1,-1).
That is a zero-sum game. (Linear transformations of the payoff matrices don’t change the game.)
It is also a game with only one player. Not really a game at all.
ETA: If you want to allow ‘games’ where only one ‘agent’ can act, then you can probably construct a non-zero-sum example by offering the active player three choices (A, B, and C). If the active player prefers A to B and B to C, and the passive player prefers B to C and C to A, then the game is non-zero-sum since they both prefer B to C.
I suppose there are cases like this in which what I would call the ‘cooperative’ solution can be reached without any cooperation—it is simply the dominant strategy for each active player. (A in the example above). But excluding that possibility, I don’t believe there are counterexamples.
Rather than telling me how my counterexample violates the spirit of what you meant, can you say what you mean more precisely? What you’re saying in 1. and 2. are literally false, even if I kind of (only kind of) see what you’re getting at.
When I make it precise, it is a tautology. Define a “strictly competitive game” as one in which all ‘pure outcomes’ (i.e. results of pure strategies by all players) are Pareto optimal. Then, in any game which is not ‘strictly competitive’, cooperation can result in an outcome that is Pareto optimal—i.e. better for both players than any outcome that can be achieved without cooperation.
The “counter-example” you supplied is ‘strictly competitive’. Some game theory authors take ‘strictly competitive’ to be synonymous with ‘zero sum’. Some, I now learn, do not.
I wasn’t arguing that cooperation is impossible. From everything you said there it looks like your understanding of morality is similar to mine:
Agents each judging possible outcomes based upon subjective values and taking actions to try to maximize those values, where the ideal strategy can vary between cooperation, competition, etc.
This makes sense I think when you say:
The members of that society do that because they prefer the outcome in which he does not suicide attack them, to one where he does.
This phrasing seems exactly right to me. The would-be bomber may elect to cooperate, but only if he feels that his long-term values are best fulfilled in that manor. It is also possible that the bomber will resent his captivity, and if released will try again to attack.
If his utility function assigns (carry out martyrdom operation against the great enemy) an astronomically higher value than his own survival or material comfort, it may be impossible for society to contrive circumstances in which he would agree to long term cooperation.
This sort of morality, where agents negotiate their actions based upon their self-interest and the impact of others actions, until they reach an equilibrium, makes perfect sense to me.