I’ve been working my way through the Sequences—and I’m wondering a lot about this essay, in light of the previously-introduce notion of ‘how do you decide what values, given to you by natural selection, you are going to keep?’
Could someone use the stances you develop here, EY, to argue for something like Aristotelian ethics? (Which, admittedly, I may not properly understand fully, but my basic idea is:)
‘You chose to keep human life, human happiness, love, and learning as values in YOUR utility function,’ says the objector, ’even though you know where they came from. You decided that you wanted them anyway. You did this because you had to start somewhere, and you claim that if you stripped away everything provided by natural selection you wouldn’t be left with anything. Under the same logic, why can’t I keep all the ethical injunctions as terminal values?
‘Your explanation of where ‘the ends does not justify the means’, is very clever and all. Your explanation of ‘thou shalt not kill’ is very clever. But so what if we know where they came from? If we know why nature selected on them, in our specific case? I’m no more obligated to dispose of it than I am to dispose of ‘human happiness is good’.′
Is the counter-argument simply that this leads to a utility function you would call inconsistent?
Oh, and...sorry for commenting on all these dead threads...it’s a pity I got here so late.
No matter how you complete this pattern, the answer is obviously yes.
The reasoning behind “in certain situations, you should not do the best thing” is based on observation that human rationality is limited, and that in certain situations it works even significantly worse than on average. It is the same line of reasoning that would make you advise people to e.g. not sign contracts while they are drunk, even if those contracts seem very good—maybe especially not when the contracts seem too good to be true.
But imagine that you are talking to a drunkard who is in deep denial about his alcoholism (“hey, I only had one bottle of vodka, that’s nothing for me!”). If you instruct him to not sign contracts while drunk, he will sign one anyway, and tell you that he was’t that drunk when he signed it. To make a rule he couldn’t dismiss so easily, you would have to teach him to e.g. never sign a contract immediately, but always read it, read it again 24 hours later, read it again 48 hours later, and use an advice of at least three different family members and refuse to sign it if two of them say no. That is a rule that would have a chance to work even when he is in denial about his state, as long as he doesn’t want to break the rule openly. If the person is a complete idiot, you may tell him to never sign anything unless he discussed it with his lawyer (and no, he is not allowed to choose a different lawyer at the last moment). Such rules are designed to protect people against their own stupidity when interpreting the rules.
Similarly, at the moments when people are least rational, they are most likely to insist that they are the smart ones who “have finally seen the light”, and everyone else in an idiot, especially those who try to make them aware of their moments of irrationality. You can’t simply give them a rule “don’t do extremely costly things with small probability of success when your rationality is impaired”, because they will just say their rationality is not impaired, and the probability of success is obviously 100%. Thus the rule is “don’t do extremely costly things, full stop”.
Technically, sometimes the rule is not optimal. But following the rule makes much less harm on average than when you try to use your impaired reasoning at the moment to evaluate whether the rule applies to this specific situation or not. Because of the nature of the impairment, the moments when it is necessary are exactly the moments it will seem it does not apply because the situation is somehow exceptional (hint: all situations are somehow exceptional).
All this is unrelated to the issue of values and utility function. (Which may further complicate the situation.)
It is simply human nature that when there is a chance to grab power, it seems from inside like there is a unique opportunity to create a lot of good (other than “getting more power for me” itself) by violating some rule of “decent behavior”. What usually happens is that the expected good does not actually happen, or is very short-lived while the negative consequences remain for long.
In theory, an artificial intelligence which did not arise by natural selection (which rewards agents pretending to others and themselves to be doing useful stuff for the tribe, when all they actually did was moving up on the power ladder, often at the total expense of the tribe) could be able to evaluate things correctly. Just like a sufficiently sheltered artificial intelligence could remain working correctly even when you pour a bottle of vodka on its case. This reasoning does not apply to humans.
I understand why the notions exist—I was trying to address the question of ‘what explainable-moral-intuitions should we keep as terminal values, and how do we tell them apart from those we shouldn’t’.
But your first sentence is taken very much to heart, sir.
Maybe I’m being silly here, in hindsight. Certain intuitive desires are reducible to others, and some, like ‘love/happiness/fun/etc.’ are probably not. It feels obvious that most people should immediately see that. Yes, they want a given ethical injunction to be obeyed, but not as a fundamental/terminal value.
Then again—there are Catholic moralists, including, I think, some Catholics I know personally, who firmly believe that (for example) stealing is wrong because stealing is wrong. Not for any other reason. Not because it brings harm to the person being stolen from. If you bring up exceptions—‘what about an orphan who will starve if they don’t steal that bread?’ they argue that this doesn’t count as stealing, not that it ‘proves that stealing isn’t really wrong.’ For them, every exception is simply to be included as another fundamental rule. At least, that’s the mindset, as far as I can tell. I saw the specific argument above being formulated for use against moral relativists, who were apparently out to destroy society by showing that different things were right for different people.
Even though this article is about AI, and even though we should not trust ourselves to understand when we should be excepted from an injunction—this seems like a belief that might eventually have some negative real-world consequences. See, potentially, ‘homosexuality is wrong because homosexuality is wrong’?
If I tried to tell any of these people about how ethical injunctions could be explained as heuristics for achieving higher terminal values—I can already feel myself being accused of shuffling things around, trying to convert goods into other incompatible goods in order to justify some sinister, contradictory worldview.
If I brought up reductionism, it seems almost trivial—while I’m simulating their mind—to point out that no one has ever provably applied reduction to morals.
So maybe let me rephrase: is there any way I could talk them out of it?
I guess some people are unable to deal with uncertainty, especially when it concerns important things (such as “I am not 100% sure whether doing A or doing B will make my soul burn forever in hell, but I have to make a decision now anyway”). The standard human way to deal with unpleasant information is to deny it. Catholic theologicians don’t have an option of denying hell, so the obvious solution is to deny uncertainty.
“There is a rule X, which is perfectly unambiguous and perfectly good.” “But here is this non-central situation where following the rule blindly seems bad.” ”There is this ad-hoc rule Y, which covers the special situation, so the whole system is perfectly unambiguous and perfectly good.” “But here is another situation where...” ”There is another ad-hoc rule Z, which covers the other situation...” “But there is also...” ”There is yet another ad-hoc rule...”
You can play this game forever, adding epicycles upon epicycles, but the answer is always going to be that the system is perfectly unambiguous and perfectly good. It is also obvious how they are cheating to achieve that. Also, the starving orphan is probably not aware of all these theological rules and exceptions, so obviously the answer is designed to make the theologician feel happy about the unambiguity of the situation.
I don’t think you can actually talk people out of their emotional needs.
Now suppose someone comes to you and tells you that they will save one billion lives if you promise to do evil for the rest of your life to the best of your ability.
Suppose you decide that overall you will not be able to do enough evil to counteract saving one billion lives. Should you make the agreement and do evil for the rest of your life to the best of your ability?
If you do, your actions will have overall good effects. And if you do, you will be doing evil, or you will not be fulfilling your promise.
If you want to talk to people, you need to first understand what they are saying. And they saying that the question that is important to them is, “Is this action good or evil,” not “Are the results good or evil?” Those are two different questions, and there is nothing to prevent them from having different answers.
Then again—there are Catholic moralists, including, I think, some Catholics I know personally, who firmly believe that (for example) stealing is wrong because stealing is wrong. Not for any other reason.
This sounds like deontological ethics. It’s not by any means unique to Catholicism; it’s just the general idea that being good involves following a (presumably carefully chosen) list of rules.
Not all Catholics are deontologists; not all deontologists are Catholic. And, I may be misreading here, but I think your worry is more about deontology than Catholicism; that is, it’s more about people who follow a list of rules instead of trying consequentialism or virtue ethics or something else along those lines. Is this accurate?
I’ve been working my way through the Sequences—and I’m wondering a lot about this essay, in light of the previously-introduce notion of ‘how do you decide what values, given to you by natural selection, you are going to keep?’
Could someone use the stances you develop here, EY, to argue for something like Aristotelian ethics? (Which, admittedly, I may not properly understand fully, but my basic idea is:)
‘You chose to keep human life, human happiness, love, and learning as values in YOUR utility function,’ says the objector, ’even though you know where they came from. You decided that you wanted them anyway. You did this because you had to start somewhere, and you claim that if you stripped away everything provided by natural selection you wouldn’t be left with anything. Under the same logic, why can’t I keep all the ethical injunctions as terminal values?
‘Your explanation of where ‘the ends does not justify the means’, is very clever and all. Your explanation of ‘thou shalt not kill’ is very clever. But so what if we know where they came from? If we know why nature selected on them, in our specific case? I’m no more obligated to dispose of it than I am to dispose of ‘human happiness is good’.′
Is the counter-argument simply that this leads to a utility function you would call inconsistent?
Oh, and...sorry for commenting on all these dead threads...it’s a pity I got here so late.
No matter how you complete this pattern, the answer is obviously yes.
The reasoning behind “in certain situations, you should not do the best thing” is based on observation that human rationality is limited, and that in certain situations it works even significantly worse than on average. It is the same line of reasoning that would make you advise people to e.g. not sign contracts while they are drunk, even if those contracts seem very good—maybe especially not when the contracts seem too good to be true.
But imagine that you are talking to a drunkard who is in deep denial about his alcoholism (“hey, I only had one bottle of vodka, that’s nothing for me!”). If you instruct him to not sign contracts while drunk, he will sign one anyway, and tell you that he was’t that drunk when he signed it. To make a rule he couldn’t dismiss so easily, you would have to teach him to e.g. never sign a contract immediately, but always read it, read it again 24 hours later, read it again 48 hours later, and use an advice of at least three different family members and refuse to sign it if two of them say no. That is a rule that would have a chance to work even when he is in denial about his state, as long as he doesn’t want to break the rule openly. If the person is a complete idiot, you may tell him to never sign anything unless he discussed it with his lawyer (and no, he is not allowed to choose a different lawyer at the last moment). Such rules are designed to protect people against their own stupidity when interpreting the rules.
Similarly, at the moments when people are least rational, they are most likely to insist that they are the smart ones who “have finally seen the light”, and everyone else in an idiot, especially those who try to make them aware of their moments of irrationality. You can’t simply give them a rule “don’t do extremely costly things with small probability of success when your rationality is impaired”, because they will just say their rationality is not impaired, and the probability of success is obviously 100%. Thus the rule is “don’t do extremely costly things, full stop”.
Technically, sometimes the rule is not optimal. But following the rule makes much less harm on average than when you try to use your impaired reasoning at the moment to evaluate whether the rule applies to this specific situation or not. Because of the nature of the impairment, the moments when it is necessary are exactly the moments it will seem it does not apply because the situation is somehow exceptional (hint: all situations are somehow exceptional).
All this is unrelated to the issue of values and utility function. (Which may further complicate the situation.)
It is simply human nature that when there is a chance to grab power, it seems from inside like there is a unique opportunity to create a lot of good (other than “getting more power for me” itself) by violating some rule of “decent behavior”. What usually happens is that the expected good does not actually happen, or is very short-lived while the negative consequences remain for long.
In theory, an artificial intelligence which did not arise by natural selection (which rewards agents pretending to others and themselves to be doing useful stuff for the tribe, when all they actually did was moving up on the power ladder, often at the total expense of the tribe) could be able to evaluate things correctly. Just like a sufficiently sheltered artificial intelligence could remain working correctly even when you pour a bottle of vodka on its case. This reasoning does not apply to humans.
I understand why the notions exist—I was trying to address the question of ‘what explainable-moral-intuitions should we keep as terminal values, and how do we tell them apart from those we shouldn’t’.
But your first sentence is taken very much to heart, sir.
Maybe I’m being silly here, in hindsight. Certain intuitive desires are reducible to others, and some, like ‘love/happiness/fun/etc.’ are probably not. It feels obvious that most people should immediately see that. Yes, they want a given ethical injunction to be obeyed, but not as a fundamental/terminal value.
Then again—there are Catholic moralists, including, I think, some Catholics I know personally, who firmly believe that (for example) stealing is wrong because stealing is wrong. Not for any other reason. Not because it brings harm to the person being stolen from. If you bring up exceptions—‘what about an orphan who will starve if they don’t steal that bread?’ they argue that this doesn’t count as stealing, not that it ‘proves that stealing isn’t really wrong.’ For them, every exception is simply to be included as another fundamental rule. At least, that’s the mindset, as far as I can tell. I saw the specific argument above being formulated for use against moral relativists, who were apparently out to destroy society by showing that different things were right for different people.
Even though this article is about AI, and even though we should not trust ourselves to understand when we should be excepted from an injunction—this seems like a belief that might eventually have some negative real-world consequences. See, potentially, ‘homosexuality is wrong because homosexuality is wrong’?
If I tried to tell any of these people about how ethical injunctions could be explained as heuristics for achieving higher terminal values—I can already feel myself being accused of shuffling things around, trying to convert goods into other incompatible goods in order to justify some sinister, contradictory worldview.
If I brought up reductionism, it seems almost trivial—while I’m simulating their mind—to point out that no one has ever provably applied reduction to morals.
So maybe let me rephrase: is there any way I could talk them out of it?
I guess some people are unable to deal with uncertainty, especially when it concerns important things (such as “I am not 100% sure whether doing A or doing B will make my soul burn forever in hell, but I have to make a decision now anyway”). The standard human way to deal with unpleasant information is to deny it. Catholic theologicians don’t have an option of denying hell, so the obvious solution is to deny uncertainty.
“There is a rule X, which is perfectly unambiguous and perfectly good.”
“But here is this non-central situation where following the rule blindly seems bad.”
”There is this ad-hoc rule Y, which covers the special situation, so the whole system is perfectly unambiguous and perfectly good.”
“But here is another situation where...”
”There is another ad-hoc rule Z, which covers the other situation...”
“But there is also...”
”There is yet another ad-hoc rule...”
You can play this game forever, adding epicycles upon epicycles, but the answer is always going to be that the system is perfectly unambiguous and perfectly good. It is also obvious how they are cheating to achieve that. Also, the starving orphan is probably not aware of all these theological rules and exceptions, so obviously the answer is designed to make the theologician feel happy about the unambiguity of the situation.
I don’t think you can actually talk people out of their emotional needs.
Here is a perfectly good rule: don’t do evil.
Now suppose someone comes to you and tells you that they will save one billion lives if you promise to do evil for the rest of your life to the best of your ability.
Suppose you decide that overall you will not be able to do enough evil to counteract saving one billion lives. Should you make the agreement and do evil for the rest of your life to the best of your ability?
If you do, your actions will have overall good effects. And if you do, you will be doing evil, or you will not be fulfilling your promise.
If you want to talk to people, you need to first understand what they are saying. And they saying that the question that is important to them is, “Is this action good or evil,” not “Are the results good or evil?” Those are two different questions, and there is nothing to prevent them from having different answers.
This sounds like deontological ethics. It’s not by any means unique to Catholicism; it’s just the general idea that being good involves following a (presumably carefully chosen) list of rules.
Not all Catholics are deontologists; not all deontologists are Catholic. And, I may be misreading here, but I think your worry is more about deontology than Catholicism; that is, it’s more about people who follow a list of rules instead of trying consequentialism or virtue ethics or something else along those lines. Is this accurate?