Just because morality is personal doesn’t make it not real. If you model people as agents with utility functions, the reason not to change is obvious—if you change, you won’t do all the things you value. Non-fanatics can do that the same as fanatics.
The difference comes when you factor in human irrationality. And sure, fanatics might resist where anyone sane would give in. “We will blow up this city unless you renounce the Leader,” something like that. But on the other hand, rational humans might resist techniques that play on human irrationality, where fanatics might even be more susceptible than average. Good cop / bad cop for example.
What about on a national scale, where, say, an evil mastermind threatens to nuke every nation that does not start worshiping the flying spaghetti monster? Well, what a rational society would do is compare benefits and downsides, and worship His Noodliness if it was worth it. Fanatics would get nuked. I fail to see how this is an argument for why we shouldn’t be rational.
if you want to win those wars, you may need something better than “we have our morals and they have theirs.”
And that’s why Strawmansylvania has never won a single battle, I agree. Just because morality is personal doesn’t make it unmomving.
Just because morality is personal doesn’t make it not real. If you model people as agents with utility functions, the reason not to change is obvious—if you change, you won’t do all the things you value.
Does this imply that if a rational actor has terminal values that are internally consistent and in principle satisfiable, it would always be irrational for the actor to change those values or allow them to change?
That doesn’t seem right either. Somehow, an individual improving their moral beliefs as they mature, the notional Vicar of Bray, and Pierre Laval are all substantially different cases of people changing their [terminal] beliefs in response to events. There’s something badly wrong with a theory that can’t distinguish those cases.
Also, my apologies if this has been already discussed to death on LW or elsewhere—I spent some time poking and didn’t see anything on this point.
Does this imply that if a rational actor has terminal values that are internally consistent and in principle satisfiable, it would always be irrational for the actor to change those values or allow them to change?
No, but it sets a high standard—If you value, say, the company of your family, then modifying to not want that (and therefore not spend much time with your family) costs as much as if you were kept away from your family by force for the rest of your life. So any threats have to be pretty damn serious, and maybe not even death would work if you know important secrets or do not highly value living without some key values.
an individual improving their moral beliefs as they mature, the notional Vicar of Bray, and Pierre Laval are all substantially different cases of people changing their [terminal] beliefs
I wouldn’t call all of those cases of modifying terminal values. From some quick googling (I didn’t know about the Vicar of Bray), what the Vicar of Bray cared about was being the vicar of Bray. What Pierre Laval cared about was being the head of the government and not being killed, maybe. So they’re maybe not good examples of changing terminal values, as opposed to instrumental ones.
Also “improving their moral beliefs as they mature” is a very odd concept once you think about it. How do you judge whether a moral belief is right to hold correctly without having a correct ultimate belief from the start, to do the judging? It’s really an example of how humans are emphatically not rational agents—we follow a bunch of evolved and cultural rules, which can appear to produce consistent behavior, but really have all these holes and internal conflicts. And things can change suddenly, without the sort of rational deliberation described above.
Also “improving their moral beliefs as they mature” is a very odd concept once you think about it. How do you judge whether a moral belief is right to hold correctly without having a correct ultimate belief from the start, to do the judging?
You could say the same about “improving our standards of scientific inference.” Circular? Perhaps, but it needn’t be a vicious circle. It’s pretty clear that we’ve accomplished it, so it must be possible.
I would cheerfully agree that humans aren’t rational and routinely change their minds about morality for non-rational reasons.
This is one of the things I was trying to get at. Ask when we should change our minds for non-rational reasons, and when we should attempt to change others’ minds using non-rational means.
The same examples I mentioned above work for these questions too.
Here’s what I had in mind with the reference to the Vicar of Bray. Imagine an individual with two terminal values: “Stay alive and employed” and the reigning orthodoxy at the moment. The individual sincerely believes in both, and whenever they start to conflict, changes their beliefs about the orthodoxy. He is quite sincere in advocating for the ruling ideology at each point in time; he really does believe in divine right of kings, just so long as it’s not a dangerous belief to hold.
The beliefs in question are at least potentially terminal moral beliefs. Without delving deep into the history, let’s stipulate for the purpose of the conversation that we’re talking about a rational actor who has a sequence of terminal moral beliefs about what constitutes a just government, and that these beliefs shift with the political climate.
Now for contrast, let’s consider a hypothetical rational but very selfish child. The child’s parents attempt and succeed in changing the child’s values to be less selfish. They do this by the usual parental tactics of punishment and example-setting, not by rational argument. By your social standard and mine, this is an improvement to the child.
Both the vicar and the child are updating their moral beliefs in response to outside pressure, not rational deliberation. The general consensus is that parents are obligated to bring up their children not to be overly self-centered and that reasoning with children is not a sufficient pedagogic technique But conversely that coercive government pressure on religion is ignoble.
Is this simply that you and I think “a change in moral beliefs, brought about by non-reasonable means is good (all else equal), if it significantly improves the beliefs of the subject by my standards”?
I think the caveats will turn out to matter a lot. One of the things that human moral beliefs do, in practice, is give other humans some reasons to trust you. If I know that you are committed, for non-instrumental reasons, to avoid manipulating* me into changing my values, that gives me reasons to trust you. Conversely, if your moral view is that it’s legitimate to lie to people to make them do what you want, people will trust you less.
Obviously, people have incentives to lie about their true values. I think equally obviously, people are paying attention and looking hard for that sort of hypocrisy.
*This sentence is true for a range of possible expansions of “manipulating”.
My statement was more observational than ideal, though. Sure, a rational agent can be averse to manipulating other people (and humans often are too), because agents can care about whatever they want. But that doesn’t bear very strongly on how the language is used compared to the fact that in real-world usage I see people say things like “improved his morals” by only three standards: consistency, how much society approves, and how much you approve.
Just because morality is personal doesn’t make it not real. If you model people as agents with utility functions, the reason not to change is obvious—if you change, you won’t do all the things you value. Non-fanatics can do that the same as fanatics.
The difference comes when you factor in human irrationality. And sure, fanatics might resist where anyone sane would give in. “We will blow up this city unless you renounce the Leader,” something like that. But on the other hand, rational humans might resist techniques that play on human irrationality, where fanatics might even be more susceptible than average. Good cop / bad cop for example.
What about on a national scale, where, say, an evil mastermind threatens to nuke every nation that does not start worshiping the flying spaghetti monster? Well, what a rational society would do is compare benefits and downsides, and worship His Noodliness if it was worth it. Fanatics would get nuked. I fail to see how this is an argument for why we shouldn’t be rational.
And that’s why Strawmansylvania has never won a single battle, I agree. Just because morality is personal doesn’t make it unmomving.
Does this imply that if a rational actor has terminal values that are internally consistent and in principle satisfiable, it would always be irrational for the actor to change those values or allow them to change?
That doesn’t seem right either. Somehow, an individual improving their moral beliefs as they mature, the notional Vicar of Bray, and Pierre Laval are all substantially different cases of people changing their [terminal] beliefs in response to events. There’s something badly wrong with a theory that can’t distinguish those cases.
Also, my apologies if this has been already discussed to death on LW or elsewhere—I spent some time poking and didn’t see anything on this point.
No, but it sets a high standard—If you value, say, the company of your family, then modifying to not want that (and therefore not spend much time with your family) costs as much as if you were kept away from your family by force for the rest of your life. So any threats have to be pretty damn serious, and maybe not even death would work if you know important secrets or do not highly value living without some key values.
I wouldn’t call all of those cases of modifying terminal values. From some quick googling (I didn’t know about the Vicar of Bray), what the Vicar of Bray cared about was being the vicar of Bray. What Pierre Laval cared about was being the head of the government and not being killed, maybe. So they’re maybe not good examples of changing terminal values, as opposed to instrumental ones.
Also “improving their moral beliefs as they mature” is a very odd concept once you think about it. How do you judge whether a moral belief is right to hold correctly without having a correct ultimate belief from the start, to do the judging? It’s really an example of how humans are emphatically not rational agents—we follow a bunch of evolved and cultural rules, which can appear to produce consistent behavior, but really have all these holes and internal conflicts. And things can change suddenly, without the sort of rational deliberation described above.
You could say the same about “improving our standards of scientific inference.” Circular? Perhaps, but it needn’t be a vicious circle. It’s pretty clear that we’ve accomplished it, so it must be possible.
I would cheerfully agree that humans aren’t rational and routinely change their minds about morality for non-rational reasons.
This is one of the things I was trying to get at. Ask when we should change our minds for non-rational reasons, and when we should attempt to change others’ minds using non-rational means.
The same examples I mentioned above work for these questions too.
Here’s what I had in mind with the reference to the Vicar of Bray. Imagine an individual with two terminal values: “Stay alive and employed” and the reigning orthodoxy at the moment. The individual sincerely believes in both, and whenever they start to conflict, changes their beliefs about the orthodoxy. He is quite sincere in advocating for the ruling ideology at each point in time; he really does believe in divine right of kings, just so long as it’s not a dangerous belief to hold.
The beliefs in question are at least potentially terminal moral beliefs. Without delving deep into the history, let’s stipulate for the purpose of the conversation that we’re talking about a rational actor who has a sequence of terminal moral beliefs about what constitutes a just government, and that these beliefs shift with the political climate.
Now for contrast, let’s consider a hypothetical rational but very selfish child. The child’s parents attempt and succeed in changing the child’s values to be less selfish. They do this by the usual parental tactics of punishment and example-setting, not by rational argument. By your social standard and mine, this is an improvement to the child.
Both the vicar and the child are updating their moral beliefs in response to outside pressure, not rational deliberation. The general consensus is that parents are obligated to bring up their children not to be overly self-centered and that reasoning with children is not a sufficient pedagogic technique But conversely that coercive government pressure on religion is ignoble.
Is this simply that you and I think “a change in moral beliefs, brought about by non-reasonable means is good (all else equal), if it significantly improves the beliefs of the subject by my standards”?
I’d agree with that. Maybe with some caveats, but generally yes.
I think the caveats will turn out to matter a lot. One of the things that human moral beliefs do, in practice, is give other humans some reasons to trust you. If I know that you are committed, for non-instrumental reasons, to avoid manipulating* me into changing my values, that gives me reasons to trust you. Conversely, if your moral view is that it’s legitimate to lie to people to make them do what you want, people will trust you less.
Obviously, people have incentives to lie about their true values. I think equally obviously, people are paying attention and looking hard for that sort of hypocrisy.
*This sentence is true for a range of possible expansions of “manipulating”.
My statement was more observational than ideal, though. Sure, a rational agent can be averse to manipulating other people (and humans often are too), because agents can care about whatever they want. But that doesn’t bear very strongly on how the language is used compared to the fact that in real-world usage I see people say things like “improved his morals” by only three standards: consistency, how much society approves, and how much you approve.