For some people, probably. Let’s take a step back.
Morality comes from the “heart”. It’s made of feelings. Utilitarianism (and much of what falls under moral philosophy) is one of many attempt to make a consistent set of rules to describe inconsistent feelings. The purpose of making a consistent set of rules is 1) to convince others of the morality of an action and 2) because we morally feel aversion to hypocrisy and crave moral consistency.
Keeping those aims in mind, drawing the line across all humans, sentient beings, etc has the following benefits:
1) The creators might feel that the equation describes the way they feel better when they factor in all humans. They might hold it as a deontological standard to care about all humans, or they might feel a sense of fairness, or they might have empathy for everyone, etc.
2) Drawing the line across all humans allows you to use the utilitarian standard to negotiate compromises with any arbitrary human you come across. Many humans, having the feelings described in [1], will instinctively accept utilitarianism as a valid way to think about things.
There are plenty of things that are problematic here, but that is why utilitarianism defaults to include the whole human race. As with all things moral, that’s just an arbitrary choice on our part, and we could easily have done it a different way. We can restrict it to a smaller subset of humans, we can broaden it to non-human things which seem agent-like enough to be worth describing with a utility function, etc. Many utilitarians include animals, for example.
Morality comes from the “heart”. It’s made of feelings.
People use feelings/System1 to do morality. That doesn’t make it an oracle. Thinking might be more accurate.
As with all things moral, that’s just an arbitrary choice on our part
If you don’t know how to solve a problem, you guess. But that doens’t mean anything goes. Would anyone include rocks
in the Circle? Probably not, since they don’t have feelings, values, or preferences. So there seem to be some constraints.
People use feelings/System1 to do morality. That doesn’t make it an oracle. Thinking might be more accurate.
Accurate? How can you speak of a moral preference being “accurate” or not? Moral preferences simply are. There are some meta-ethics sequences here that explain the arbitrariness of our moral preferences more eloquently , and here is a fun story that tangentially illustrates it
Would anyone include rocks in the Circle? Probably not, since they don’t have feelings, values, or preferences. So there seem to be some constraints.
I bet I can find you someone who would say that burning the Quran or the Bible is inherently immoral.
Quite a few of them no doubt. Of course, the overwhelming majority of people who would say that burning the Quran or the Bible is inherently immoral would also say that it’s immoral by virtue of the preferences of an entity that, on their view, is in fact capable of having preferences.
Of course, I’m sure I could find someone who would say rocks have feelings, values, and preferences.
“the overwhelming majority of people who would say that burning the Quran or the Bible is inherently immoral would also say that it’s immoral by virtue of the preferences of an entity that, on their view, is in fact capable of having preferences.”
Conversationalists will want to preserve ecosystems, even where those ecosystems are already well studied by science, even when the ecosystem contains no sentient beings (plants, fungi, microbes), even when destroying the ecosystem has many advantages for humans, because they think the ecosystem is intrinsically valuable independently of the effect on beings with feelings, values, and preferences.
Some looser examples...
Pro-life advocates say that beings without preferences have rights by virtue of future preferences. Not all of them are religious.
Hindus treat books (all books in general) with reverence because they are vehicles of learning, despite not necessarily believing in deities.
Many social conservatives report being unwilling to slap their fathers, even with permission, as part of a play.
The classic trolley problem implies that many people’s moral intuitions hinge on the act of murder being wrong, rather than the effect that the death has on the values, feelings, and preferences being morally wrong.
Of course, if you are a moral realist, you can just say that these people’s intuitions are “wrong”...but the point is that “feelings, values, and preferences”—in a word, utilitarianism—isn’t the only guiding moral principle that humans care about.
And yes, you could argue that this is all a deity’s preferences...but why did they decide that those were in fact the deity’s preferences? Doesn’t it hint that they might have an underlying feeling of those preferences in themselves, that they would project those wishes on a deity?
Conversationalists will want to preserve ecosystems
No doubt some of them will, but I suspect you meant “conservationists.” And yes, I agree that some of those will assign intrinsic value to “nature” in various forms, or at least claim to, as you describe.
Pro-life advocates say that beings without preferences have rights by virtue of future preferences. Not all of them are religious.
Some of them do, yes. Indeed, I suspect the ones who say that are disproportionately non-religious.
why did they decide that those were in fact the deity’s preferences?
A fine question.
Doesn’t it hint that they might have an underlying feeling of those preferences in themselves, that they would project those wishes on a deity?
Quite a few of them no doubt. Of course, the overwhelming majority of people who would say that burning the Quran or the Bible is inherently immoral would also say that it’s immoral by virtue of the preferences of an entity that, on their view, is in fact capable of having preferences.
And, again, if destroying entity X is wrong because some other entity Y says so, that is not inherent.
I bet I can find you someone who would say that burning the Quran or the Bible is inherently immoral.
We sometimes extend morality to inanimate objects , but only ones that mean something to us, such as works of art and religious artefacts. That isn’t actually inherent because of the “to us” clause, although some people might claim
that it is.
There are some meta-ethics sequences here that explain the arbitrariness of our moral preferences more eloquently , and here is a fun story that tangentially illustrates it
Pebble sorting is a preference. That’s it. I don’t have to believe it is a moral preference or a correct moral preference.
How can you speak of a moral preference being “accurate” or not?
Moral objectivism isn’t obviously wrong, and system 2 isn’t obviously the wrong way to realise moral truths.
IOW, moral subjectivism isn’t obviously true.
NB: Objectivism isn’t universalism.
Moral preferences simply are.
Beliefs simply are. And some are true and some are not. You seem to be assuming the non-existence of anything that
could verify or disprove a moral preference in order to prove more or less the same thing.
We sometimes extend morality to inanimate objects , but only ones that mean something to us, such as works of art and religious artefacts. That isn’t actually inherent because of the “to us” clause, although some people might claim that it is.
I would say that the “to us” clause actually applies to everything, and that nothing is “inherent”, as you put it. Pebble sorting means something to the pebble sorters. Humans mean something to me. The entirity of morality boils down to what is important “to us”?
To me, moral objectivism is obviously wrong and subjectivism is obviously true, and this is embedded in my definition of morality. I’m actually unsure how anyone could think of it in any other coherent way.
I think it’s time to unpack “morality”. I think morality is feelings produced in the human mind about how people aught to act. That is, I think “murder is bad” is in some ways analogous to “Brussels sprouts are gross”. From this definition, it follows that I see moral objectivism as obviously wrong—akin to saying, “no man, Brussels sprouts are objectively, inherently gross! In the same way that the sky is objectively blue! / In the same way that tautologies are true!” (Actually, replace blue with the appropriate wavelengths to avoid arguments about perception)
What do you think “morality” is, and where do you suppose it comes from?
I think morality is behaving so as to take into account the values and preferences of others as well as ones own. You can succed or fail in that, hence “accurate”.
Morality may manifest in the form of a feeling for many people, but not for everybody and not all feelings are equal. So I don’t think that is inherent, or definitional.
I don’t think the sprout analogy works, because your feeling that you don’t like sprouts doesn’t seriously affect others, but the psychoaths fondndess for murder does.
The feelings that are relevant to morality are the empathic ones, not personal preferences. That is a clue that
morality is about behaving so as to take into account the values and preferences of others as well as ones own.
if you think morlaity is the same as a personal preference...what makes it morality? Why don’t we just have one word and one way of thinking?
what makes it morality? Why don’t we just have one word and one way of thinking?
Because they feel different to us from the inside—for the same reason that we separate “thinking” and “feeling” even though in the grand scheme of things they are both ways to influence behavior.
Mathematical statements aren’t empirical facts eitherl but convergence is uncontroversial there.
In Math, empirical evidence is replaced by axioms. In Science, the axioms are the empirical evidence.
The point is that all rational agents will converge upon mathematical statements, and will not converge upon moral statements. Do you disagree?
Are you quite sure that morlaity isn’t implicit in the logic of how-a-society-if-entities-wth-varying-prefernces manage-to-rub-along?
I think morality is behaving so as to take into account the values and preferences of others as well as ones own
I’m very, very sure that my morality doesn’t work that way.
Imagine you lived on a world with two major factions, A and B.
A has a population of 999999. B has a population of 1000.
Every individual in A has a very mild preference for horrifically torturing B, and the motivation is sadism and hatred. The torture and slow murder of B is a bonding activity for A, and the shared hatred keeps the society cohesive.
Every individual in B has a strong, strong preference not to be tortured, but it doesn’t even begin to outweigh the collective preferences of A.
From the standpoint of preference utilitarianism, this scenario is analogous to Torture vs. Dust Specks. Preference Utilitarians choose torture, and a good case could be made even under good old human morality to choose torture as the lesser of two evils. This is a problem which I’d give serious weight to choosing torture
Preference utilitarian agents would let A torture B—“shut up and multiply”. However, from the standpoint of my human morality, this scenario is very different from torture vs. dust specks, and I wouldn’t even waste a fraction of a second in deciding what is right in this scenario. Torture for the sake of malice is wrong (to me) and it really doesn’t matter what everyone else’s preferences are—if it’s in my power, I’m not letting A torture B!
Are you quite sure that morlaity isn’t implicit in the logic of how-a-society-if-entities-wth-varying-prefernces manage-to-rub-along?
Morality evolved as a function of how it benefited single alleles, not societies. Under different conditions, it could have evolved differently. You can’t generalize from the way morality works in humans to the way it might work in all possible societies of entities.
Mathematical statements aren’t empirical facts eitherl but convergence is uncontroversial there.
In Math, empirical evidence is replaced by axioms.
The point is that all rational agents will converge upon mathematical statements, and will not converge upon moral statements. Do you disagree?
Agreement isn’t important: arguments are important. You apparently made the argument that convergence on morality isn’t possible because it would require empirically detectable moral objects. I made the counterargument that convergence on morality could work like convergence on mathematical truth. So it
seems that convergence on morlaity could happen, since there is a way it could work.
I think morality is behaving so as to take into account the values and preferences of others as well as ones own
I’m very, very sure that my morality doesn’t work that way. [Argument against utilitarianism].
OK. Utilitarianism sucks. That doens’t mean other objective approaches don’t work—you could be a deontologist. And it doesn’t mean subjectivism does work.
Morality evolved as a function of how it benefited single alleles, not societies. Under different conditions, it could have evolved differently. You can’t generalize from the way morality works in humans to the way it might work in all possible societies of entities
Says who? We can generalise language, maths and physics beyond our instinctive System I understandings. And we have.
I think morality is behaving so as to take into account the values and preferences of others as well as ones own.
is the reason why I said that my morality isn’t preference utilitarian. If “taking into account the values and preferences of others as well as your own”, then preference utilitarianism seems to be the default way to do that.
Alright...so if I’m understanding correctly, you are saying that moral facts exist and people can converge upon them independently, in the same ways that people will converge on mathematical facts. And I’m saying we can’t, and that morality is a preference linked to emotions. Neither of us have really done anything but restate our positions here. My position seems more or less inherent in my definition of morality, and I think you understand my position...but I still don’t understand yours.
Can I have a rudimentary definition of morality, an example of a moral fact, and a process by which two agents can converge upon it?
Can you give me a method of evaluating a moral fact which doesn’t at some point refer to our instincts? Do moral facts necessarily have to conform to our instincts? As in, if I proved a moral fact to you, but your instincts said it was wrong, would you still accept that it was right?
is the reason why I said that my morality isn’t preference utilitarian. If “taking into account the values and preferences of others as well as your own”, then preference utilitarianism seems to be the default way to do that.
For lexicographers, the default is apparently deontology
“conformity to the rules of right conduct”
“Principles concerning the distinction between right and wrong or good and bad behavior.”
etc.
Can I have a rudimentary definition of morality, an example of a moral fact, and a process by which two agents can converge upon it?
1 A means by which communities of entities with preferences act in accordance with all their preferences.
2 Murder is wrong.
3 Since agents do not wish to be murdered, it is in their interests to agree to refrain from murder under an
arrangement in which other agents agree to refrain from removing them.
Can you give me a method of evaluating a moral fact which doesn’t at some point refer to our instincts?
I don’t see why I need to, Utilitarianism and ontology take preferences and intuitions into account. Your argument against utilitarinism that it comes to to conclusions which go against your instincts. That isn’t just an assumption that morality has to something to do with instincts, it is a further assumption that your instincts trump all further constderations It is an assumption of subjectivism.
You are saying objectivism is false because subjectivism is true.
If utilitarianism worked, it would take intuitions and preferences into account, and arrive at some arrangement that
minimises the number of people who don’t get their instincts or preferences satisfied. Some people have to lose You have decided that is unaccpetable because you have decided that you must not lose. But utilitariansim
still works in the sense that a set of subjective prefefernces can be treated as objective facts, and aggregated together. There is nothing to stop different utilitarians (of the same variety) converging on a decision. U-ism
“works” in that sense.You objection is not that convergence is not possible, but that what is converged upon is not moral, because your instincts say not.
But you don’t have any argument beyond an assumption that morality just is what your instincts say. The other side
of the argument doesn’t have to deny the instinctive or subjective aspect of morality, it only needs to deny that your instincts are supreme. And it can argue that since morality is about the regulation of conduct amongst groups, the very notion of subjective morality is incoherent (parallel: language is all about communication, so a language that is only understood by one person is a paradox).
As in, if I proved a moral fact to you, but your instincts said it was wrong, would you still accept that it was right?
Maybe. Almost everybody who has had their mind changed about sexual conduct had overridden an instinct.
3 Since agents do not wish to be murdered, it is in their interests to agree to refrain from murder under an arrangement in which other agents agree to refrain from removing them.
So there are several things I don’t like about this..
0) It’s not in their interests to play the cooperative strategy if they are more powerful, since the other agent can’t remove them.
1) It’s not a given that all agents do not wish to be murdered. It’s only luck that we wish not to die. Sentient beings could just as easily have come out of insects who allow themselves to be eaten by mates, or by their offspring.
2) So you sidestep this, and say that this only applies to beings that wish to be murdered. Well now, this is utilitarianism. You’d essentially be saying that all agents want their preferences fulfilled, therefore we should all agree to fulfill each others preferences.
You have decided that is unaccpetable because you have decided that you must not lose.
Essentially yes. But to rephrase: I know that the behavior of all agents (including myself) will work to bring about the agent’s preferences to the best of the agent’s ability, and this is true by definition of what a “preference” is.
Maybe. Almost everybody who has had their mind changed about sexual conduct had overridden an instinct.
I’m not sure I follow what you mean by this. My ideas about sexual conduct are in line with my instincts. A highly religious person’s ideas about sexual conduct are in line with the instincts that society drilled into them. If I converted that person into sex-positivism, they would shed the societal conditioning and their morality and feelings would change. Who is not in alignment with their instincts?
(Instincts here means feelings with no rational basis, rather than genetically programmed or reflexive behaviors)
0) It’s not in their interests to play the cooperative strategy if they are more powerful, since the other agent can’t remove them.
I am not sure what the argument is here. The objectivist claim is not that every entity actually will be moral in practice, and it’t not the claim that every agent will be interested in settling moral question: it’s just the claim that agents who are interested in settling moral questions, and have the same set of facts available (ie live in the same society) will be able to converge. (Which is as objective as anything else.The uncontentious claim that mathematics is objective doens’t imply that everyone is a mathematician, or knows all mathematical truths).
It’s not a given that all agents do not wish to be murdered. It’s only luck that we wish not to die. Sentient beings could just as easily have come out of insects who allow themselves to be eaten by mates, or by their offspring.
I have described morality as an arrangement within a society. Alien societies might have different morality to go with their diffrent biology. That is not in favour of subjectivism, because subjectivism requires morality to vary with personal preference, not objective facts about biology. Objectivism does not mean universalism. It means agents, given the same facts, and the willingness to draw moral conclusions from them, will converge. It doens’t mean the facts never vary. if they do, so will the conclusions
You’d essentially be saying that all agents want their preferences fulfilled, therefore we should all agree to fulfill each others preferences.
All agents want their preferences fulfilled, and what “should” means is being in accordance some arrangement for resolving the resulting conflicts, whether utilitarian, deontological, or something else.
My ideas about sexual conduct are in line with my instincts. A highly religious person’s ideas about sexual conduct are in line with the instincts that society drilled into them. If I converted that person into sex-positivism, they would shed the societal conditioning and their morality and feelings would change. Who is not in alignment with their instincts?
The convertee. In my expererience, people are generally converted by arguments...reasoning...system 2.
So when people are converted, they go from Instinct to Reason.
But perhaps you know of some process by which subjective feelings are transferred directly, without the involvement of system 2.
it’s just the claim that agents who are interested in settling moral questions, and have the same set of facts available (ie live in the same society) will be able to converge.
It means agents, given the same facts, and the willingness to draw moral conclusions from them, will converge. It doens’t mean the facts never vary. if they do, so will the conclusions
But don’t you see what you’re doing here? You are defining a set of moral claims M, and then saying that any agents who are interested in M will converge on M!
The qualifier “agents who are interested in moral questions” restricts the set of agents to those who already agree with you about what morality is. Obviously, if we all start from the same moral axioms, we’ll converge onto the same moral postulates—the point is that the moral axioms are arbitrarily set by the user’s preferences.
All agents want their preferences fulfilled, and what “should” means is being in accordance some arrangement for resolving the resulting conflicts, whether utilitarian, deontological, or something else.
Wait, so you are defining morality is defined as a system of conflict resolution between agents? I actually do like that definition...even though it doesn’t imply convergence.
Then Utilitarianism is the solution that all agents should maximize preferences, deontological is the solution that there exist a set of rules to follow when arbitrating conflict, etc.
Counterexample—Imagine a person who isn’t religious, who also believes incest between consenting adults is wrong (even for old infertile people, even if no one else gets to know about it). There is no conflict between the two agents involved—would you say that this person is not exhibiting a moral preference, but something else entirely?
But perhaps you know of some process by which subjective feelings are transferred directly, without the involvement of system 2.
The vast majority of people are not convinced by argument, but by life experience. For most people, all the moral rhetoric in the world isn’t as effective as a picture of two gay men crying with happiness as they get married.
That’s besides the point, though—you are right that it is possible (though difficult) to alter someone’s moral stance through argument alone. However, “System 1” and “System 2“ share a brain. You can influence “system 1” via “system 2”—reasoning can effect feelings, and vice versa. I can use logical arguments to change someone’s feelings on moral issues. That doesn’t change the fact that the moral attitude stems from the feelings.
If you can establish a shared set of “moral axioms” with someone, you can convince them of the rightness or wrongness of something with logic alone. This might make it seem like any two agents can converge on morality—but just because most humans have certain moral preferences hardwired into them doesn’t mean every agent has the same set of preferences. I have some moral axioms, you have some moral axioms, and we can use shared moral axioms to convince each other of things… but we won’t be able to convince any agent which has moral axioms that do not match with ours.
But don’t you see what you’re doing here? You are defining a set of moral claims M, and then saying that any agents who are interested in M will converge on M!
I haven’t defined a set of moral claims. You asked me for an example of one claim. I can argue the point with specifying any moral conclusions. The facts I mentioned as the input to the process are not moral per se.
The qualifier “agents who are interested in moral questions” restricts the set of agents to those who already agree with you about what morality is.
In a sense, yes. But only in the sense that “agents who are interested in mathematical questions” restricts the set of agents who are interested in “mathematics” as I understand it. On the other had, nothing is implied about the set of object level claims moral philosophers would converge on.
Obviously, if we all start from the same moral axioms, we’ll converge onto the same moral postulates—the point is that the moral axioms are arbitrarily set by the user’s preferences.
I don’t have to accept that, because I am not use a subjective criterion for “morailty”. If you have a preference
for Tutti Frutti, that is not a moral preference, because it does not affect anybody else. The definition of morality I am using is not based on any personal preference of mine, it’s based on a recognition that morality has a job to do.
Wait, so you are defining morality is defined as a system of conflict resolution between agents? I actually do like that definition...even though it doesn’t imply convergence.
If no convergence takes place, how can you have an implementable system? People are either imprisoned or not, they cannot be imprisoned for some agents but not for others.
Counterexample—Imagine a person who isn’t religious, who also believes incest between consenting adults is wrong (even for old infertile people, even if no one else gets to know about it). There is no conflict between the two agents involved—would you say that this person is not exhibiting a moral preference, but something else entirely?
You are tacilty assuming that no action will be taken on the basis of feelings of wrongness, that nobody ever campaigns to ban things they don’t like.
That doesn’t change the fact that the moral attitude stems from the feelings.
If system 1 was influenced by system 2 , then what stems from system 1 stemmed from system 2, and so on. You are drawing an arbitrary line.
If you can establish a shared set of “moral axioms” with someone, you can convince them of the rightness or wrongness of something with logic alone.
If moral axioms are completely separate from everything else, then you would need to change their axioms. If
they are not, then not. For instance, you can argue that some moral attitudes someone has are inconsistent with others. Consistency is not a purely moral criterion.
I have some moral axioms, you have some moral axioms, and we can use shared moral axioms to convince each other of things… but we won’t be able to convince any agent which has moral axioms that do not match with ours.
If “moral axioms” overlap with rational axioms, and if moral axioms are constrained by the functional role of morality, there is plenty of scope for rational agents to converge.
Does it follow, then, that rational agents will always be “moral”? Does it mean that the most rational choice for maximizing any set of preferences, is also in line with “morality”?
That would put morality into decision theory, which would be kind of nice.
But I can’t think how an agent whose utility function simply read “Commit Murder” could possibly make a choice that was both moral (the way morality is traditionally defined) and rational.
But I can’t think how an agent whose utility function simply read “Commit Murder” could possibly make a choice that was both moral (the way morality is traditionally defined) and rational.
People who believe in the Convergence thesis tend not to believe in Orthogonality thesis. They tend to use the traditional definition of rationality
“In its primary sense, rationality is a normative concept that philosophers have generally tried to characterize in such a way that, for any action, belief, or desire, if it is rational we ought to choose it” (WP)
in detail, they tend to see rationalists as having a preference for objectivity, consistency and non-arbitrariness, including in their preference. Thus they would tend to see Clippies as having highly rational though and highly irrational (because arbitrary) desires. Likewise they would see a murderer who does not want to be murdered as inconsistent, and therefore irrational.
Another way of looking at it is that they would see highly intelligent and rational agents as climbing the Maslow hierarchy.
Does it follow, then, that rational agents will always be “moral”? Does it mean that the most rational choice for maximizing any set of preferences, is also in line with “morality”?
Depends how rational. For a Convergence theorist, an ideal, supremely rational agent will have rational desires and preferences. A less ideal one might fall short as a result of being non-ideal.
Thanks for that term. This makes things clearer. Based on what you are arguing, does that make you a convergence theorist then? (Or at least, you seem to be defending convergence theory here, even if you don’t wholeheartedly accept it)
I dunno...I just find the orthogonality thesis as intuitively obvious, and I’m having real trouble grasping what exactly the thought process that leads one to become a convergence theorist might be. I’m hoping you can show me what that thought process is.
hus to deny the Orthogonality thesis is to assert that there is a goal system G, such that...There cannot exist any efficient real-world algorithm with goal G.
Now, I agree that there exist some G such that this is the case, but I don’t think this set would have anything to do with morality as humans understand it.
You seem to be making the argument that one of the characteristics that would automatically qualify something as a candidate for G is immorality.
This makes no intuitive sense. Why couldn’t you make an efficient real world algorithm to destroy all life forms? It seems like, in the absence of some serious mathematical arguments to the contrary, we aught to dismiss claims that efficient real world algorithms for murder are impossible offhand.
Thanks for that term. This makes things clearer. Based on what you are arguing, does that make you a convergence theorist then?
Why is that important?
I dunno...I just find the orthogonality thesis as intuitively obvious, and I’m having real trouble grasping what exactly the thought process that leads one to become a convergence theorist might be. I’m hoping you can show me what that thought process is.
I think I can see where the intuitive appeal comes from, and I think I can see where the errors are too.
“Thus to deny the Orthogonality thesis is to assert that there is a goal system G, such that...There cannot exist any efficient real-world algorithm with goal G.”
I can see why that is appealing, but it is not equivalent to the claim that any intelligent and rational entity could have any goal. Of course you can write a dumb algorithm to efficiently make paperclips, just as you can build a dumb machine that makes paperclips. And of course an AI could....technically …design and/or implement such an algorithm, But it doesn’t follow that an AGI would do so. (Which is two propositions: it doesn’t follow that an AI could be persuaded to adopt such a goal, and it doesn’t follow that such a goal could be programmed in ab initio and remain stable).
The Convergentist would want to claim:
“To assert the Orthogonality Thesis is to assert that no matter how intelligent and rational an agent, no matter the breadth of its understanding, no matter the strength of its commitment to objectivity, no matter its abilities to self-reflect and update, it would still never realise that making huge numbers of paperclips is arbitrary and unworthy of its abilities”
The orthogonality claim only has bite against Convergence/Moral Realism if it relates to all or most or typical
rational intelligent agents, because that is how moral realists define their claim: they claim that ideal
rational agents of a typical kind will converge, or that most rational-enough and intelligent-enough agents will converge. You might be able to build a (genuinely intelligent, reflecting and updating) Clippy, but that wouldn’t prove anything. The natural existence of sociopaths doesn’t disprove MR because they are statiscally rare, and their typicality is in doubt. You can’t prove anything about morality by genetically engineering a sociopath.
As an argument against MR/C, Orthogonality has to claim that the typical, statistically
common kind of agent could have arbitrary goals, and that the evidence of convergence amongst humans
is explained by specific cultural or genetic features, not by rationality in general.
ETA:
Ben: [The Orthogonality Thesis] may be true, but who cares about possibility “in principle”? The question is whether any level of intelligence is PLAUSIBLY LIKELY to be combined with more or less any final goal in practice. And I really doubt it. I guess I could posit the alternative: Interdependency Thesis: Intelligence and final goals are in practice highly and subtly interdependent.
If we don’t understand the relationship between instrumental intelligence and goals, Clippies will seem possible—in the way that p-zombies do if you don’t understand the relationship between matter and consciousness.
Because I want to be sure that I’m understanding what the claim you’re making is.
The Convergentist would want to claim:
“To assert the Orthogonality Thesis is to assert that no matter how intelligent and rational an agent, no matter the breadth of its understanding, no matter the strength of its commitment to objectivity, no matter its abilities to self-reflect and update, it would still never realise that making huge numbers of paperclips is arbitrary and unworthy of its abilities”
Okay...so I agree with the Convergence theorist on what the implications of the Orthogonality Thesis are, and I still think the Orthogonality Thesis is true.
if it relates to all or most or typical rational intelligent agents, because that is how moral realists define their claim
Hold on now...that makes the claim completely different than what I thought we were talking about up till now. I thought we were talking about whether or not all rational agents would be in agreement about what morality is, independent of specifically human preferences?
We can have the other discussion too...but not before settling whether or not the Orthogonality Thesis is in fact true “in principle”. Remember, we originally started this discussion with my claim that morality is feelings/preference, as opposed to something you could figure out (i.e. something embedded into logic/game theory or the universe itself.) We weren’t originally talking about rational agents to shed light on evolution or plausible AI...we brought them in as hypothetical agents who converge upon the correct answer to any answerable question, to explore whether or not “what is good” is independent from “what do humans think is good”.
I thought we were talking about whether morality was something that could be discovered objectively.
I said:
Morality comes from the “heart”. It’s made of feelings.
Then you said:
People use feelings/System1 to do morality. That doesn’t make it an oracle. Thinking might be more accurate.
Then I said
Accurate? How can you speak of a moral preference being “accurate” or not? Moral preferences simply are. Accurate? How can you speak of a moral preference being “accurate” or not? Moral preferences simply are.
You disagreed, and said
Moral objectivism isn’t obviously wrong,
To which I countered
all rational agents will converge upon mathematical statements, and will not converge upon moral statements.
You disagreed:
morality could work like convergence on mathematical truth.
Which is why
I thought we were talking about whether or not all rational agents would be in agreement about what morality is, independent of specifically human preferences?
Hence
if it relates to all or most or typical rational intelligent agents
doesn’t make any sense in our discussion. All rational agents converge on mathematical and ontological facts, by definition. My argument was that there is no such thing as a “moral fact: and moral statements can only be discussed when in reference to the psychology of a small set of creatures which includes humans and some other mammals. I argued that moral statements can’t be “discovered” true or false in any ontological or mathematical sense, nor are they deeply embedded into game theory (meaning it is not always in the interest of all rational agents to follow human morality) - even though game theory does explain how we evolved morality given our circumstances.
If you admit that at least one of all possible rational agent doesn’t converge upon morality, you’ve been in agreement with me this entire time—which means we’ve been talking about different things this entire time...so what did you think we were talking about?
All rational agents converge on mathematical and ontological facts, by definition.
Only by a definition whereby “rational” means “ideally rational”. In the ordinary sense of the term, it perfectly possible for someone who is deemed “rational” in a more-or-less, good-enough sense to fail to understand some mathematical truths. The existence of the innumerate does not disprove the objectivity of mathematics, and the existence of sociopaths does not disprove the objectivity of morality.
If you admit that at least one of all possible rational agent doesn’t converge upon morality,
Do you believe that it is possible for a rational agent to fail to understand a mathematical truth? Because that seems rather commonplace to me. Unless you mean ideally rational....
The whole point of invoking an ideal rational agent in the first place was to demonstrate that moral “truths” aren’t like empirical or mathematical truths in that you can’t discover them objectively through philosophy or mathematics (even if you are infinitely smart). Rather, moral “truths” are peculiar to humans.
If you want to illustrate the non-objectivity of morality, then stating that even ideal rational agents won’t converge on them is one of expressing the point, although it helps to state the “ideal” explicitly. However, that is still only the expression of a claim, not the “demonstration” of one.
There are some meta-ethics sequences here that explain the arbitrariness of our moral preferences more
eloquently , and here is a fun story that tangentially illustrates it
I think you’ve misunderstood the meta-ethics sequences, then, or I have, because
Pebble sorting is a preference. That’s it. I don’t have to believe it is a moral preference or a correct moral preference.
is quite similar to Eliezer’s position. Although Juno_Watt may have reached it from another direction.
I read it as a warning about expecting sufficiently rational beings to automatically acquire human morality, in the same way that sufficiently rational beings would automatically acquire knowledge about true statements (science, etc). The lesson is that preferences (morality, etc) is different from fact.
If you want to know Eliezer’s views, he spells them out explicitly here—although I think the person most famous for this view is Nietzsche (not that he’s the first to have held this view).
To me, “No universally compelling arguments” means this—two rational agents will converge upon factual statements, but they need not converge upon preferences (moral or otherwise) because moral statements aren’t “facts”.
It really doesn’t matter if you define the pebble sorting as a “moral” preference or a plain old preference.The point is, that humans have a morality module—but that module is in the brain and not a feature which is implicit in logical structures, nor is it a feature implicit in the universe itself.
I read it as a warning about expecting sufficiently rational beings to automatically acquire human morality, in the same way that sufficiently rational beings would automatically acquire knowledge about true statements (science, etc). The lesson is that preferences (morality, etc) is different from fact.
I agree that is what it is trying to say, but...as you made illustrate above..it only appears to work if the reader is willing to bel fuzzy about the difference between preference and moral preference.
If you want to know Eliezer’s views, he spells them out explicitly
For some value of “explicit”. He doesn’t even restrict the range of agents to rational agents, and no-one
expects irrationali agents to agree with each other, or rational ones.
To me, “No universally compelling arguments” means this—two rational agents will converge upon factual statements, but they need not converge upon preferences (moral or otherwise) because moral statements aren’t “facts”.
Mathematical statements aren’t empirical facts eitherl but convergence is uncontroversial there.
It really doesn’t matter if you define the pebble sorting as a “moral” preference or a plain old preference.The point is, that humans have a morality module—but that module is in the brain and not a feature which is implicit in logical structures, nor is it a feature implicit in the universe itself.
Are you quite sure that morlaity isn’t implicit in the logic of how-a-society-if-entities-wth-varying-prefernces manage-to-rub-along ?
For some people, probably. Let’s take a step back.
Morality comes from the “heart”. It’s made of feelings. Utilitarianism (and much of what falls under moral philosophy) is one of many attempt to make a consistent set of rules to describe inconsistent feelings. The purpose of making a consistent set of rules is 1) to convince others of the morality of an action and 2) because we morally feel aversion to hypocrisy and crave moral consistency.
Keeping those aims in mind, drawing the line across all humans, sentient beings, etc has the following benefits:
1) The creators might feel that the equation describes the way they feel better when they factor in all humans. They might hold it as a deontological standard to care about all humans, or they might feel a sense of fairness, or they might have empathy for everyone, etc.
2) Drawing the line across all humans allows you to use the utilitarian standard to negotiate compromises with any arbitrary human you come across. Many humans, having the feelings described in [1], will instinctively accept utilitarianism as a valid way to think about things.
There are plenty of things that are problematic here, but that is why utilitarianism defaults to include the whole human race. As with all things moral, that’s just an arbitrary choice on our part, and we could easily have done it a different way. We can restrict it to a smaller subset of humans, we can broaden it to non-human things which seem agent-like enough to be worth describing with a utility function, etc. Many utilitarians include animals, for example.
People use feelings/System1 to do morality. That doesn’t make it an oracle. Thinking might be more accurate.
If you don’t know how to solve a problem, you guess. But that doens’t mean anything goes. Would anyone include rocks in the Circle? Probably not, since they don’t have feelings, values, or preferences. So there seem to be some constraints.
Accurate? How can you speak of a moral preference being “accurate” or not? Moral preferences simply are. There are some meta-ethics sequences here that explain the arbitrariness of our moral preferences more eloquently , and here is a fun story that tangentially illustrates it
I bet I can find you someone who would say that burning the Quran or the Bible is inherently immoral.
Quite a few of them no doubt. Of course, the overwhelming majority of people who would say that burning the Quran or the Bible is inherently immoral would also say that it’s immoral by virtue of the preferences of an entity that, on their view, is in fact capable of having preferences.
Of course, I’m sure I could find someone who would say rocks have feelings, values, and preferences.
I don’t think this is an accurate formulation of the general religious attitude towards morality.
I agree. Do you also think it’s a false statement?
Let’s just say the expression “it’s immoral by virtue of the preferences of an entity” is not actually a good ‘translation’ of the phrase they’d use.
Um… well, I’m not really sure what to do with that statement, but I’m happy to leave the topic there if you prefer.
Ok, maybe I misunderstood your question in the grandparent. Which statement was it referring to?
“the overwhelming majority of people who would say that burning the Quran or the Bible is inherently immoral would also say that it’s immoral by virtue of the preferences of an entity that, on their view, is in fact capable of having preferences.”
They’d phrase it in terms of sacredness, which isn’t quite the same thing, e.g., how would you apply your argument to flag burning?
Fair enough.
Conversationalists will want to preserve ecosystems, even where those ecosystems are already well studied by science, even when the ecosystem contains no sentient beings (plants, fungi, microbes), even when destroying the ecosystem has many advantages for humans, because they think the ecosystem is intrinsically valuable independently of the effect on beings with feelings, values, and preferences.
Some looser examples...
Pro-life advocates say that beings without preferences have rights by virtue of future preferences. Not all of them are religious.
Hindus treat books (all books in general) with reverence because they are vehicles of learning, despite not necessarily believing in deities.
Many social conservatives report being unwilling to slap their fathers, even with permission, as part of a play.
The classic trolley problem implies that many people’s moral intuitions hinge on the act of murder being wrong, rather than the effect that the death has on the values, feelings, and preferences being morally wrong.
Of course, if you are a moral realist, you can just say that these people’s intuitions are “wrong”...but the point is that “feelings, values, and preferences”—in a word, utilitarianism—isn’t the only guiding moral principle that humans care about.
And yes, you could argue that this is all a deity’s preferences...but why did they decide that those were in fact the deity’s preferences? Doesn’t it hint that they might have an underlying feeling of those preferences in themselves, that they would project those wishes on a deity?
No doubt some of them will, but I suspect you meant “conservationists.” And yes, I agree that some of those will assign intrinsic value to “nature” in various forms, or at least claim to, as you describe.
Some of them do, yes. Indeed, I suspect the ones who say that are disproportionately non-religious.
A fine question.
That’s one possibility, yes.
And, again, if destroying entity X is wrong because some other entity Y says so, that is not inherent.
Indeed. Do you mean to say that you don’t expect it to be said, or merely that those saying it are confused?
The latter.
We sometimes extend morality to inanimate objects , but only ones that mean something to us, such as works of art and religious artefacts. That isn’t actually inherent because of the “to us” clause, although some people might claim that it is.
Pebble sorting is a preference. That’s it. I don’t have to believe it is a moral preference or a correct moral preference.
Moral objectivism isn’t obviously wrong, and system 2 isn’t obviously the wrong way to realise moral truths. IOW, moral subjectivism isn’t obviously true.
NB: Objectivism isn’t universalism.
Beliefs simply are. And some are true and some are not. You seem to be assuming the non-existence of anything that could verify or disprove a moral preference in order to prove more or less the same thing.
I would say that the “to us” clause actually applies to everything, and that nothing is “inherent”, as you put it. Pebble sorting means something to the pebble sorters. Humans mean something to me. The entirity of morality boils down to what is important “to us”?
To me, moral objectivism is obviously wrong and subjectivism is obviously true, and this is embedded in my definition of morality. I’m actually unsure how anyone could think of it in any other coherent way.
I think it’s time to unpack “morality”. I think morality is feelings produced in the human mind about how people aught to act. That is, I think “murder is bad” is in some ways analogous to “Brussels sprouts are gross”. From this definition, it follows that I see moral objectivism as obviously wrong—akin to saying, “no man, Brussels sprouts are objectively, inherently gross! In the same way that the sky is objectively blue! / In the same way that tautologies are true!” (Actually, replace blue with the appropriate wavelengths to avoid arguments about perception)
What do you think “morality” is, and where do you suppose it comes from?
I think morality is behaving so as to take into account the values and preferences of others as well as ones own. You can succed or fail in that, hence “accurate”.
Morality may manifest in the form of a feeling for many people, but not for everybody and not all feelings are equal. So I don’t think that is inherent, or definitional.
I don’t think the sprout analogy works, because your feeling that you don’t like sprouts doesn’t seriously affect others, but the psychoaths fondndess for murder does.
The feelings that are relevant to morality are the empathic ones, not personal preferences. That is a clue that morality is about behaving so as to take into account the values and preferences of others as well as ones own.
if you think morlaity is the same as a personal preference...what makes it morality? Why don’t we just have one word and one way of thinking?
Because they feel different to us from the inside—for the same reason that we separate “thinking” and “feeling” even though in the grand scheme of things they are both ways to influence behavior.
In Math, empirical evidence is replaced by axioms. In Science, the axioms are the empirical evidence.
The point is that all rational agents will converge upon mathematical statements, and will not converge upon moral statements. Do you disagree?
I’m very, very sure that my morality doesn’t work that way.
Imagine you lived on a world with two major factions, A and B.
A has a population of 999999. B has a population of 1000.
Every individual in A has a very mild preference for horrifically torturing B, and the motivation is sadism and hatred. The torture and slow murder of B is a bonding activity for A, and the shared hatred keeps the society cohesive.
Every individual in B has a strong, strong preference not to be tortured, but it doesn’t even begin to outweigh the collective preferences of A.
From the standpoint of preference utilitarianism, this scenario is analogous to Torture vs. Dust Specks. Preference Utilitarians choose torture, and a good case could be made even under good old human morality to choose torture as the lesser of two evils. This is a problem which I’d give serious weight to choosing torture
Preference utilitarian agents would let A torture B—“shut up and multiply”. However, from the standpoint of my human morality, this scenario is very different from torture vs. dust specks, and I wouldn’t even waste a fraction of a second in deciding what is right in this scenario. Torture for the sake of malice is wrong (to me) and it really doesn’t matter what everyone else’s preferences are—if it’s in my power, I’m not letting A torture B!
Morality evolved as a function of how it benefited single alleles, not societies. Under different conditions, it could have evolved differently. You can’t generalize from the way morality works in humans to the way it might work in all possible societies of entities.
Agreement isn’t important: arguments are important. You apparently made the argument that convergence on morality isn’t possible because it would require empirically detectable moral objects. I made the counterargument that convergence on morality could work like convergence on mathematical truth. So it seems that convergence on morlaity could happen, since there is a way it could work.
OK. Utilitarianism sucks. That doens’t mean other objective approaches don’t work—you could be a deontologist. And it doesn’t mean subjectivism does work.
Says who? We can generalise language, maths and physics beyond our instinctive System I understandings. And we have.
is the reason why I said that my morality isn’t preference utilitarian. If “taking into account the values and preferences of others as well as your own”, then preference utilitarianism seems to be the default way to do that.
Alright...so if I’m understanding correctly, you are saying that moral facts exist and people can converge upon them independently, in the same ways that people will converge on mathematical facts. And I’m saying we can’t, and that morality is a preference linked to emotions. Neither of us have really done anything but restate our positions here. My position seems more or less inherent in my definition of morality, and I think you understand my position...but I still don’t understand yours.
Can I have a rudimentary definition of morality, an example of a moral fact, and a process by which two agents can converge upon it?
Can you give me a method of evaluating a moral fact which doesn’t at some point refer to our instincts? Do moral facts necessarily have to conform to our instincts? As in, if I proved a moral fact to you, but your instincts said it was wrong, would you still accept that it was right?
For lexicographers, the default is apparently deontology
“conformity to the rules of right conduct”
“Principles concerning the distinction between right and wrong or good and bad behavior.”
etc.
1 A means by which communities of entities with preferences act in accordance with all their preferences.
2 Murder is wrong.
3 Since agents do not wish to be murdered, it is in their interests to agree to refrain from murder under an arrangement in which other agents agree to refrain from removing them.
I don’t see why I need to, Utilitarianism and ontology take preferences and intuitions into account. Your argument against utilitarinism that it comes to to conclusions which go against your instincts. That isn’t just an assumption that morality has to something to do with instincts, it is a further assumption that your instincts trump all further constderations It is an assumption of subjectivism.
You are saying objectivism is false because subjectivism is true. If utilitarianism worked, it would take intuitions and preferences into account, and arrive at some arrangement that minimises the number of people who don’t get their instincts or preferences satisfied. Some people have to lose You have decided that is unaccpetable because you have decided that you must not lose. But utilitariansim still works in the sense that a set of subjective prefefernces can be treated as objective facts, and aggregated together. There is nothing to stop different utilitarians (of the same variety) converging on a decision. U-ism “works” in that sense.You objection is not that convergence is not possible, but that what is converged upon is not moral, because your instincts say not.
But you don’t have any argument beyond an assumption that morality just is what your instincts say. The other side of the argument doesn’t have to deny the instinctive or subjective aspect of morality, it only needs to deny that your instincts are supreme. And it can argue that since morality is about the regulation of conduct amongst groups, the very notion of subjective morality is incoherent (parallel: language is all about communication, so a language that is only understood by one person is a paradox).
Maybe. Almost everybody who has had their mind changed about sexual conduct had overridden an instinct.
So there are several things I don’t like about this..
0) It’s not in their interests to play the cooperative strategy if they are more powerful, since the other agent can’t remove them.
1) It’s not a given that all agents do not wish to be murdered. It’s only luck that we wish not to die. Sentient beings could just as easily have come out of insects who allow themselves to be eaten by mates, or by their offspring.
2) So you sidestep this, and say that this only applies to beings that wish to be murdered. Well now, this is utilitarianism. You’d essentially be saying that all agents want their preferences fulfilled, therefore we should all agree to fulfill each others preferences.
Essentially yes. But to rephrase: I know that the behavior of all agents (including myself) will work to bring about the agent’s preferences to the best of the agent’s ability, and this is true by definition of what a “preference” is.
I’m not sure I follow what you mean by this. My ideas about sexual conduct are in line with my instincts. A highly religious person’s ideas about sexual conduct are in line with the instincts that society drilled into them. If I converted that person into sex-positivism, they would shed the societal conditioning and their morality and feelings would change. Who is not in alignment with their instincts?
(Instincts here means feelings with no rational basis, rather than genetically programmed or reflexive behaviors)
I am not sure what the argument is here. The objectivist claim is not that every entity actually will be moral in practice, and it’t not the claim that every agent will be interested in settling moral question: it’s just the claim that agents who are interested in settling moral questions, and have the same set of facts available (ie live in the same society) will be able to converge. (Which is as objective as anything else.The uncontentious claim that mathematics is objective doens’t imply that everyone is a mathematician, or knows all mathematical truths).
I have described morality as an arrangement within a society. Alien societies might have different morality to go with their diffrent biology. That is not in favour of subjectivism, because subjectivism requires morality to vary with personal preference, not objective facts about biology. Objectivism does not mean universalism. It means agents, given the same facts, and the willingness to draw moral conclusions from them, will converge. It doens’t mean the facts never vary. if they do, so will the conclusions
All agents want their preferences fulfilled, and what “should” means is being in accordance some arrangement for resolving the resulting conflicts, whether utilitarian, deontological, or something else.
The convertee. In my expererience, people are generally converted by arguments...reasoning...system 2. So when people are converted, they go from Instinct to Reason. But perhaps you know of some process by which subjective feelings are transferred directly, without the involvement of system 2.
But don’t you see what you’re doing here? You are defining a set of moral claims M, and then saying that any agents who are interested in M will converge on M!
The qualifier “agents who are interested in moral questions” restricts the set of agents to those who already agree with you about what morality is. Obviously, if we all start from the same moral axioms, we’ll converge onto the same moral postulates—the point is that the moral axioms are arbitrarily set by the user’s preferences.
Wait, so you are defining morality is defined as a system of conflict resolution between agents? I actually do like that definition...even though it doesn’t imply convergence.
Then Utilitarianism is the solution that all agents should maximize preferences, deontological is the solution that there exist a set of rules to follow when arbitrating conflict, etc.
Counterexample—Imagine a person who isn’t religious, who also believes incest between consenting adults is wrong (even for old infertile people, even if no one else gets to know about it). There is no conflict between the two agents involved—would you say that this person is not exhibiting a moral preference, but something else entirely?
The vast majority of people are not convinced by argument, but by life experience. For most people, all the moral rhetoric in the world isn’t as effective as a picture of two gay men crying with happiness as they get married.
That’s besides the point, though—you are right that it is possible (though difficult) to alter someone’s moral stance through argument alone. However, “System 1” and “System 2“ share a brain. You can influence “system 1” via “system 2”—reasoning can effect feelings, and vice versa. I can use logical arguments to change someone’s feelings on moral issues. That doesn’t change the fact that the moral attitude stems from the feelings.
If you can establish a shared set of “moral axioms” with someone, you can convince them of the rightness or wrongness of something with logic alone. This might make it seem like any two agents can converge on morality—but just because most humans have certain moral preferences hardwired into them doesn’t mean every agent has the same set of preferences. I have some moral axioms, you have some moral axioms, and we can use shared moral axioms to convince each other of things… but we won’t be able to convince any agent which has moral axioms that do not match with ours.
I haven’t defined a set of moral claims. You asked me for an example of one claim. I can argue the point with specifying any moral conclusions. The facts I mentioned as the input to the process are not moral per se.
In a sense, yes. But only in the sense that “agents who are interested in mathematical questions” restricts the set of agents who are interested in “mathematics” as I understand it. On the other had, nothing is implied about the set of object level claims moral philosophers would converge on.
I don’t have to accept that, because I am not use a subjective criterion for “morailty”. If you have a preference for Tutti Frutti, that is not a moral preference, because it does not affect anybody else. The definition of morality I am using is not based on any personal preference of mine, it’s based on a recognition that morality has a job to do.
If no convergence takes place, how can you have an implementable system? People are either imprisoned or not, they cannot be imprisoned for some agents but not for others.
You are tacilty assuming that no action will be taken on the basis of feelings of wrongness, that nobody ever campaigns to ban things they don’t like.
If system 1 was influenced by system 2 , then what stems from system 1 stemmed from system 2, and so on. You are drawing an arbitrary line.
If moral axioms are completely separate from everything else, then you would need to change their axioms. If they are not, then not. For instance, you can argue that some moral attitudes someone has are inconsistent with others. Consistency is not a purely moral criterion.
If “moral axioms” overlap with rational axioms, and if moral axioms are constrained by the functional role of morality, there is plenty of scope for rational agents to converge.
Does it follow, then, that rational agents will always be “moral”? Does it mean that the most rational choice for maximizing any set of preferences, is also in line with “morality”?
That would put morality into decision theory, which would be kind of nice.
But I can’t think how an agent whose utility function simply read “Commit Murder” could possibly make a choice that was both moral (the way morality is traditionally defined) and rational.
People who believe in the Convergence thesis tend not to believe in Orthogonality thesis. They tend to use the traditional definition of rationality
“In its primary sense, rationality is a normative concept that philosophers have generally tried to characterize in such a way that, for any action, belief, or desire, if it is rational we ought to choose it” (WP)
in detail, they tend to see rationalists as having a preference for objectivity, consistency and non-arbitrariness, including in their preference. Thus they would tend to see Clippies as having highly rational though and highly irrational (because arbitrary) desires. Likewise they would see a murderer who does not want to be murdered as inconsistent, and therefore irrational.
Another way of looking at it is that they would see highly intelligent and rational agents as climbing the Maslow hierarchy.
Depends how rational. For a Convergence theorist, an ideal, supremely rational agent will have rational desires and preferences. A less ideal one might fall short as a result of being non-ideal.
Thanks for that term. This makes things clearer. Based on what you are arguing, does that make you a convergence theorist then? (Or at least, you seem to be defending convergence theory here, even if you don’t wholeheartedly accept it)
I dunno...I just find the orthogonality thesis as intuitively obvious, and I’m having real trouble grasping what exactly the thought process that leads one to become a convergence theorist might be. I’m hoping you can show me what that thought process is.
The page even says it:
Now, I agree that there exist some G such that this is the case, but I don’t think this set would have anything to do with morality as humans understand it.
You seem to be making the argument that one of the characteristics that would automatically qualify something as a candidate for G is immorality.
This makes no intuitive sense. Why couldn’t you make an efficient real world algorithm to destroy all life forms? It seems like, in the absence of some serious mathematical arguments to the contrary, we aught to dismiss claims that efficient real world algorithms for murder are impossible offhand.
Why is that important?
I think I can see where the intuitive appeal comes from, and I think I can see where the errors are too.
I can see why that is appealing, but it is not equivalent to the claim that any intelligent and rational entity could have any goal. Of course you can write a dumb algorithm to efficiently make paperclips, just as you can build a dumb machine that makes paperclips. And of course an AI could....technically …design and/or implement such an algorithm, But it doesn’t follow that an AGI would do so. (Which is two propositions: it doesn’t follow that an AI could be persuaded to adopt such a goal, and it doesn’t follow that such a goal could be programmed in ab initio and remain stable).
The Convergentist would want to claim:
“To assert the Orthogonality Thesis is to assert that no matter how intelligent and rational an agent, no matter the breadth of its understanding, no matter the strength of its commitment to objectivity, no matter its abilities to self-reflect and update, it would still never realise that making huge numbers of paperclips is arbitrary and unworthy of its abilities”
The orthogonality claim only has bite against Convergence/Moral Realism if it relates to all or most or typical rational intelligent agents, because that is how moral realists define their claim: they claim that ideal rational agents of a typical kind will converge, or that most rational-enough and intelligent-enough agents will converge. You might be able to build a (genuinely intelligent, reflecting and updating) Clippy, but that wouldn’t prove anything. The natural existence of sociopaths doesn’t disprove MR because they are statiscally rare, and their typicality is in doubt. You can’t prove anything about morality by genetically engineering a sociopath.
As an argument against MR/C, Orthogonality has to claim that the typical, statistically common kind of agent could have arbitrary goals, and that the evidence of convergence amongst humans is explained by specific cultural or genetic features, not by rationality in general.
ETA:
If we don’t understand the relationship between instrumental intelligence and goals, Clippies will seem possible—in the way that p-zombies do if you don’t understand the relationship between matter and consciousness.
Because I want to be sure that I’m understanding what the claim you’re making is.
Okay...so I agree with the Convergence theorist on what the implications of the Orthogonality Thesis are, and I still think the Orthogonality Thesis is true.
Hold on now...that makes the claim completely different than what I thought we were talking about up till now. I thought we were talking about whether or not all rational agents would be in agreement about what morality is, independent of specifically human preferences?
We can have the other discussion too...but not before settling whether or not the Orthogonality Thesis is in fact true “in principle”. Remember, we originally started this discussion with my claim that morality is feelings/preference, as opposed to something you could figure out (i.e. something embedded into logic/game theory or the universe itself.) We weren’t originally talking about rational agents to shed light on evolution or plausible AI...we brought them in as hypothetical agents who converge upon the correct answer to any answerable question, to explore whether or not “what is good” is independent from “what do humans think is good”.
I don’t see how. What did you think we were talking about?
I thought we were talking about whether morality was something that could be discovered objectively.
I said:
Then you said:
Then I said
You disagreed, and said
To which I countered
You disagreed:
Which is why
Hence
doesn’t make any sense in our discussion. All rational agents converge on mathematical and ontological facts, by definition. My argument was that there is no such thing as a “moral fact: and moral statements can only be discussed when in reference to the psychology of a small set of creatures which includes humans and some other mammals. I argued that moral statements can’t be “discovered” true or false in any ontological or mathematical sense, nor are they deeply embedded into game theory (meaning it is not always in the interest of all rational agents to follow human morality) - even though game theory does explain how we evolved morality given our circumstances.
If you admit that at least one of all possible rational agent doesn’t converge upon morality, you’ve been in agreement with me this entire time—which means we’ve been talking about different things this entire time...so what did you think we were talking about?
Only by a definition whereby “rational” means “ideally rational”. In the ordinary sense of the term, it perfectly possible for someone who is deemed “rational” in a more-or-less, good-enough sense to fail to understand some mathematical truths. The existence of the innumerate does not disprove the objectivity of mathematics, and the existence of sociopaths does not disprove the objectivity of morality.
Do you believe that it is possible for a rational agent to fail to understand a mathematical truth? Because that seems rather commonplace to me. Unless you mean ideally rational....
I did mean ideally rational.
The whole point of invoking an ideal rational agent in the first place was to demonstrate that moral “truths” aren’t like empirical or mathematical truths in that you can’t discover them objectively through philosophy or mathematics (even if you are infinitely smart). Rather, moral “truths” are peculiar to humans.
If you want to illustrate the non-objectivity of morality, then stating that even ideal rational agents won’t converge on them is one of expressing the point, although it helps to state the “ideal” explicitly. However, that is still only the expression of a claim, not the “demonstration” of one.
I’m not sure what you mean by “statistically common” here. Do you mean a randomly picked agent out of the set of all possible agents?
I mean likely to be encountered, likely to evolve or to be built (unless you are actually trying to build a Clippy)
I think you’ve misunderstood the meta-ethics sequences, then, or I have, because
is quite similar to Eliezer’s position. Although Juno_Watt may have reached it from another direction.
I read it as a warning about expecting sufficiently rational beings to automatically acquire human morality, in the same way that sufficiently rational beings would automatically acquire knowledge about true statements (science, etc). The lesson is that preferences (morality, etc) is different from fact.
If you want to know Eliezer’s views, he spells them out explicitly here—although I think the person most famous for this view is Nietzsche (not that he’s the first to have held this view).
To me, “No universally compelling arguments” means this—two rational agents will converge upon factual statements, but they need not converge upon preferences (moral or otherwise) because moral statements aren’t “facts”.
It really doesn’t matter if you define the pebble sorting as a “moral” preference or a plain old preference.The point is, that humans have a morality module—but that module is in the brain and not a feature which is implicit in logical structures, nor is it a feature implicit in the universe itself.
I agree that is what it is trying to say, but...as you made illustrate above..it only appears to work if the reader is willing to bel fuzzy about the difference between preference and moral preference.
For some value of “explicit”. He doesn’t even restrict the range of agents to rational agents, and no-one expects irrationali agents to agree with each other, or rational ones.
Mathematical statements aren’t empirical facts eitherl but convergence is uncontroversial there.
Are you quite sure that morlaity isn’t implicit in the logic of how-a-society-if-entities-wth-varying-prefernces manage-to-rub-along ?
Juno Watt has read the sequences, but still doesn’t know what Eliezer’s position is.