Is there a strong reason to think that morality is improving? Contrast with science, in which better understanding of physics leads to building better airplanes, notwithstanding the highly persuasive critiques of science from Kuhn, et al. But morality has no objective test.
100 years ago, women were considered inherently inferior. 200 years ago, chattel slavery was widespread. 500 years ago, Europe practiced absolute monarchy. I certainly think today is an improvement. But proponents of those moralities disagree. Since the laws of the universe don’t have a variable for justice, how can I say they are wrong?
It’s tempting to give in to the Whig Theory of History and concede that the “good guys” always win eventually, because this does seem (at least superficially) to be the case; the Nazis and Soviets both lost out, slavery got abolished, feminism and the civil rights movement happened. The question is, though, did the good guys win out because they were “good”, or are they seen as good because they won?
It’s not quite that simple. The descendants of the victors generally see the victors as good, but that doesn’t mean the descendants of the vanquished see the defeated as evil. Nazism seems to be a case where the defeated society really has strongly repudiated its past, but there is plenty of Soviet nostalgia in Russia and Confederate nostalgia in Dixie.
“Morality is improving” is a bit underspecified, as is “science is improving.” But assuming “morality is improving” means something like “on average, people’s moral beliefs are better than they used to be” (which seems to be what you mean), you’re right of course that the question only makes sense if you have some way of identifying “better”.
But then, similar things are true of science and airplanes. A 2011 airplane isn’t “objectively better” than a 1955 airplane. It’s objectively different, certainly, but to assert that the differences are improvements is to imply a value system.
If you’re confident enough in your value system to judge airplanes based on it, what makes judging moral systems based on it any different?
Science is not airplanes, but the capability to produce airplanes. In 2011, we know how to make 1955 airplanes (as well as 2011 airplanes). In 1955, we only knew how to make 1955 airplanes. Science is advancing.
A 2011 airplane isn’t “objectively better” than a 1955 airplane.
I don’t think there is a dispute that the social purpose of an airplane is to move people a substantial distance in exchange for fuel.
Modern airplanes move more people for less fuel than 1955 airplanes. Therefore, they are objectively better than older airplanes. And that doesn’t even address speed.
If you’re confident enough in your value system to judge airplanes based on it, what makes judging moral systems based on it any different?
I’m very confident that I am more moral than Louis XIV. I suspect he would disagree. How should we decide who is right?
I could quibble about the lack of dispute—I know plenty of people who object to the environmental impact of modern planes, for example, some of whom argue that the aviation situation is worse in 2011 than it was in 1955 precisely because they value low environmental impact more than they value moving more people (or, at least, they claim to) -- but that’s really beside my point. My point is just that asserting that moving more people (faster, more comfortably, more cheaply, etc.) for less fuel is what makes airplanes better is asserting a value system. That it is ubiquitously agreed upon (supposing it were) makes it no less a value system.
How should we decide who is right?
Regardless of how we should decide, or even if there is a way that we should decide, the way we will decide is that you will evaluate moral(TimS) - moral(Louis XIV) based on your value system, and I will evaluate it based on mine. (What Louis XIV’s opinion on the matter would have been, had he ever considered it, doesn’t matter much to me, and it certainly doesn’t matter to Louis, who is dead. Does it matter to you?)
Just like you evaluate good(2011 airplanes) - good(1955 airplanes) based on your value system, and I evaluate it based on mine.
Today, everyone agrees that slavery is wrong. So wrong that attempting to implement slavery will cause you to be charged with all sorts of crimes. Yet our ancestors didn’t think slavery was wrong. Were they just idiots?
That it is ubiquitously agreed upon (supposing it were) makes it no less a value system.
I’m not going to argue that Science isn’t a value system, but it succeeds on its own terms. Even if you think that The Structure of Scientific Revolutions is brilliant and insightful, Science shows that it succeeds at what it aims for.
A similar critique of morality can be found in books like Nietzche’s On the Genealogy of Morals. What is morality’s response?
I’m not going to argue that Science isn’t a value system
That’s good to know, but I didn’t claim that science was a value system. I claimed that “what makes an airplane better is carrying more people further with less fuel” is a value system. So is “what makes an airplane better is being painted bright colors”. (As far as I know, nobody holds that one.)
Science may be a value system, but it isn’t one that tells us that carrying more passengers with less fuel is better than carrying fewer passengers with more fuel, nor that having bright colors is better than having non-bright colors. Science helps us find ways to carry more passengers with less fuel, it also helps us find ways to make colors brighter.
A similar critique of morality can be found in books like Nietzche’s On the Genealogy of Morals. What is morality’s response?
Then how did they fail to notice that slavery is wrong?
That’s good to know, but I didn’t claim that science was a value system.
Science the investigative part of humanity’s attempt to control Nature. It is objectively the case that we control Nature better than we once did. I assert that there is evolutionary pressure on our attempts to control nature. Specifically, bad Science fails to control nature.
A similar critique of morality can be found in books like Nietzche’s On the Genealogy of Morals. What is morality’s response?
I don’t understand what this question is asking.
Very paraphrased Structure of Scientific Revolutions: Science makes progress via paradigm shifts.
Very paraphrased Nietzsche: Paradigm shifts have occurred in morality.
If paradigm shifts don’t seem like a radical claim about either Science or Morality, then perhaps I should write a discussion post about why the claim is extraordinary.
I appreciate your feedback. I’m struggling with whether this idea is high enough quality to make a discussion post. And my experience is that I underestimate the problem of inferential distance.
Most people underestimate inferential distance, so that’s a pretty good theory.
If it helps, I think the primary problem I’m having is that you have a habit of substituting discussion of one idea for discussion of another (e.g, “morality’s response” to Nietzsche vs. the radical/extraordinary nature of paradigm shifts, the value system that sorts airplanes vs. the “investigative part of humanity’s attempt to control Nature,” etc.) without explicitly mapping the two.
I assume it’s entirely obvious to you, for example, how you would convert an opinion about paradigm shifts in morality into a statement about morality’s response to Nietzsche and vice-versa, so from your perspective you’re simply alternating synonyms to make your writing more interesting. But it’s not obvious to me, so from my perspective each such transition is basically changing the subject completely, so each round of discussion seems only vaguely related to the round before. Eventually the conversation feels like trying to nail Jello to a tree.
Again, I don’t mean here to accuse you of changing the subject or of having incoherent ideas; for all I know your discussion has been perfectly consistent and coherent, I just lack your ability (and, evidently, atorm’s) to map the various pieces of it to one another (let alone to my own comments). So, something that might help close the inferential distance is to start over and restate your thesis using consistent and clearly defined terms.
Thank you for defining your terms. I agree that we have the same basic neural/behavioral architecture that our stone-age ancestors had, and that we arrange the world such that others suffer less harm (per capita) than our stone-age ancestors did, and that this is a good thing.
There are a lot of people who would argue morality has been getting worse since their own youth. It doesn’t matter when or where we are talking about, it is pretty much always true that a lot of people think this. The same is true of fashion.
If it’s not directly measurable, it must be a hidden node. What are its children? What data would you anticipate seeing if moral goodness is increasing?
I’m asking these basic questions to prompt you to clarify your thinking. If the concept that you label ‘moral goodness’ is not providing any predictions, you should ask yourself why you’re worried about it at all.
“Morality” is a useful word in that it labels a commonly used cluster of ideaspace. Points in that cluster, however, are not castable to an integer or floating point type. You seem to believe that they do implement comparison operators. How do those work, in your view?
You are using some terminology that I don’t recognize, so I’m uncertain if this is responsive, but here goes.
We are faced with “choices” all the time. The things that motivate us to make a particular decision in a choice are called “values.” As it happens, values can be roughly divided into categories like aesthetic values, moral values, etc.
Value can conflict (i.e. support inconsistent decisions). Functionally, every person has a table listing all the values that the person finds persuasive. The values are ranked, so that a person faced with a decision where value A supports a different decision than value B knows that the decision to make is to follow the higher ranked value.
Thus, Socrates says that Aristotle made an immoral choice iff Aristotle was faced with a choice that Socrates would decide using moral values, and Aristotle made a different choice than Socrates would make.
Caveats:
I’m describing a model, not asserting a theory about the territory (i.e. I’m no neurologist)
My statements are attempting to provide a more rigorous definition of value. Hopefully, it and the other words I invoke rigorously (choice, moral, decision) correspond well to ordinary usage of those words.
That’s a good start. Let’s take as given that “morality” refers to an ordered list of values. How do you compare two such lists? Is the greater morality:
The longer list?
The list that prohibits more actions?
The list that prohibits fewer actions?
The closest to alphabetical ordering?
Something else?
Once you decide what actually makes one list better than another, then consider what observable evidence that difference would produce. With a prediction in hand, you can look at the world and gather evidence for or against the hypothesis that “morality” is increasing.
People measure morality be comparing their agreement on moral choices. It’s purely behavioral.
As a corollary, a morality that does not tell a person how to make a choice is functionally defective, but it is not immoral.
There are lots of ways of resolving moral disputes (majority rule, check the oracle, might makes right). But the decision of which resolution method to pick is itself a moral choice. You can force me to make a particular choice, but you can’t use force to make me think that choice was right.
Ok, I like “ordered list of (abstract concepts people use to make decisions).”
I reiterate my points above: When people say a decision is better, they mean the decision was more consistent with their list than alternative decisions. When people disagree about how to make a choice, the conflict resolution procedure each side prefers is also determined by their list.
“Morality” seems to me to be a fuzzy algebraic sum of many different actions that we approve or disapprove of. So the first step might be to list the actions, then whether we approve or disapprove of it and how much. That should keep people busy for a good while. Just trying to decide how to “measure” how much we approve or disapprove of a specific action is likely to be a significant problem.
The same way you make any other moral judgement—whatever way that is.
That is, you are really asking “what, if anything, is morality?” If you had an answer to that question, answering the one you explicitly asked would just be a matter of historical research, and if you don’t, there’s no possibility of answering the one you asked.
Fair enough. I think the combination of historical evidence and the lack of a term for justice in physics equations is strong evidence that morality is not real. And that bothers me. Because it seems like society would have noticed, and society clearly thinks that morality is real.
Perhaps it is real, but is not the sort of thing you are assuming it must be, to be real.
I can’t point to the number 2, and some people, perplexed by this, have asserted that numbers are not real.
I can point to a mountain, or to a river, but I can’t point to what makes a mountain a mountain or a river a river. Some people, perplexed by this, conclude there are no such things as mountains and rivers.
I can’t point to my mind....and so on.
Can I even point? What makes this hand a pointer, and how can anyone else be sure they know what I am pointing to?
Stare at anything hard enough, and you can cultivate perplexity at its existence, and conclude that nothing exists at all. This is a failure mode of the mind, not an insight into reality.
Have you seen the meta-ethics sequence? The meta-ethical position you are arguing is moral nihilism, the belief that there is no such thing as morality. There are plenty of others to consider before deciding for or against nihilism.
It’s funny that I push on the problem of moral nihilism just a little, and suddenly someone thinks I don’t believe in reality. :)
I’ve read the beginning and the end of the meta-ethics sequence, but not the middle. I agree with Eliezer that recursive questions are always possible, but you must stop asking them at some point or you miss more interesting issues. And I agree with his conclusion that the best formulation of modern ethics is consideration for the happiness of beings capable of recursive thought.
I like to write a discussion post (or a series of posts) on this issue, but I don’t know where to start. Someone else responded to me (EDIT: with what seemed to me like] questioning the assertion that science is a one-way ratchet, always getting better, never getting worse. [EDIT: But we don’t seem to have actually communicated at all, which isn’t a success on my part.]
In case you want a connection to Artificial Intelligence:
Eliezer talks about the importance of provably Friendly AI, and I agree with his point. If we create super-intelligence and it doesn’t care about our desires, that would be very bad for us. But I think that the problem I’m highlighting says something about the possibility of proving that an AI is Friendly.
Someone else responded to me by questioning the assertion that science is a one-way ratchet, always getting better, never getting worse.
It seems likely to me that I’m the person you’re referring to. If so, I don’t endorse your summary. More generally, I’m not sure either of us understood the other one clearly enough in that exchange to merit confident statements on either of our parts about what was actually said, short of literal quotes .
Those of our ancestors who were slaves did not like being slaves, and those who were oppressed by monarchies did not like being oppressed. Now some of them may have supported slavery and monarchy in principle, but their morality was clearly broken because they were made deeply unhappy by institutions which they approved of behind a Rawlsian veil of ignorance.
Women didn’t like the particulars of gendered oppression, so we’ve clearly made progress by their standards.
Actually, victims of oppressive systems often support them. Many girls get clitoridectomies because their mothers demand it, even against their fathers’ wishes.
Yes, you can find ways in which victims are made to be complicit in their oppression. But it’s not hard to find ways in which victims genuinely suffer, and that’s all that’s needed for an objective moral standard.
Is there a strong reason to think that morality is improving? Contrast with science, in which better understanding of physics leads to building better airplanes, notwithstanding the highly persuasive critiques of science from Kuhn, et al. But morality has no objective test.
100 years ago, women were considered inherently inferior. 200 years ago, chattel slavery was widespread. 500 years ago, Europe practiced absolute monarchy. I certainly think today is an improvement. But proponents of those moralities disagree. Since the laws of the universe don’t have a variable for justice, how can I say they are wrong?
Funnily enough, I just wrote an essay on the related meta-ethics topic, Singer’s Whiggish ‘expanding circle’ thesis: http://www.gwern.net/Notes#the-narrowing-circle
It’s tempting to give in to the Whig Theory of History and concede that the “good guys” always win eventually, because this does seem (at least superficially) to be the case; the Nazis and Soviets both lost out, slavery got abolished, feminism and the civil rights movement happened. The question is, though, did the good guys win out because they were “good”, or are they seen as good because they won?
It’s not quite that simple. The descendants of the victors generally see the victors as good, but that doesn’t mean the descendants of the vanquished see the defeated as evil. Nazism seems to be a case where the defeated society really has strongly repudiated its past, but there is plenty of Soviet nostalgia in Russia and Confederate nostalgia in Dixie.
“Morality is improving” is a bit underspecified, as is “science is improving.” But assuming “morality is improving” means something like “on average, people’s moral beliefs are better than they used to be” (which seems to be what you mean), you’re right of course that the question only makes sense if you have some way of identifying “better”.
But then, similar things are true of science and airplanes. A 2011 airplane isn’t “objectively better” than a 1955 airplane. It’s objectively different, certainly, but to assert that the differences are improvements is to imply a value system.
If you’re confident enough in your value system to judge airplanes based on it, what makes judging moral systems based on it any different?
Science is not airplanes, but the capability to produce airplanes. In 2011, we know how to make 1955 airplanes (as well as 2011 airplanes). In 1955, we only knew how to make 1955 airplanes. Science is advancing.
Fair point.
I don’t think there is a dispute that the social purpose of an airplane is to move people a substantial distance in exchange for fuel.
Modern airplanes move more people for less fuel than 1955 airplanes. Therefore, they are objectively better than older airplanes. And that doesn’t even address speed.
I’m very confident that I am more moral than Louis XIV. I suspect he would disagree. How should we decide who is right?
I could quibble about the lack of dispute—I know plenty of people who object to the environmental impact of modern planes, for example, some of whom argue that the aviation situation is worse in 2011 than it was in 1955 precisely because they value low environmental impact more than they value moving more people (or, at least, they claim to) -- but that’s really beside my point. My point is just that asserting that moving more people (faster, more comfortably, more cheaply, etc.) for less fuel is what makes airplanes better is asserting a value system. That it is ubiquitously agreed upon (supposing it were) makes it no less a value system.
Regardless of how we should decide, or even if there is a way that we should decide, the way we will decide is that you will evaluate moral(TimS) - moral(Louis XIV) based on your value system, and I will evaluate it based on mine. (What Louis XIV’s opinion on the matter would have been, had he ever considered it, doesn’t matter much to me, and it certainly doesn’t matter to Louis, who is dead. Does it matter to you?)
Just like you evaluate good(2011 airplanes) - good(1955 airplanes) based on your value system, and I evaluate it based on mine.
Why in the world would we do anything else?
Today, everyone agrees that slavery is wrong. So wrong that attempting to implement slavery will cause you to be charged with all sorts of crimes. Yet our ancestors didn’t think slavery was wrong. Were they just idiots?
I’m not going to argue that Science isn’t a value system, but it succeeds on its own terms. Even if you think that The Structure of Scientific Revolutions is brilliant and insightful, Science shows that it succeeds at what it aims for.
A similar critique of morality can be found in books like Nietzche’s On the Genealogy of Morals. What is morality’s response?
No.
That’s good to know, but I didn’t claim that science was a value system. I claimed that “what makes an airplane better is carrying more people further with less fuel” is a value system. So is “what makes an airplane better is being painted bright colors”. (As far as I know, nobody holds that one.)
Science may be a value system, but it isn’t one that tells us that carrying more passengers with less fuel is better than carrying fewer passengers with more fuel, nor that having bright colors is better than having non-bright colors. Science helps us find ways to carry more passengers with less fuel, it also helps us find ways to make colors brighter.
I don’t understand what this question is asking.
Then how did they fail to notice that slavery is wrong?
Science the investigative part of humanity’s attempt to control Nature. It is objectively the case that we control Nature better than we once did. I assert that there is evolutionary pressure on our attempts to control nature. Specifically, bad Science fails to control nature.
Very paraphrased Structure of Scientific Revolutions: Science makes progress via paradigm shifts. Very paraphrased Nietzsche: Paradigm shifts have occurred in morality.
If paradigm shifts don’t seem like a radical claim about either Science or Morality, then perhaps I should write a discussion post about why the claim is extraordinary.
I’m having a very hard time following your point, so if you can present it in a more systematic fashion in a discussion post, that might be best.
I think I followed the point pretty well, although I don’t know that I can explain it any better. It’s worth its own post, TimS.
I appreciate your feedback. I’m struggling with whether this idea is high enough quality to make a discussion post. And my experience is that I underestimate the problem of inferential distance.
Most people underestimate inferential distance, so that’s a pretty good theory.
If it helps, I think the primary problem I’m having is that you have a habit of substituting discussion of one idea for discussion of another (e.g, “morality’s response” to Nietzsche vs. the radical/extraordinary nature of paradigm shifts, the value system that sorts airplanes vs. the “investigative part of humanity’s attempt to control Nature,” etc.) without explicitly mapping the two.
I assume it’s entirely obvious to you, for example, how you would convert an opinion about paradigm shifts in morality into a statement about morality’s response to Nietzsche and vice-versa, so from your perspective you’re simply alternating synonyms to make your writing more interesting. But it’s not obvious to me, so from my perspective each such transition is basically changing the subject completely, so each round of discussion seems only vaguely related to the round before. Eventually the conversation feels like trying to nail Jello to a tree.
Again, I don’t mean here to accuse you of changing the subject or of having incoherent ideas; for all I know your discussion has been perfectly consistent and coherent, I just lack your ability (and, evidently, atorm’s) to map the various pieces of it to one another (let alone to my own comments). So, something that might help close the inferential distance is to start over and restate your thesis using consistent and clearly defined terms.
.
Thank you for defining your terms. I agree that we have the same basic neural/behavioral architecture that our stone-age ancestors had, and that we arrange the world such that others suffer less harm (per capita) than our stone-age ancestors did, and that this is a good thing.
There are a lot of people who would argue morality has been getting worse since their own youth. It doesn’t matter when or where we are talking about, it is pretty much always true that a lot of people think this. The same is true of fashion.
What measurable quantity are you talking about here?
Moral goodness is the quality I’m referencing, but measurable isn’t an adjective easily applied to moral goodness.
If it’s not directly measurable, it must be a hidden node. What are its children? What data would you anticipate seeing if moral goodness is increasing?
I’m asking these basic questions to prompt you to clarify your thinking. If the concept that you label ‘moral goodness’ is not providing any predictions, you should ask yourself why you’re worried about it at all.
I don’t understand, since I don’t think your position is “morality does not exist for lack of ability to measure.”
“Morality” is a useful word in that it labels a commonly used cluster of ideaspace. Points in that cluster, however, are not castable to an integer or floating point type. You seem to believe that they do implement comparison operators. How do those work, in your view?
You are using some terminology that I don’t recognize, so I’m uncertain if this is responsive, but here goes.
We are faced with “choices” all the time. The things that motivate us to make a particular decision in a choice are called “values.” As it happens, values can be roughly divided into categories like aesthetic values, moral values, etc.
Value can conflict (i.e. support inconsistent decisions). Functionally, every person has a table listing all the values that the person finds persuasive. The values are ranked, so that a person faced with a decision where value A supports a different decision than value B knows that the decision to make is to follow the higher ranked value.
Thus, Socrates says that Aristotle made an immoral choice iff Aristotle was faced with a choice that Socrates would decide using moral values, and Aristotle made a different choice than Socrates would make.
Caveats:
I’m describing a model, not asserting a theory about the territory (i.e. I’m no neurologist)
My statements are attempting to provide a more rigorous definition of value. Hopefully, it and the other words I invoke rigorously (choice, moral, decision) correspond well to ordinary usage of those words.
Is this what you are asking?
That’s a good start. Let’s take as given that “morality” refers to an ordered list of values. How do you compare two such lists? Is the greater morality:
The longer list?
The list that prohibits more actions?
The list that prohibits fewer actions?
The closest to alphabetical ordering?
Something else?
Once you decide what actually makes one list better than another, then consider what observable evidence that difference would produce. With a prediction in hand, you can look at the world and gather evidence for or against the hypothesis that “morality” is increasing.
People measure morality be comparing their agreement on moral choices. It’s purely behavioral.
As a corollary, a morality that does not tell a person how to make a choice is functionally defective, but it is not immoral.
There are lots of ways of resolving moral disputes (majority rule, check the oracle, might makes right). But the decision of which resolution method to pick is itself a moral choice. You can force me to make a particular choice, but you can’t use force to make me think that choice was right.
Sorry, I don’t know what morality is. I thought we were talking about “morality”. Taboo your words.
Ok, I like “ordered list of (abstract concepts people use to make decisions).”
I reiterate my points above: When people say a decision is better, they mean the decision was more consistent with their list than alternative decisions. When people disagree about how to make a choice, the conflict resolution procedure each side prefers is also determined by their list.
“Morality” seems to me to be a fuzzy algebraic sum of many different actions that we approve or disapprove of. So the first step might be to list the actions, then whether we approve or disapprove of it and how much. That should keep people busy for a good while. Just trying to decide how to “measure” how much we approve or disapprove of a specific action is likely to be a significant problem.
What is immoral about monarchy (relative to democracy)?
Absolute monarchy vs. Limited Monarchy
I confess I don’t know much about the little European monarchies you highlighted, but I strongly suspect that they are not Absolute Monarchies.
You mean relative to republic. All of these are democracies.
The same way you make any other moral judgement—whatever way that is.
That is, you are really asking “what, if anything, is morality?” If you had an answer to that question, answering the one you explicitly asked would just be a matter of historical research, and if you don’t, there’s no possibility of answering the one you asked.
Fair enough. I think the combination of historical evidence and the lack of a term for justice in physics equations is strong evidence that morality is not real. And that bothers me. Because it seems like society would have noticed, and society clearly thinks that morality is real.
Society has failed to notice lots of things.
Perhaps it is real, but is not the sort of thing you are assuming it must be, to be real.
I can’t point to the number 2, and some people, perplexed by this, have asserted that numbers are not real.
I can point to a mountain, or to a river, but I can’t point to what makes a mountain a mountain or a river a river. Some people, perplexed by this, conclude there are no such things as mountains and rivers.
I can’t point to my mind....and so on.
Can I even point? What makes this hand a pointer, and how can anyone else be sure they know what I am pointing to?
Stare at anything hard enough, and you can cultivate perplexity at its existence, and conclude that nothing exists at all. This is a failure mode of the mind, not an insight into reality.
Have you seen the meta-ethics sequence? The meta-ethical position you are arguing is moral nihilism, the belief that there is no such thing as morality. There are plenty of others to consider before deciding for or against nihilism.
How hard do you think it would be to summarize the content of the meta-ethics sequence that isn’t implicit from the Human’s Guide to Words?
I never recommend anyone read the ethics sequence fist.
It’s funny that I push on the problem of moral nihilism just a little, and suddenly someone thinks I don’t believe in reality. :)
I’ve read the beginning and the end of the meta-ethics sequence, but not the middle. I agree with Eliezer that recursive questions are always possible, but you must stop asking them at some point or you miss more interesting issues. And I agree with his conclusion that the best formulation of modern ethics is consideration for the happiness of beings capable of recursive thought.
I like to write a discussion post (or a series of posts) on this issue, but I don’t know where to start. Someone else responded to me (EDIT: with what seemed to me like] questioning the assertion that science is a one-way ratchet, always getting better, never getting worse. [EDIT: But we don’t seem to have actually communicated at all, which isn’t a success on my part.]
In case you want a connection to Artificial Intelligence:
Eliezer talks about the importance of provably Friendly AI, and I agree with his point. If we create super-intelligence and it doesn’t care about our desires, that would be very bad for us. But I think that the problem I’m highlighting says something about the possibility of proving that an AI is Friendly.
It seems likely to me that I’m the person you’re referring to. If so, I don’t endorse your summary. More generally, I’m not sure either of us understood the other one clearly enough in that exchange to merit confident statements on either of our parts about what was actually said, short of literal quotes .
Those of our ancestors who were slaves did not like being slaves, and those who were oppressed by monarchies did not like being oppressed. Now some of them may have supported slavery and monarchy in principle, but their morality was clearly broken because they were made deeply unhappy by institutions which they approved of behind a Rawlsian veil of ignorance.
Women didn’t like the particulars of gendered oppression, so we’ve clearly made progress by their standards.
EDIT: Why the downvotes?
Actually, victims of oppressive systems often support them. Many girls get clitoridectomies because their mothers demand it, even against their fathers’ wishes.
Yes, you can find ways in which victims are made to be complicit in their oppression. But it’s not hard to find ways in which victims genuinely suffer, and that’s all that’s needed for an objective moral standard.
NMDV, but maybe it’s for “their morality was clearly broken”?