I find it impossible to engage thoughtfully with philosophical questions about morality because I remain unconvinced of the soundness of the first principles that are applied in moral judgments. I am not interested in a moral claim that does not have a basis in some fundamental idea with demonstrable validity. I will try to contain my critique to those claims that do attempt at least what I think to be this basic level of intellectual rigor.
Note 1: I recognize that I introduced many terms in the above statement that are open to challenge as loaded and biased. I hope this will not distract from my real concern, as stated below.
Note 2: I recognize that I am likely ignorant of the thought of philosophers who have dug into this question. If you can present to me any of these ideas, if they respond clearly and directly to my objections, please do.
I find moral problems intractable and even ridiculous because I have not managed to find foundations for moral judgment through my own inquiry and those foundations proposed by others have all proven specious, at least in my judgment. Examples of the latter include religious ethical systems that claim basis in the mind of a deity, ethical systems based on an individual’s emotional response to X scenario, and pragmatic claims, such as X is moral because it is useful. I admit that the latter is the argument that comes nearest to intriguing me.
Overall, I am frustrated with the a priori assumption that morality must exist (an assumption seemingly based on the fact that the word exists), so let us set out to FIND it. Perhaps it should be found first, and assumptions can come later. Until it is, I have to be neutral and frustrated.
I have not yet had a conversation in which my interlocutor, while making moral claims, could provide convincing definitions of fundamental principies, including justice, duty, the good, vice, etc., though the speaker will have readily made use of these terms. It is not helpful that the population of individuals in society who have attempted to understand and establish such principles on their own, and done so carefully and analytically, is near zero.
As for scenarios resembling the Torture vs. Dust Specks problem: My response is to reject the premise. No, I do not need to make that choice! Morality, at least as that word seems to be applied in non-academic fashion (meaning in daily use), has nothing to do with such abstraction. Moral choices involve actual theft, actual death, actual starvation, actual inequality, etc. etc. The Torture vs. Dust Specks choice is one that no one will ever need to make, so while it might be an intriguing question, I think it avoids the actual subject of morality, or what you might want to call “applied morality”. I feel the same about psychological studies that ask questions about pushing people in front of trains. This is a field built only of theory with no area that actually touches human experience.
I support and agree with every paragraph except the last one.
I cannot come up with any sensible, useful separation between “actual scenarios” and “thought experiments”. Consider the following question: “You have the ability to instantly make an arbitrarily high number of copies of a book, and distribute them to billions of people, at a negligible cost to everyone. Should you compensate the author before doing so?”. To us this is an everyday matter, but to a medieval scholar it is about as much of an abstraction as “torture vs. dust specks”. It is likely more abstract to him than “push the fat guy on the train tracks” is to us.
While I don’t believe that morality is well-founded, I do believe that the word “morality” has some meaning, even if it is incorrect—in the same way that theism isn’t well-founded, but “theism” still means something. And I consider that the word “morality” must indicate a function (or a set of functions) having its domain included in the set of logically possible universes, and its codomain in some sort of algebraic structure (utilitarians should think it’s a totally ordered field, but it needn’t be such a powerful structure). I do not think that whether we might actually experience a given universe is a relevant criterion to morality as most people intend the word.
If I take you correctly, you are pointing out that thought experiments, now abstract, can become actual through progress and chance of time, circumstance, technology, etc., and thus are useful in understanding morality.
If this is an unfair assessment, correct me!
I agree with you, but I also hold to my original claim, as I do not think that they contradict. I agree that the thought experiment can be a useful tool for talking about morality as a set of ideas and reactions out-of-time. However, I do not agree that the thought experiments I have read have convinced me of anything about morality in actual practice. This is for one reason alone: I am not convinced that the operation of human reason is the same in all cases, and in this particular, in the two cases of the theoretical and the physical/actual.
I am not convinced that if a fat man were actually standing there waiting to be shoved piteously onto the tracks that the human mind would necessarily function in the same way it does when sitting in a cafe and discussing the fate of said to-be switch-pusher.
If I were to stake the distinction between the actual and the theoretical on anything, it would be on the above point. What data have we on the reliability of these—I think you must agree that, regardless of the hypothetical opinions of medieval scholar types, the Torture vs. Dust Specks scenario abstract for us now and here—thought experiments to predict human behavior when, to retreat to the cliche, one is actually in the trenches?
This may have some connection to the often-experienced phenomenon when in conversation of casual nonchalance and liberalism about issues that do not affect the speaker and a sudden and contradictory conservatism about issues that do affect the speaker. This is a phenomenon I encounter very often as a college student. It is gratis to be easy-going about topics that never impact oneself, but when circumstances change and a price is paid, reason does not reliably produce similar conclusions. Perhaps this is not a fair objection however, as we could claim that such a person is being More Wrong.
If you can convince me of a reliable connection, you’ll have convinced me of the larger point.
I am not convinced that if a fat man were actually standing there waiting to be shoved piteously onto the tracks that the human mind would necessarily function in the same way it does when sitting in a cafe and discussing the fate of said to-be switch-pusher.
If I were to stake the distinction between the actual and the theoretical on anything, it would be on the above point. What data have we on the reliability of these—I think you must agree that, regardless of the hypothetical opinions of medieval scholar types, the Torture vs. Dust Specks scenario abstract for us now and here—thought experiments to predict human behavior when, to retreat to the cliche, one is actually in the trenches?
I… don’t think the point of such thought experiments was ever to predict what a human will do. That we do not make the same choices under pressure that we do when given reflection and distance is quite obvious. If you are interested in predicting what people will do, you should look at psychological battery tests, which (should) strive to strike a balance between realism and measurability.
The point of train tracks-type experiments was to force one to demand some coherence from their “moral intuition”, and to this end the fact that you’re making such choices sitting in a café is a feature, not a bug, because it lets you carefully figure out logical conclusions on which (at least in theory) you will then be able to unthinkingly rely once you’re in the heat of the moment (probably not an actual train track scenario, but situations like giving money to beggars or voting during jury duty where you only have seconds or hours to make a choice). When you’re actually in the trenches, as you put it, your brain is going to be overwhelmed by a zillion more cognitive biases than usual, so it’s very much in your interest to try and pre-make as many choices as possible while you have the luxury of double-checking every one of your assumptions and implications.
One problem I have with the way such thought experiments are phrased is that they often ask “what would you do?” rather than “what’s the best thing to do?”, which muddles this notion of being more interested in my moral intuitions about the latter than my predictions about the former.
But I realize that people and cultures vary widely in how they interpret phrases like that.
That is, many people demonstrably do endorse and reject certain kinds of situations in a way that we are inclined to categorize as a moral (rather than an aesthetic or arbitrary or pragmatic) judgment, and demonstrably do signal those endorsements and rejections to one another.
All of that behavior has demonstrable influences on how people are born and live and die, suffer and thrive, are educated and remain ignorant, discover truths and believe falsehoods, are happy and sad, etc.
I believe all of that stuff matters, so I believe how the moral intuitions that influence that stuff are formed, and how they can be influenced, is worth understanding.
And, of course, the SIAI folks believe it matters because they want to engineer an artificial system whose decisions have predictable relationships to our moral intuitions even when it doesn’t directly consult those intuitions.
Now, maybe none of that stuff is what you’re talking about… it’s hard to tell precisely what you are rejecting the study of, actually, so I may be talking past you.
If what you mean to reject is the study of morality as something that exists in the world outside of our minds and our behavior, for example, I agree with you.
I suspect the best way to encourage the rejection of that is to study the actual roots of our moral judgments; as more and more of our judgments can be rigorously explained, there will be less and less room for a “god of the moral gaps” to explain them.
And I agree with NihilCredo that the distinction between “applied morality” and “theoretical morality” is not a stable one—especially when considering large-scale engineering projects—so refusing to consider theoretical questions simply ensures that we’re unprepared for the future.
Also, thought experiments are often useful tools to clarify what our intuitions actually are.
I think we may indeed be talking past each other, so I will try to state my case more cogently.
I am not denying that people do possess ideas about something named “morality”. It would be absurd to claim otherwise, as we are here discussing such ideas.
I am denying that, even if I accept all of their assumptions, individuals who claim these ideas as more-than-subjective—by that I think I mean that they claim their ideas able to be applied to a group rather than only to one man, the holder of the ideas—can convince me that these ideas are not wholly subjective and individual-dependent.
If it is the case that morality is individual only, then that is an interesting conclusion and something to talk about, but it does seem, at least to a first approximation, that for a judgment to be considered moral, it must have some broader applicability among individuals, rather than concerning but one person. What can Justice be if it is among one man only? This seems a critical part of what is meant by “morality”. It is in this latter, broad case, that moral philosophy appears null.
If you possess an idea of morality desire that I consider it to have some connection with the world and with all persons—and surely I must require that it have such a connection, as moral claims attempt to dictate the interaction between people, and thus cannot be content to be contained in one mind alone—at least enough of a connection that you can, through reasoned argument, convince me that your claims are both valid and sound, then surely your ideas must make reference to principles that I can discover individually to both exist and serve as predicates to your ideas. If you cannot elucidate these foundations, then how can I be brought to your view through reason? This was the intent of my original criticism, to ask why these foundations are so lousy and to beg that someone make them otherwise if moral claims are to be made.
I think that this is the crux of my objection. I cannot find moral claims that I can be brought to accept through reason alone, as even in the most impressive cases such claims are deeply infected by subjective assumptions that are incommunicable and—dare I write it? --- irrational.
(This is to change the subject somewhat, but I find that the quality of an idea that allows it to be communicated is necessary to its being considered the result of reason and objective. I use that last word with 10,000 pounds of hesitation.)
However, and now I think that we are talking to each other directly, if, when you write of moral ideas, you refer only to those ideas that currently do exist, whether logically well-constructed or not, and you say that you are interested in studying these for their effects, then I am agreed.
I certainly agree that, whether I am convinced of its validity or use, morality does exist as a thing in the minds of men and thus as an influence on human life. But, I think that restricting ourselves to this case has gargantuan ramifications for the definition of “moral” and drastically cuts the domain of objects on which moral ideas can act. It seems this domain can include only those which involve human beings in some fashion. If morality is exclusively a consequence of the history of human evolution and particular to our biology—and I do agree that it is—then I feel that I am bound by it only as far as my own biology has imprinted this moral sense upon me. If it is just biological and not possible to derive through application of reason, then, if I desire to make of myself a creature of reason alone, what care have I for it, but as a curiosity of anthropology?
I suspect that we agree, but that I took a bottom-up approach to get there and left the conclusion implicit, if present at all. All apologies.
Avoided in this post has been struggle with the word “morality” itself. I suspect we could write reams on that. If you think it worthwhile, we should, as the debate may be swung on the ability or inability to pin-down this notion.
(Note: As for SIAI, I think imprinting upon an AI human notions of moral judgments would be hideously dangerous for two reasons:
1) Human beings seem capable in almost every situation of overthrowing such judgments. If said AI is bound in similar manner, then what matters it for controlling or predicting its behavior?
2) If said AI is to possess a notion of justice and of a being who has abdicated certain rights due to immoral conduct, what will its judgment be of the humanity that has taught it morals? Can it not glance, not at history, but simply at the current state of the world and find immediately and with disgust ample grounds for the conclusion that very many humans have surrendered any claim to the moral life? It would be a strange moral algorithm if an AI did not come to this conclusion. Perhaps that is rather the point, as morality even among humans is a strange and often-blind algorithm.)
I agree with your basic point that moral intuitions reflect psychological realities, and that attempts to derive moral truths without explicitly referring to those realities will inevitably turn out to implicitly embed them.
That said, I think you might be introducing unnecessary confusion by talking about “subjective” and “individual.” To pick a simple and trivial objection, it might be that two people, by happenstance, share a set of moral intuitions, and those intuitions might include references to other people. For example, they might each believe “it is best to satisfy the needs of others,” or “it is best to believe things believed by the majority” or “it is best to believe things confirmed by experiment.” Indeed, hundreds of people might share those intuitions, either by happenstance or by mutual influence. In this case, the intuition would not be inter-subjective and non-individual, but still basically the kind of thing we’re talking about.
I assume you mean to contrast it with objective, global things like, say, gravity. Which is fine, but it gets tricky to say that precisely.
It seems this domain can include only those which involve human beings in some fashion.
Here, again, things get slippery. First, I can have moral intuitions about non-humans… for example, I can believe that it’s wrong to club cute widdle baby seals. Second, it’s not obvious that non-humans can’t have moral intuitions.
if I desire to make of myself a creature of reason alone, what care have I for it, but as a curiosity of anthropology?
If that is in fact your desire, then you haven’t a care for it. Or, indeed, for much of anything else.
Speaking personally, though, I would be loathe to give up my love of pie, despite acknowledging that it is a consequence of my own biology and history.
Agreed that imprinting an AI with human notions of moral judgments, especially doing so with the same loose binding to actual behavior humans demonstrate, would be relatively foolish. This is, of course, different from building an AI that is constrained to behave consistently with human moral intuitions.
Agreed that such an AI would easily conclude that humans are not bound by the same constraints that it is bound by. Whether this would elicit disgust or not depends on a lot of things. Sharks are not bound by my moral intuitions, but they don’t disgust me.
I think we might still be talking past each other, but here goes:
The reason I posit and emphasize a distinction between subjective judgments and those that are otherwise—I have a weak reason for not using the term “objective” here—is to highlight a particular feature of moral claims that is lacking, and in thus being lacked, weakens them. That is, I take a claim to be subjective if to hold it myself I must come upon it by chance. I cannot be brought to it through reason alone. It is an opinion or intuition that I cannot trace logically in my own thought, so I cannot communicate it to you by guiding you down the same line. The reason I think that this distinction matters, is that without this logical structure, it not possible for someone to bring me to experience the same intuition through reasoned argument or demonstration. Without this feature, morality must be an island state. This is ruinous, because morality inevitably and necessarily touches upon interactions between people. If it cannot do this, it cannot do much.
Perhaps we should come to common agreement, are at least agreed-upon disagreement on this point before we try other things.
Other Things:
I suspect—this is an idea I have only recently invented have not entirely examined—that any idea that is irrational needs must be essentially incommunicable. How could it be otherwise? If you can lay out the logic behind a thought and give support to its predicates carefully and patiently, and of course your logic is valid and your predicates sound, how can I not, if I am open to reason, not accept what you say as true? That is, if you can demonstrate your ideas as the logical consequences of some set of known truths, I must, because that is what logical consequence is, accept your ideas as true.
I have not witnessed with done with moral notions. Hence my doubt about there existence as rational ideas. I do not doubt that people have moral ideas, but I doubt that they can be communicated to people who have not already come upon them by chance, and who then can only be partially sure that you are of common mind.
Perhaps I can draw a parallel with the distinction between Greek and Babylonian mathematics. The difference between demonstration by proof and attempted demonstration by repeated example. The first (except to mathematicians of the subtle variety), if done properly, seems to be able, in its nature, to be powered to accomplish the goal of communication in every case. Can this be said of the latter type? I think only in the case when the examples given are logically structured so as to be a form of the first type.
"I agree with your basic point that moral intuitions reflect psychological realities, and that attempts to derive moral truths without explicitly referring to those realities will inevitably turn out to implicitly embed them."
I have not wanted to make this claim. What I am claiming is that this claim does appear, thus far, to hold water. However, absence of evidence is not evidence of absence, etc. etc. I am asking for someone to show me the light, as it were.
"First, I can have moral intuitions about non-humans... for example, I can believe that it's wrong to club cute widdle baby seals. Second, it's not obvious that non-humans can't have moral intuitions."
As for your first objection, have not you given precisely the sort of case I was talking about? The moral judgment stated is not about bears clubbing baby seals, it is about humans doing it! Clearly that does involve humans. Come up with a moral judgment about trees overusing carbon dioxide and you’ll have me pinned.
"If that is in fact your desire, then you haven't a care for it. Or, indeed, for much of anything else."
That is just silly, is it not? I must at least care for reason itself. The desire to be rational is a passion indeed. If I must be paradoxical at least that far, I will take it and move on. As for your love of pie, if it is really a consequence of your biology and history, then you CANNOT give it up. You cannot will yourself to unlove it, or it must thus not be the product of the aforesaid forces alone.
I am fairly sure that we aren’t talking past each other, I just disagree with you on some points. Just to try and clarify those points...
You seem to believe that a moral theory must, first and foremost, be compelling… if moral theory X does not convince others, then it can’t do much worth doing. I am not convinced of this. For example, working out my own moral theory in detail allows me to recognize situations that present moral choices, and identify the moral choices I endorse, more accurately… which lowers my chances of doing things that, if I understood better, I would reject. This seems worth doing, even if I’m the only person who ever subscribes to that theory.
You seem to believe that if moral theory X is not rationally compelling, then we cannot come to agree on the specific claims of X except by chance. I’m unconvinced of that. People come to agree on all kinds of things where there is a payoff to agreement, even where the choices themselves are arbitrary. Heck, people often agree on things that are demonstrably false.
Relatedly, you seem to believe that if X logically entails Y, then everyone in the world who endorses X necessarily endorses Y. I’d love to live in that world, but I see no evidence that I do. (That said, it’s possible that you are actually making a moral claim that having logically consistent beliefs is good, rather than a claim that people actually do have such beliefs. I’m inclined to agree with the former.)
I can have a moral intuition that bears clubbing baby seals is wrong, also. Now, I grant you that I, as a human, am less likely to have moral intuitions about things that don’t affect humans in any way… but my moral intuitions might nevertheless be expressible as a general principle which turns out to apply to non-humans as well.
You seem to believe that things I’m biologically predisposed to desire, I will necessarily desire. But lots of biological predispositions are influenced by local environment. My desire for pie may be stronger in some settings than others, and it may be brought lower than my desire for the absence of pie via a variety of mechanisms, and etc. Sure, maybe I can’t “will myself to unlove it,” but I have stronger tools available than unaided will, and we’re developing still-stronger tools every year.
I agree that the desire to be rational is a desire like any other. I intended “much of anything else” to denote an approximate absence of desire, not a complete one.
I think an important part of our disagreement, at least for me, is that you are interested in people generally and morality as it is now—at least your examples come from this set—while I am trying to restrict my inquiry to the most rational type of person, so that I can discover a morality that all rational people can be brought to through reason alone without need for error or chance. If such a morality does not exist among people generally, then I have no interest for the morality of people generally. To bring it up is a non sequitur in such a case.
I do not see that people coming to agree on things that are demonstrably false is a point against me. This fact is precisely why I am turned-off by the current state of ethical thought, as it seems infested with examples of this circumstance. I am not impressed by people who will agree to an intellectual point because it is convenient. I take truth first, at least that is the point of this inquiry.
I am asking a single question: Is there (or can we build) a morality that can be derived with logic from first principles that are obvious to everyone and require no Faith?
You’re right, I’m concerned with morality as it applies to people generally.
If you are exclusively concerned with sufficiently rational people, then we have indeed been talking past each other. Thanks for clarifying that.
As to your question: I submit that for that community, there are only two principles that matter:
Come to agreement with the rest of the community about how to best optimize your shared environment to satisfy your collective preferences.
Abide by that agreement as long as doing so is in the long-term best interests of everyone you care about.
...and the justification for those principles is fairly self-evident. Perhaps that isn’t a morality, but if it isn’t I’m not sure what use that community would have for a morality in the first place. So I say: either of course there is, or there’s no reason to care.
The specifics of that agreement will, of course, depend on the particular interests of the people involved, and will therefore change regularly. There’s no way to build that without actually knowing about the specific community at a specific point in time. But that’s just implementation. It’s like the difference between believing it’s right to not let someone die, and actually having the medical knowledge to save them.
That said, if this community is restricted to people who, as you implied earlier, care only for rationality, then the resulting agreement process is pretty simple. (If they invite people who also care for other things, it will get more complex.)
I am asking a single question: Is there (or can we build) a morality that can be derived with logic from first principles that are obvious to everyone and require no Faith?
I think it’s one of Yudkowsky’s better articles. (On a tangential note, I’m amused to find on re-reading it that I had almost the exact same reaction to The Golden Transcendence, though I had no conscious recollection of the connection when I got around to reading it myself.)
I have not yet had a conversation in which my interlocutor, while making moral claims, could provide convincing definitions of fundamental principies, including justice, duty, the good, vice, etc., though the speaker will have readily made use of these terms. It is not helpful that the population of individuals in society who have attempted to understand and establish such principles on their own, and done so carefully and analytically, is near zero.
Near zero? The number of people who have given careful and analytic thought to the foundational principles of ethics easily numbers in the thousands. Comparable to the number of people who have given equivalent foundational consideration to logic or thermodynamics or statistics. Given that there are seven billion people in the world, it is understandable that you have not yet had a conversation with one of these people. But it is easy enough to find the things they have written—online or in libraries. Give it a look. I can’t promise it will be convincing, or even that it will improve your opinion of the field. But you will find that a lot of serious thought has been given to the subject.
Forgive me for being sloppy with my language. Given what I wrote, your objection is entirely reasonable.
The idea that I meant to express is that, while it seems safe to assume that virtually everyone who has ever lived long enough to become a thinking person has encountered some kind of moral question in his life, we cannot say that an appreciable percentage of these people has sat and carefully analyzed these questions.
Even if we restrict ourselves only to people alive today and living in the United States—an enormous restriction considering the perhaps 100 billion people who have lived ever—the population of thousands you point to is pathetically small. Certainly I agree that Socrates, Mill, Kant, & their Merry Band have approached the subject seriously, but beyond these we’ve but a paucity, which I think is truly surprising given the apparent universality of moral experience.
The comparison to studying thermodynamics or logic (perhaps not quite so with logic), is that while we can say that everyone is of course affected by thermodynamics, almost no one attempts to think about it. The effect of thermodynamics on a person’s life is not impacted by that person’s ignorance of thermodynamic laws. However, a huge number of people do attempt to think and talk about morality, and I am not convinced that this latter group does so rigorously or well, which does have a real and critical effect on what morality is in actual practice.
However, I hope you will notice that this is a minor point, and was not a premise to the larger objection I was putting forward.
I find it impossible to engage thoughtfully with philosophical questions about morality because I remain unconvinced of the soundness of the first principles that are applied in moral judgments. I am not interested in a moral claim that does not have a basis in some fundamental idea with demonstrable validity. I will try to contain my critique to those claims that do attempt at least what I think to be this basic level of intellectual rigor.
Note 1: I recognize that I introduced many terms in the above statement that are open to challenge as loaded and biased. I hope this will not distract from my real concern, as stated below.
Note 2: I recognize that I am likely ignorant of the thought of philosophers who have dug into this question. If you can present to me any of these ideas, if they respond clearly and directly to my objections, please do.
I find moral problems intractable and even ridiculous because I have not managed to find foundations for moral judgment through my own inquiry and those foundations proposed by others have all proven specious, at least in my judgment. Examples of the latter include religious ethical systems that claim basis in the mind of a deity, ethical systems based on an individual’s emotional response to X scenario, and pragmatic claims, such as X is moral because it is useful. I admit that the latter is the argument that comes nearest to intriguing me.
Overall, I am frustrated with the a priori assumption that morality must exist (an assumption seemingly based on the fact that the word exists), so let us set out to FIND it. Perhaps it should be found first, and assumptions can come later. Until it is, I have to be neutral and frustrated.
I have not yet had a conversation in which my interlocutor, while making moral claims, could provide convincing definitions of fundamental principies, including justice, duty, the good, vice, etc., though the speaker will have readily made use of these terms. It is not helpful that the population of individuals in society who have attempted to understand and establish such principles on their own, and done so carefully and analytically, is near zero.
As for scenarios resembling the Torture vs. Dust Specks problem: My response is to reject the premise. No, I do not need to make that choice! Morality, at least as that word seems to be applied in non-academic fashion (meaning in daily use), has nothing to do with such abstraction. Moral choices involve actual theft, actual death, actual starvation, actual inequality, etc. etc. The Torture vs. Dust Specks choice is one that no one will ever need to make, so while it might be an intriguing question, I think it avoids the actual subject of morality, or what you might want to call “applied morality”. I feel the same about psychological studies that ask questions about pushing people in front of trains. This is a field built only of theory with no area that actually touches human experience.
I support and agree with every paragraph except the last one.
I cannot come up with any sensible, useful separation between “actual scenarios” and “thought experiments”. Consider the following question: “You have the ability to instantly make an arbitrarily high number of copies of a book, and distribute them to billions of people, at a negligible cost to everyone. Should you compensate the author before doing so?”. To us this is an everyday matter, but to a medieval scholar it is about as much of an abstraction as “torture vs. dust specks”. It is likely more abstract to him than “push the fat guy on the train tracks” is to us.
While I don’t believe that morality is well-founded, I do believe that the word “morality” has some meaning, even if it is incorrect—in the same way that theism isn’t well-founded, but “theism” still means something. And I consider that the word “morality” must indicate a function (or a set of functions) having its domain included in the set of logically possible universes, and its codomain in some sort of algebraic structure (utilitarians should think it’s a totally ordered field, but it needn’t be such a powerful structure). I do not think that whether we might actually experience a given universe is a relevant criterion to morality as most people intend the word.
If I take you correctly, you are pointing out that thought experiments, now abstract, can become actual through progress and chance of time, circumstance, technology, etc., and thus are useful in understanding morality.
If this is an unfair assessment, correct me!
I agree with you, but I also hold to my original claim, as I do not think that they contradict. I agree that the thought experiment can be a useful tool for talking about morality as a set of ideas and reactions out-of-time. However, I do not agree that the thought experiments I have read have convinced me of anything about morality in actual practice. This is for one reason alone: I am not convinced that the operation of human reason is the same in all cases, and in this particular, in the two cases of the theoretical and the physical/actual.
I am not convinced that if a fat man were actually standing there waiting to be shoved piteously onto the tracks that the human mind would necessarily function in the same way it does when sitting in a cafe and discussing the fate of said to-be switch-pusher.
If I were to stake the distinction between the actual and the theoretical on anything, it would be on the above point. What data have we on the reliability of these—I think you must agree that, regardless of the hypothetical opinions of medieval scholar types, the Torture vs. Dust Specks scenario abstract for us now and here—thought experiments to predict human behavior when, to retreat to the cliche, one is actually in the trenches?
This may have some connection to the often-experienced phenomenon when in conversation of casual nonchalance and liberalism about issues that do not affect the speaker and a sudden and contradictory conservatism about issues that do affect the speaker. This is a phenomenon I encounter very often as a college student. It is gratis to be easy-going about topics that never impact oneself, but when circumstances change and a price is paid, reason does not reliably produce similar conclusions. Perhaps this is not a fair objection however, as we could claim that such a person is being More Wrong.
If you can convince me of a reliable connection, you’ll have convinced me of the larger point.
I… don’t think the point of such thought experiments was ever to predict what a human will do. That we do not make the same choices under pressure that we do when given reflection and distance is quite obvious. If you are interested in predicting what people will do, you should look at psychological battery tests, which (should) strive to strike a balance between realism and measurability.
The point of train tracks-type experiments was to force one to demand some coherence from their “moral intuition”, and to this end the fact that you’re making such choices sitting in a café is a feature, not a bug, because it lets you carefully figure out logical conclusions on which (at least in theory) you will then be able to unthinkingly rely once you’re in the heat of the moment (probably not an actual train track scenario, but situations like giving money to beggars or voting during jury duty where you only have seconds or hours to make a choice). When you’re actually in the trenches, as you put it, your brain is going to be overwhelmed by a zillion more cognitive biases than usual, so it’s very much in your interest to try and pre-make as many choices as possible while you have the luxury of double-checking every one of your assumptions and implications.
One problem I have with the way such thought experiments are phrased is that they often ask “what would you do?” rather than “what’s the best thing to do?”, which muddles this notion of being more interested in my moral intuitions about the latter than my predictions about the former.
But I realize that people and cultures vary widely in how they interpret phrases like that.
Moral intuitions demonstrably exist.
That is, many people demonstrably do endorse and reject certain kinds of situations in a way that we are inclined to categorize as a moral (rather than an aesthetic or arbitrary or pragmatic) judgment, and demonstrably do signal those endorsements and rejections to one another.
All of that behavior has demonstrable influences on how people are born and live and die, suffer and thrive, are educated and remain ignorant, discover truths and believe falsehoods, are happy and sad, etc.
I believe all of that stuff matters, so I believe how the moral intuitions that influence that stuff are formed, and how they can be influenced, is worth understanding.
And, of course, the SIAI folks believe it matters because they want to engineer an artificial system whose decisions have predictable relationships to our moral intuitions even when it doesn’t directly consult those intuitions.
Now, maybe none of that stuff is what you’re talking about… it’s hard to tell precisely what you are rejecting the study of, actually, so I may be talking past you.
If what you mean to reject is the study of morality as something that exists in the world outside of our minds and our behavior, for example, I agree with you.
I suspect the best way to encourage the rejection of that is to study the actual roots of our moral judgments; as more and more of our judgments can be rigorously explained, there will be less and less room for a “god of the moral gaps” to explain them.
And I agree with NihilCredo that the distinction between “applied morality” and “theoretical morality” is not a stable one—especially when considering large-scale engineering projects—so refusing to consider theoretical questions simply ensures that we’re unprepared for the future.
Also, thought experiments are often useful tools to clarify what our intuitions actually are.
I think we may indeed be talking past each other, so I will try to state my case more cogently.
I am not denying that people do possess ideas about something named “morality”. It would be absurd to claim otherwise, as we are here discussing such ideas.
I am denying that, even if I accept all of their assumptions, individuals who claim these ideas as more-than-subjective—by that I think I mean that they claim their ideas able to be applied to a group rather than only to one man, the holder of the ideas—can convince me that these ideas are not wholly subjective and individual-dependent.
If it is the case that morality is individual only, then that is an interesting conclusion and something to talk about, but it does seem, at least to a first approximation, that for a judgment to be considered moral, it must have some broader applicability among individuals, rather than concerning but one person. What can Justice be if it is among one man only? This seems a critical part of what is meant by “morality”. It is in this latter, broad case, that moral philosophy appears null.
If you possess an idea of morality desire that I consider it to have some connection with the world and with all persons—and surely I must require that it have such a connection, as moral claims attempt to dictate the interaction between people, and thus cannot be content to be contained in one mind alone—at least enough of a connection that you can, through reasoned argument, convince me that your claims are both valid and sound, then surely your ideas must make reference to principles that I can discover individually to both exist and serve as predicates to your ideas. If you cannot elucidate these foundations, then how can I be brought to your view through reason? This was the intent of my original criticism, to ask why these foundations are so lousy and to beg that someone make them otherwise if moral claims are to be made.
I think that this is the crux of my objection. I cannot find moral claims that I can be brought to accept through reason alone, as even in the most impressive cases such claims are deeply infected by subjective assumptions that are incommunicable and—dare I write it? --- irrational.
(This is to change the subject somewhat, but I find that the quality of an idea that allows it to be communicated is necessary to its being considered the result of reason and objective. I use that last word with 10,000 pounds of hesitation.)
However, and now I think that we are talking to each other directly, if, when you write of moral ideas, you refer only to those ideas that currently do exist, whether logically well-constructed or not, and you say that you are interested in studying these for their effects, then I am agreed.
I certainly agree that, whether I am convinced of its validity or use, morality does exist as a thing in the minds of men and thus as an influence on human life. But, I think that restricting ourselves to this case has gargantuan ramifications for the definition of “moral” and drastically cuts the domain of objects on which moral ideas can act. It seems this domain can include only those which involve human beings in some fashion. If morality is exclusively a consequence of the history of human evolution and particular to our biology—and I do agree that it is—then I feel that I am bound by it only as far as my own biology has imprinted this moral sense upon me. If it is just biological and not possible to derive through application of reason, then, if I desire to make of myself a creature of reason alone, what care have I for it, but as a curiosity of anthropology?
I suspect that we agree, but that I took a bottom-up approach to get there and left the conclusion implicit, if present at all. All apologies.
Avoided in this post has been struggle with the word “morality” itself. I suspect we could write reams on that. If you think it worthwhile, we should, as the debate may be swung on the ability or inability to pin-down this notion.
(Note: As for SIAI, I think imprinting upon an AI human notions of moral judgments would be hideously dangerous for two reasons: 1) Human beings seem capable in almost every situation of overthrowing such judgments. If said AI is bound in similar manner, then what matters it for controlling or predicting its behavior? 2) If said AI is to possess a notion of justice and of a being who has abdicated certain rights due to immoral conduct, what will its judgment be of the humanity that has taught it morals? Can it not glance, not at history, but simply at the current state of the world and find immediately and with disgust ample grounds for the conclusion that very many humans have surrendered any claim to the moral life? It would be a strange moral algorithm if an AI did not come to this conclusion. Perhaps that is rather the point, as morality even among humans is a strange and often-blind algorithm.)
I agree with your basic point that moral intuitions reflect psychological realities, and that attempts to derive moral truths without explicitly referring to those realities will inevitably turn out to implicitly embed them.
That said, I think you might be introducing unnecessary confusion by talking about “subjective” and “individual.” To pick a simple and trivial objection, it might be that two people, by happenstance, share a set of moral intuitions, and those intuitions might include references to other people. For example, they might each believe “it is best to satisfy the needs of others,” or “it is best to believe things believed by the majority” or “it is best to believe things confirmed by experiment.” Indeed, hundreds of people might share those intuitions, either by happenstance or by mutual influence. In this case, the intuition would not be inter-subjective and non-individual, but still basically the kind of thing we’re talking about.
I assume you mean to contrast it with objective, global things like, say, gravity. Which is fine, but it gets tricky to say that precisely.
Here, again, things get slippery. First, I can have moral intuitions about non-humans… for example, I can believe that it’s wrong to club cute widdle baby seals. Second, it’s not obvious that non-humans can’t have moral intuitions.
If that is in fact your desire, then you haven’t a care for it. Or, indeed, for much of anything else.
Speaking personally, though, I would be loathe to give up my love of pie, despite acknowledging that it is a consequence of my own biology and history.
Agreed that imprinting an AI with human notions of moral judgments, especially doing so with the same loose binding to actual behavior humans demonstrate, would be relatively foolish. This is, of course, different from building an AI that is constrained to behave consistently with human moral intuitions.
Agreed that such an AI would easily conclude that humans are not bound by the same constraints that it is bound by. Whether this would elicit disgust or not depends on a lot of things. Sharks are not bound by my moral intuitions, but they don’t disgust me.
I think we might still be talking past each other, but here goes:
The reason I posit and emphasize a distinction between subjective judgments and those that are otherwise—I have a weak reason for not using the term “objective” here—is to highlight a particular feature of moral claims that is lacking, and in thus being lacked, weakens them. That is, I take a claim to be subjective if to hold it myself I must come upon it by chance. I cannot be brought to it through reason alone. It is an opinion or intuition that I cannot trace logically in my own thought, so I cannot communicate it to you by guiding you down the same line. The reason I think that this distinction matters, is that without this logical structure, it not possible for someone to bring me to experience the same intuition through reasoned argument or demonstration. Without this feature, morality must be an island state. This is ruinous, because morality inevitably and necessarily touches upon interactions between people. If it cannot do this, it cannot do much.
Perhaps we should come to common agreement, are at least agreed-upon disagreement on this point before we try other things.
Other Things:
I suspect—this is an idea I have only recently invented have not entirely examined—that any idea that is irrational needs must be essentially incommunicable. How could it be otherwise? If you can lay out the logic behind a thought and give support to its predicates carefully and patiently, and of course your logic is valid and your predicates sound, how can I not, if I am open to reason, not accept what you say as true? That is, if you can demonstrate your ideas as the logical consequences of some set of known truths, I must, because that is what logical consequence is, accept your ideas as true.
I have not witnessed with done with moral notions. Hence my doubt about there existence as rational ideas. I do not doubt that people have moral ideas, but I doubt that they can be communicated to people who have not already come upon them by chance, and who then can only be partially sure that you are of common mind.
Perhaps I can draw a parallel with the distinction between Greek and Babylonian mathematics. The difference between demonstration by proof and attempted demonstration by repeated example. The first (except to mathematicians of the subtle variety), if done properly, seems to be able, in its nature, to be powered to accomplish the goal of communication in every case. Can this be said of the latter type? I think only in the case when the examples given are logically structured so as to be a form of the first type.
I have not wanted to make this claim. What I am claiming is that this claim does appear, thus far, to hold water. However, absence of evidence is not evidence of absence, etc. etc. I am asking for someone to show me the light, as it were.
As for your first objection, have not you given precisely the sort of case I was talking about? The moral judgment stated is not about bears clubbing baby seals, it is about humans doing it! Clearly that does involve humans. Come up with a moral judgment about trees overusing carbon dioxide and you’ll have me pinned.
That is just silly, is it not? I must at least care for reason itself. The desire to be rational is a passion indeed. If I must be paradoxical at least that far, I will take it and move on. As for your love of pie, if it is really a consequence of your biology and history, then you CANNOT give it up. You cannot will yourself to unlove it, or it must thus not be the product of the aforesaid forces alone.
I am fairly sure that we aren’t talking past each other, I just disagree with you on some points. Just to try and clarify those points...
You seem to believe that a moral theory must, first and foremost, be compelling… if moral theory X does not convince others, then it can’t do much worth doing. I am not convinced of this. For example, working out my own moral theory in detail allows me to recognize situations that present moral choices, and identify the moral choices I endorse, more accurately… which lowers my chances of doing things that, if I understood better, I would reject. This seems worth doing, even if I’m the only person who ever subscribes to that theory.
You seem to believe that if moral theory X is not rationally compelling, then we cannot come to agree on the specific claims of X except by chance. I’m unconvinced of that. People come to agree on all kinds of things where there is a payoff to agreement, even where the choices themselves are arbitrary. Heck, people often agree on things that are demonstrably false.
Relatedly, you seem to believe that if X logically entails Y, then everyone in the world who endorses X necessarily endorses Y. I’d love to live in that world, but I see no evidence that I do. (That said, it’s possible that you are actually making a moral claim that having logically consistent beliefs is good, rather than a claim that people actually do have such beliefs. I’m inclined to agree with the former.)
I can have a moral intuition that bears clubbing baby seals is wrong, also. Now, I grant you that I, as a human, am less likely to have moral intuitions about things that don’t affect humans in any way… but my moral intuitions might nevertheless be expressible as a general principle which turns out to apply to non-humans as well.
You seem to believe that things I’m biologically predisposed to desire, I will necessarily desire. But lots of biological predispositions are influenced by local environment. My desire for pie may be stronger in some settings than others, and it may be brought lower than my desire for the absence of pie via a variety of mechanisms, and etc. Sure, maybe I can’t “will myself to unlove it,” but I have stronger tools available than unaided will, and we’re developing still-stronger tools every year.
I agree that the desire to be rational is a desire like any other. I intended “much of anything else” to denote an approximate absence of desire, not a complete one.
I think an important part of our disagreement, at least for me, is that you are interested in people generally and morality as it is now—at least your examples come from this set—while I am trying to restrict my inquiry to the most rational type of person, so that I can discover a morality that all rational people can be brought to through reason alone without need for error or chance. If such a morality does not exist among people generally, then I have no interest for the morality of people generally. To bring it up is a non sequitur in such a case.
I do not see that people coming to agree on things that are demonstrably false is a point against me. This fact is precisely why I am turned-off by the current state of ethical thought, as it seems infested with examples of this circumstance. I am not impressed by people who will agree to an intellectual point because it is convenient. I take truth first, at least that is the point of this inquiry.
I am asking a single question: Is there (or can we build) a morality that can be derived with logic from first principles that are obvious to everyone and require no Faith?
You’re right, I’m concerned with morality as it applies to people generally.
If you are exclusively concerned with sufficiently rational people, then we have indeed been talking past each other. Thanks for clarifying that.
As to your question: I submit that for that community, there are only two principles that matter:
Come to agreement with the rest of the community about how to best optimize your shared environment to satisfy your collective preferences.
Abide by that agreement as long as doing so is in the long-term best interests of everyone you care about.
...and the justification for those principles is fairly self-evident. Perhaps that isn’t a morality, but if it isn’t I’m not sure what use that community would have for a morality in the first place. So I say: either of course there is, or there’s no reason to care.
The specifics of that agreement will, of course, depend on the particular interests of the people involved, and will therefore change regularly. There’s no way to build that without actually knowing about the specific community at a specific point in time. But that’s just implementation. It’s like the difference between believing it’s right to not let someone die, and actually having the medical knowledge to save them.
That said, if this community is restricted to people who, as you implied earlier, care only for rationality, then the resulting agreement process is pretty simple. (If they invite people who also care for other things, it will get more complex.)
Very well put.
Perhaps you’ve already encountered this, but your question calls to mind the following piece by Yudkowsky: No Universally Compelling Arguments, which is near the start of his broader metaethics sequence.
I think it’s one of Yudkowsky’s better articles.
(On a tangential note, I’m amused to find on re-reading it that I had almost the exact same reaction to The Golden Transcendence, though I had no conscious recollection of the connection when I got around to reading it myself.)
I agree vehemently with your comment.
Near zero? The number of people who have given careful and analytic thought to the foundational principles of ethics easily numbers in the thousands. Comparable to the number of people who have given equivalent foundational consideration to logic or thermodynamics or statistics. Given that there are seven billion people in the world, it is understandable that you have not yet had a conversation with one of these people. But it is easy enough to find the things they have written—online or in libraries. Give it a look. I can’t promise it will be convincing, or even that it will improve your opinion of the field. But you will find that a lot of serious thought has been given to the subject.
Forgive me for being sloppy with my language. Given what I wrote, your objection is entirely reasonable.
The idea that I meant to express is that, while it seems safe to assume that virtually everyone who has ever lived long enough to become a thinking person has encountered some kind of moral question in his life, we cannot say that an appreciable percentage of these people has sat and carefully analyzed these questions.
Even if we restrict ourselves only to people alive today and living in the United States—an enormous restriction considering the perhaps 100 billion people who have lived ever—the population of thousands you point to is pathetically small. Certainly I agree that Socrates, Mill, Kant, & their Merry Band have approached the subject seriously, but beyond these we’ve but a paucity, which I think is truly surprising given the apparent universality of moral experience.
The comparison to studying thermodynamics or logic (perhaps not quite so with logic), is that while we can say that everyone is of course affected by thermodynamics, almost no one attempts to think about it. The effect of thermodynamics on a person’s life is not impacted by that person’s ignorance of thermodynamic laws. However, a huge number of people do attempt to think and talk about morality, and I am not convinced that this latter group does so rigorously or well, which does have a real and critical effect on what morality is in actual practice.
However, I hope you will notice that this is a minor point, and was not a premise to the larger objection I was putting forward.