After reading lots of debates on these topics, I’m no longer sure what the terms mean. Is a paperclip maximizer a “moral nihilist”? If yes, then so am I. Same for no.
a paperclip maximizer is something that, as you know, requires incredible amount of work put into defining what a paperclip is (if that is even possible without fixing a model). It subsequently has an incredibly complex moral system, very stupid one, but incredibly complex nonetheless. Try something like equation-solver.
I see no reason to think a paperclip maximizer would need to have any particular meta-ethics. There are possible paperclip maximizers that are and one’s that aren’t. As rule of thumb, an agent’s normative ethics, that is, what it cares about, be it human flourishing or paperclips does not logically constrain it’s meta-ethical views.
That’s a nice and unexpected answer, so I’ll continue asking questions I have no clue about :-)
If metaethics doesn’t influence paperclip maximization, then why do I need metaethics? Can we point out the precise difference between humans and paperclippers that gives humans the need for metaethics? Is it the fact that we’re not logically omniscient about our own minds, or is it something deeper?
Perhaps I misunderstood. There are definitely possible scenarios in which metaethics could matter to a paperclip maximizer. It’s just that answering “what meta-ethics would the best paperclip maximizer have?” isn’t any easier than answering “what is the ideal metaethics?”. Varying an agent’s goal structure doesn’t change the question.
That said, if you think humans are just like paperclip maximizers except they’re trying to maximize something else than you’re already 8/10ths of the way to moral anti-realism (Come! Take those last two steps the water is fine!).
Of course it’s also the case that meta-ethics probably matters more to humans than paperclip maximizers: In particular metaethics matters for humans because of individual moral uncertainty, group and individual moral change, differences in between individual moralities, and the overall complexity of our values. There are probably similar possible issues for paperclip maximizers—like how should they resolve uncertainty over what counts as a paperclip or deal with agents that are ignorant of the ultimate value of paperclips—and thinking about them pumps my anti-realist intuitions.
Is it the fact that we’re not logically omniscient about our own minds, or is it something deeper?
Well, there’s certainly that. Also, human algorithms for decision-making can feel different from simply looking up a utility—the algorithm can be something more like a “treasure map” for locating morality, looking out at the world in a way that can feel as if morality was a light shining from outside.
Consider dealings with agents that have morals that conflict with your own. Obviously, major value conflicts preclude co-existence. Let’s assume it is a minor conflict—Bob believes eating cow milk and beef at the same meal is immoral.
It is possible to develop instrumental or terminal values to resolve how much you tolerate Bob’s different value—without reference to any meta-ethical theory. But I think that meta-ethical considerations play a large role in how tolerance of value conflict is resolved—for some people, at least.
I would certainly say a paperclip maximizer morality falls in the former camp (objective, able to be scientifically investigated, real), although I’m not intimately familiar with the realist/anti-realist terminology.
Hank, why would Clippy believe that maximizing paperclips is based on something external to its own mind? Clippy could just as easily be programmed to desire staples, and Clippy is probably intelligent enough to know that.
That said, I think Jack’s general point about the relationship between ethics and meta-ethics is probably right.
Presumably Clippy has a hard-coded utility function sitting in his source code somewhere. It’s a real set of 0s and 1s sitting on a disk somewhere, and we could open the source file and investigate the code.
Clippy’s value system is a specific objective, pre-programmed utility function that’s inherent in his design and independent/prior to any knowledge or other cognitive content Clippy eventually gains or invents.
And yes, it could have been easily changed (until such a time as Clippy is all grown up and would prevent such change) to make him desire staples. But then that’s a different design and we’d probably better call him Stapley at that point.
Ideally, one would like objective facts to be universally compelling. If Clippy shows its source code to me, or to another AGI, neither of us would update in favor of believing that paper-clip maximizing is an appropriate terminal value.
Ah, no, I don’t mean “objective morality” in the sense of something like a “one true, universal, metaphysical morality for all mind-designs” like the Socratic/Platonic Form of Good or any such nonsense. I just mean what I said above, something in reality that’s mind-independent and can be investigated scientifically—a definite “is” from which we can make true “ought” statements relative to that “is”.
No. The physical instantiation of a utility function is not an argument for moral realism. On the complete contrary, defining moral actions as “whatever an agents utility function says” is straight-forward, definitional, no-bones-about-it moral subjectivism.
Put it this way: the paperclip maximizer is not going to approve of your behavior.
Hank, I definitely don’t think there’s any activity that (1) can reasonably be labeled “scientific investigation” and (2) can solve the is-ought divide.
First, that essay is aimed primarily at those who think dualism is required in order to talk about morality at all—obviously that’s not the discussion we are having.
Second, the issue is not whether there are (1) universal (2) morally relevant (3) human preferences that (4) have been created by evolution. The answer to that question is yes (i.e. hunger, sexual desire). But that alone does not show that there is a universal way for humans to resolve moral dilemma.
If we study and quantify the godshatter to the point that we can precisely describe “human nature,” we aren’t guaranteed in advance to know that appeals to human nature will resolve every moral dilemma. If reference to human nature doesn’t, then evolutionary preference doesn’t prove moral realism.
I’m not sure why “universality” is really that important here. Suppose we are just talking about one person, why can’t they reduce their value judgments down to their own precisely described nature to resolve every moral dilemma they face? With a read-out of their actual terminal values defined by the godshatter, they can employ the usual consequentialist expected utility calculus to solve any question, in principle.
But that’s just a confusion between two different meanings “objective vs. subjective”.
People apparently tend to interpret “objective” as something “universal” in the sense of like some metaphysical Form of Good, as opposed to “subjective” meaning “relative to a person”. That distinction is completely stupid and wouldn’t even occur to me.
I’m using it in the sense of, something relative to a person but still “a fact of reality able to be investigated by science that is independent/prior to any of the mind’s later acquisition of knowledge/content”, versus “something that is not an independent/prior fact of reality, but rather some later invention of the mind”.
So lets clear something up: the two attributes objective/subjective and universal/relative are logically distinct. You can can have objective relativism (“What is moral is the law and the law varies from place to place.”) and subjective universalism (“What is moral is just our opinions, but we all have the same opinions”).
a fact of reality able to be investigated by science that is independent/prior to any of the mind’s later acquisition of knowledge/content”, versus “something that is not an independent/prior fact of reality, but rather some later invention of the mind”.
The attribute “objective” or “subjective” in meta-ethics refers to the status of moral judgments themselves not descriptive facts about what moral judgments people actually make or the causal/mental facts that lead people to make them. Of course it is the case that people make moral judgments, that we can observe those judgments and can learn something about the brains that make them. No one here is denying that there are objective facts about moral psychology. The entire question is about the status of the moral judgments themselves. What makes it true when I say “Murder is immoral”? If your answer references my mind your answer is subjectivist, if your answer is “nothing” than you are a non-cognitivist or an error theorist. All those camps are anti-realist camps. Objectivist answers include “the Categorical Imperative” and “immorality supervenes on human suffering”.
Is there accessible discussion out there of why one might expect real world correlation between objectivity and universality.
I see that subjective universalism is logically coherent, but I wouldn’t expect it to be true—it seems like too much of a coincidence that nothing objective requires people have the same beliefs, yet people do anyway.
Lots of things can cause a convergence of belief other than objective truth, e.g. adaptive fitness. But certainly, objectivity usually implies universality.
After reading lots of debates on these topics, I’m no longer sure what the terms mean. Is a paperclip maximizer a “moral nihilist”? If yes, then so am I. Same for no.
a paperclip maximizer is something that, as you know, requires incredible amount of work put into defining what a paperclip is (if that is even possible without fixing a model). It subsequently has an incredibly complex moral system, very stupid one, but incredibly complex nonetheless. Try something like equation-solver.
I suspect the OP is asking whether you are a moral realist or anti-realist.
Okay, is a paperclip maximizer a moral realist?
I see no reason to think a paperclip maximizer would need to have any particular meta-ethics. There are possible paperclip maximizers that are and one’s that aren’t. As rule of thumb, an agent’s normative ethics, that is, what it cares about, be it human flourishing or paperclips does not logically constrain it’s meta-ethical views.
That’s a nice and unexpected answer, so I’ll continue asking questions I have no clue about :-)
If metaethics doesn’t influence paperclip maximization, then why do I need metaethics? Can we point out the precise difference between humans and paperclippers that gives humans the need for metaethics? Is it the fact that we’re not logically omniscient about our own minds, or is it something deeper?
Perhaps I misunderstood. There are definitely possible scenarios in which metaethics could matter to a paperclip maximizer. It’s just that answering “what meta-ethics would the best paperclip maximizer have?” isn’t any easier than answering “what is the ideal metaethics?”. Varying an agent’s goal structure doesn’t change the question.
That said, if you think humans are just like paperclip maximizers except they’re trying to maximize something else than you’re already 8/10ths of the way to moral anti-realism (Come! Take those last two steps the water is fine!).
Of course it’s also the case that meta-ethics probably matters more to humans than paperclip maximizers: In particular metaethics matters for humans because of individual moral uncertainty, group and individual moral change, differences in between individual moralities, and the overall complexity of our values. There are probably similar possible issues for paperclip maximizers—like how should they resolve uncertainty over what counts as a paperclip or deal with agents that are ignorant of the ultimate value of paperclips—and thinking about them pumps my anti-realist intuitions.
Well, there’s certainly that. Also, human algorithms for decision-making can feel different from simply looking up a utility—the algorithm can be something more like a “treasure map” for locating morality, looking out at the world in a way that can feel as if morality was a light shining from outside.
Consider dealings with agents that have morals that conflict with your own. Obviously, major value conflicts preclude co-existence. Let’s assume it is a minor conflict—Bob believes eating cow milk and beef at the same meal is immoral.
It is possible to develop instrumental or terminal values to resolve how much you tolerate Bob’s different value—without reference to any meta-ethical theory. But I think that meta-ethical considerations play a large role in how tolerance of value conflict is resolved—for some people, at least.
Not obvious. (How does this “preclusion” work? Is it the best decision available to both agents?)
Well, if I don’t include that sentence, someone nitpicks by saying:
I was trying preempt by making it clear that McH gets imprisoned or killed, even by moral anti-realists (unless they are exceptionally stupid).
I would certainly say a paperclip maximizer morality falls in the former camp (objective, able to be scientifically investigated, real), although I’m not intimately familiar with the realist/anti-realist terminology.
Hank, why would Clippy believe that maximizing paperclips is based on something external to its own mind? Clippy could just as easily be programmed to desire staples, and Clippy is probably intelligent enough to know that.
That said, I think Jack’s general point about the relationship between ethics and meta-ethics is probably right.
Presumably Clippy has a hard-coded utility function sitting in his source code somewhere. It’s a real set of 0s and 1s sitting on a disk somewhere, and we could open the source file and investigate the code.
Clippy’s value system is a specific objective, pre-programmed utility function that’s inherent in his design and independent/prior to any knowledge or other cognitive content Clippy eventually gains or invents.
And yes, it could have been easily changed (until such a time as Clippy is all grown up and would prevent such change) to make him desire staples. But then that’s a different design and we’d probably better call him Stapley at that point.
Ideally, one would like objective facts to be universally compelling. If Clippy shows its source code to me, or to another AGI, neither of us would update in favor of believing that paper-clip maximizing is an appropriate terminal value.
Ah, no, I don’t mean “objective morality” in the sense of something like a “one true, universal, metaphysical morality for all mind-designs” like the Socratic/Platonic Form of Good or any such nonsense. I just mean what I said above, something in reality that’s mind-independent and can be investigated scientifically—a definite “is” from which we can make true “ought” statements relative to that “is”.
See drethelin’s comment below.
Clippy’s code is his mind.
No. The physical instantiation of a utility function is not an argument for moral realism. On the complete contrary, defining moral actions as “whatever an agents utility function says” is straight-forward, definitional, no-bones-about-it moral subjectivism.
Put it this way: the paperclip maximizer is not going to approve of your behavior.
Hank, I definitely don’t think there’s any activity that (1) can reasonably be labeled “scientific investigation” and (2) can solve the is-ought divide.
I didn’t think you would :) I’m curious about the consensus on LW, though. But incidentally, what do you think of Thou Art Godshatter?
First, that essay is aimed primarily at those who think dualism is required in order to talk about morality at all—obviously that’s not the discussion we are having.
Second, the issue is not whether there are (1) universal (2) morally relevant (3) human preferences that (4) have been created by evolution. The answer to that question is yes (i.e. hunger, sexual desire). But that alone does not show that there is a universal way for humans to resolve moral dilemma.
If we study and quantify the godshatter to the point that we can precisely describe “human nature,” we aren’t guaranteed in advance to know that appeals to human nature will resolve every moral dilemma. If reference to human nature doesn’t, then evolutionary preference doesn’t prove moral realism.
I’m not sure why “universality” is really that important here. Suppose we are just talking about one person, why can’t they reduce their value judgments down to their own precisely described nature to resolve every moral dilemma they face? With a read-out of their actual terminal values defined by the godshatter, they can employ the usual consequentialist expected utility calculus to solve any question, in principle.
This says it better than I could.
But that’s just a confusion between two different meanings “objective vs. subjective”.
People apparently tend to interpret “objective” as something “universal” in the sense of like some metaphysical Form of Good, as opposed to “subjective” meaning “relative to a person”. That distinction is completely stupid and wouldn’t even occur to me.
I’m using it in the sense of, something relative to a person but still “a fact of reality able to be investigated by science that is independent/prior to any of the mind’s later acquisition of knowledge/content”, versus “something that is not an independent/prior fact of reality, but rather some later invention of the mind”.
So lets clear something up: the two attributes objective/subjective and universal/relative are logically distinct. You can can have objective relativism (“What is moral is the law and the law varies from place to place.”) and subjective universalism (“What is moral is just our opinions, but we all have the same opinions”).
The attribute “objective” or “subjective” in meta-ethics refers to the status of moral judgments themselves not descriptive facts about what moral judgments people actually make or the causal/mental facts that lead people to make them. Of course it is the case that people make moral judgments, that we can observe those judgments and can learn something about the brains that make them. No one here is denying that there are objective facts about moral psychology. The entire question is about the status of the moral judgments themselves. What makes it true when I say “Murder is immoral”? If your answer references my mind your answer is subjectivist, if your answer is “nothing” than you are a non-cognitivist or an error theorist. All those camps are anti-realist camps. Objectivist answers include “the Categorical Imperative” and “immorality supervenes on human suffering”.
Is there accessible discussion out there of why one might expect real world correlation between objectivity and universality.
I see that subjective universalism is logically coherent, but I wouldn’t expect it to be true—it seems like too much of a coincidence that nothing objective requires people have the same beliefs, yet people do anyway.
Lots of things can cause a convergence of belief other than objective truth, e.g. adaptive fitness. But certainly, objectivity usually implies universality.