They could, but they don’t have to disagree about any observation to do that. ‘Should’ statements cannot be made from ‘is’ statements. Confusingly, ‘is (morally) right’ is a ‘should’ statement, not an ‘is’ statement.
‘Should’ statements cannot be made from ‘is’ statements.
Do you notice the difficulty in your own statement there?
If I say, “We should derive ‘should’ statements from ‘is’ statements”, you can’t refute my should statement; you can only contradict it. You might try to prove it impossible to derive ‘should’ from ‘is’—but even assuming you succeed, proving an impossibility is by your own statement proving only what is, not what should be.
“Hume’s Guillotine” always cuts itself in half first.
You’re right, in that I can’t refute the core statement of a system of ethics.
Perhaps genies should grant wishes- but developing a system that creates a moral imperative for genies to grant wishes doesn’t make genies or grant wishes. Even if you believe that it is morally right and proper to build perpetual motion machines, you don’t actually get to build perpetual motion machines.
Okay. Now take step two—try to show that, in fact, a ‘should’ really cannot be derived from an ‘is’.
“Perpetual motion machines cannot be built” can be demonstrated to be true based on empirically-observable facts. If “‘Should’ statements cannot be made from ‘is’ statements” is a true ‘is’ statement, it will also be possible to show it is true based entirely on empirically-observable facts, right?
The usual mistake people make at this point is to claim that various “shoulds” contradict what “is”. But what people think should be is not proof of what is. No matter how hard people believe genies should give wishes, it won’t bring them into existence. What people believe morality should say doesn’t prove what morality is.
(Unless, of course, you argue that morality is whatever people say it should be. But then you’re deriving your should—morality—from what is—what people say.)
Nope. Incompleteness shows that there are some statements which are true which cannot be proven to be true.
However, empirically observed facts in the absence of moral imperative do not create a moral imperative. Typically ethics are formed around by a value judgement and then molded and polished by facts. I see that you are trying to trap me by saying that “I believe that this is better” is a fact, rather than allowing the value judgement “This is better” to stand.
Morality is, among other things, subjective. There is no basis in fact to prefer any system over any other system, any more than there is a basis in fact to prefer one genre of movies over another. I prefer internal consistency to internal inconsistency, and I believe that the majority of people who tend to think things through also prefer that, but I have no factual basis for that preference.
Claiming that falling down (as opposed to up) is a moral act, while not technically refutable, is hard to swallow.
They would do well to taboo the vague “right” and maybe discuss what their respective extrapolated volitions are, whether “maximize pencils” is actually pencil-maximizer’s preference or an unfortunate result of a hardware glitch which on reflection should be corrected, what constitutes a “pencil” for the purpose of proper pencil-maximization, what is the proper attitude towards risk (dependence of utility on the number of pencils can take many shapes, even if it’s a given that it’s monotonic and depends on nothing else), etc.
If they were both conceptual beings, couldn’t they argue about whether it “is right” to maximize pencils or paper clips?
They could, but they don’t have to disagree about any observation to do that. ‘Should’ statements cannot be made from ‘is’ statements. Confusingly, ‘is (morally) right’ is a ‘should’ statement, not an ‘is’ statement.
Do you notice the difficulty in your own statement there?
If I say, “We should derive ‘should’ statements from ‘is’ statements”, you can’t refute my should statement; you can only contradict it. You might try to prove it impossible to derive ‘should’ from ‘is’—but even assuming you succeed, proving an impossibility is by your own statement proving only what is, not what should be.
“Hume’s Guillotine” always cuts itself in half first.
You’re right, in that I can’t refute the core statement of a system of ethics.
Perhaps genies should grant wishes- but developing a system that creates a moral imperative for genies to grant wishes doesn’t make genies or grant wishes. Even if you believe that it is morally right and proper to build perpetual motion machines, you don’t actually get to build perpetual motion machines.
Okay. Now take step two—try to show that, in fact, a ‘should’ really cannot be derived from an ‘is’.
“Perpetual motion machines cannot be built” can be demonstrated to be true based on empirically-observable facts. If “‘Should’ statements cannot be made from ‘is’ statements” is a true ‘is’ statement, it will also be possible to show it is true based entirely on empirically-observable facts, right?
The usual mistake people make at this point is to claim that various “shoulds” contradict what “is”. But what people think should be is not proof of what is. No matter how hard people believe genies should give wishes, it won’t bring them into existence. What people believe morality should say doesn’t prove what morality is.
(Unless, of course, you argue that morality is whatever people say it should be. But then you’re deriving your should—morality—from what is—what people say.)
Nope. Incompleteness shows that there are some statements which are true which cannot be proven to be true.
However, empirically observed facts in the absence of moral imperative do not create a moral imperative. Typically ethics are formed around by a value judgement and then molded and polished by facts. I see that you are trying to trap me by saying that “I believe that this is better” is a fact, rather than allowing the value judgement “This is better” to stand.
Morality is, among other things, subjective. There is no basis in fact to prefer any system over any other system, any more than there is a basis in fact to prefer one genre of movies over another. I prefer internal consistency to internal inconsistency, and I believe that the majority of people who tend to think things through also prefer that, but I have no factual basis for that preference.
Claiming that falling down (as opposed to up) is a moral act, while not technically refutable, is hard to swallow.
They would do well to taboo the vague “right” and maybe discuss what their respective extrapolated volitions are, whether “maximize pencils” is actually pencil-maximizer’s preference or an unfortunate result of a hardware glitch which on reflection should be corrected, what constitutes a “pencil” for the purpose of proper pencil-maximization, what is the proper attitude towards risk (dependence of utility on the number of pencils can take many shapes, even if it’s a given that it’s monotonic and depends on nothing else), etc.
...great. Now I’m wondering whether a paperclip made of lead could be called a pencil and thus reconcile their optimization processes.
Excellent idea. This might well be the result of negotiations between two optimizers, preferable to fighting it out for control of the resources.
Two optimizers might negotiate based on the reality of their goals and power, instead of fighting over who owns the label “right”? What crazy talk!
And don’t tell them. I don’t want them joining forces.