I was talking about “extended family values” in the sense of “it is good for families to stick together and spend time with each other”; this preference can (and often does) apply to other families as well. I see no analogue for that with slavery.
It seems that at least some people valued slavery in the sense of wanting to preserve a culture and way of life that included slavery. The following quotes from https://www.battlefields.org/learn/articles/why-non-slaveholding-southerners-fought seem to strongly suggest that slavery/racism (it seems hard to disentangle these) was a terminal value at least for some (again assuming moral anti-realism):
For Harris, the choice was clear. Mississippi would “rather see the last of her race, men, women, and children, immolated in one common funeral pyre than see them subjugated to the degradation of civil, political and social equality with the negro race.”
What Southerner, Hale asked, “can without indignation and horror contemplate the triumph of negro equality, and see his own sons and daughters in the not distant future associating with free negroes upon terms of political and social equality?”
Non-slaveholders, he predicted, were also in danger. “It will be to the non-slaveholder, equally with the largest slaveholder, the obliteration of caste and the deprivation of important privileges,” he cautioned. “The color of the white man is now, in the South, a title of nobility in his relations as to the negro,” he reminded his readers. “In the Southern slaveholding States, where menial and degrading offices are turned over to be per formed exclusively by the Negro slave, the status and color of the black race becomes the badge of inferiority, and the poorest non-slaveholder may rejoice with the richest of his brethren of the white race, in the distinction of his color. He may be poor, it is true; but there is no point upon which he is so justly proud and sensitive as his privilege of caste; and there is nothing which he would resent with more fierce indignation than the attempt of the Abolitionist to emancipate the slaves and elevate the Negroes to an equality with himself and his family.”
Back to you:
But yeah, you could argue that racism can be a terminal value, and that slave owners would develop it, as a justification for what might have started as an instrumental value.
What scares me is the possibility that moral anti-realism is false, but we build an AI under the assumption that it’s true, and it “synthesizes” or “learns” or “extrapolates” some terminal value like or analogous to racism, which turns out to be wrong.
What scares me is the possibility that moral anti-realism is false, but we build an AI under the assumption that it’s true
One way of dealing with this, in part, is to figure out what would convince you that moral realism was true, and put that in as a strong conditional meta-preference.
I can see two possible ways to convince me that moral realism is true:
I spend hundreds or more years in a safe environment with a bunch of other philosophically minded people and we try to come up with arguments for and against moral realism, counterarguments, counter-counterarguments and so on, and we eventually exhaust the space of such arguments and reach a consensus that moral realism is true.
We solve metaphilosophy, program/teach an AI to “do philosophy”, somehow reach high confidence that we did that correctly, and the AI solves metaethics and gives us a convincing argument that moral realism is true.
Do these seem like things that could be “put in as a strong conditional meta-preference” in your framework?
Do these seem like things that could be “put in as a strong conditional meta-preference” in your framework?
Yes, very easily.
The main issue is whether these should count as an overwhelming meta-preference—one that over-weights all other considerations. And, currently as I have things set up, the answer is no. I have no doubt that you feel strongly about potentially true moral realism. But I’m certain that this strong feeling is not absurdly strong compared to other preferences at other moments in your life. So if we synthesised your current preferences, and 1. or 2. ended up being true, then the moral realism would end up playing a large-but-not-dominating role in your moral preferences.
I wouldn’t want to change that, because what I’m aiming for is an accurate synthesis of your current preferences, and your current preference for moral-realism-if-it’s-true is not, in practice, dominating your preferences. If you wanted to ensure the potential dominance of moral realism, you’d have to put that directly into the synthesis process, as a global meta-preference (section 2.8 of the research agenda).
But the whole discussion feels a bit peculiar, to me. One property of moral realism that is often assumed, is that it is, in some sense, ultimately convincing—that all systems of morality (or all systems derived from humans) will converge to it. Yet when I said a “large-but-not-dominating role in your moral preferences”, I’m positing that moral realism is true, but that we have a system of morality - UH - that does not converge to it. I’m not really grasping how this could be possible (you could argue that the moral realism UR is some sort of acausal trade convergent function, but that gives an instrumental reason to follow UR, not an actual reason to have UR; and I know that a moral system need not be a utility function ^_^).
So yes, I’m a bit confused by true-but-not-convincing moral realisms.
It seems that at least some people valued slavery in the sense of wanting to preserve a culture and way of life that included slavery. The following quotes from https://www.battlefields.org/learn/articles/why-non-slaveholding-southerners-fought seem to strongly suggest that slavery/racism (it seems hard to disentangle these) was a terminal value at least for some (again assuming moral anti-realism):
Back to you:
What scares me is the possibility that moral anti-realism is false, but we build an AI under the assumption that it’s true, and it “synthesizes” or “learns” or “extrapolates” some terminal value like or analogous to racism, which turns out to be wrong.
One way of dealing with this, in part, is to figure out what would convince you that moral realism was true, and put that in as a strong conditional meta-preference.
I can see two possible ways to convince me that moral realism is true:
I spend hundreds or more years in a safe environment with a bunch of other philosophically minded people and we try to come up with arguments for and against moral realism, counterarguments, counter-counterarguments and so on, and we eventually exhaust the space of such arguments and reach a consensus that moral realism is true.
We solve metaphilosophy, program/teach an AI to “do philosophy”, somehow reach high confidence that we did that correctly, and the AI solves metaethics and gives us a convincing argument that moral realism is true.
Do these seem like things that could be “put in as a strong conditional meta-preference” in your framework?
Yes, very easily.
The main issue is whether these should count as an overwhelming meta-preference—one that over-weights all other considerations. And, currently as I have things set up, the answer is no. I have no doubt that you feel strongly about potentially true moral realism. But I’m certain that this strong feeling is not absurdly strong compared to other preferences at other moments in your life. So if we synthesised your current preferences, and 1. or 2. ended up being true, then the moral realism would end up playing a large-but-not-dominating role in your moral preferences.
I wouldn’t want to change that, because what I’m aiming for is an accurate synthesis of your current preferences, and your current preference for moral-realism-if-it’s-true is not, in practice, dominating your preferences. If you wanted to ensure the potential dominance of moral realism, you’d have to put that directly into the synthesis process, as a global meta-preference (section 2.8 of the research agenda).
But the whole discussion feels a bit peculiar, to me. One property of moral realism that is often assumed, is that it is, in some sense, ultimately convincing—that all systems of morality (or all systems derived from humans) will converge to it. Yet when I said a “large-but-not-dominating role in your moral preferences”, I’m positing that moral realism is true, but that we have a system of morality - UH - that does not converge to it. I’m not really grasping how this could be possible (you could argue that the moral realism UR is some sort of acausal trade convergent function, but that gives an instrumental reason to follow UR, not an actual reason to have UR; and I know that a moral system need not be a utility function ^_^).
So yes, I’m a bit confused by true-but-not-convincing moral realisms.