In particular, one way (the way?) his metaethics might spit up something that looks a lot like moral realism is if there is strong convergence of values upon agents receiving better information, time enough to work out contradictions in their values, etc. But the “strong convergence of values” thesis hasn’t really been argued, so I remain unclear as to why Eliezer finds it plausible.
When you say “agents” here, did you mean to say “psychologically normal humans”? Because the general claim I think Eliezer would reject, based on what he says on No Universally Compelling Arguments. But I do think he would accept the narrower claim about psychologically normal humans, or as he sometimes says “neurologically intact humans.” And the argument for that is found in places like The Psychological Unity of Humankind, though I think there’s an even better link for it somewhere—I seem to distinctly remember a post where he says something about how you should be very careful about attributing moral disagreements to fundamentally different values.
EDIT: Here is the other highly relevant post I was thinking of.
Yeah, I meant to remain ambiguous about how wide Eliezer means to cast the net around agents. Maybe it’s psychologically normal humans, maybe it’s wider or narrower than that.
Some of the sources you are hand waving towards are (quite rightly) pointing out that rational agents need not converge, but they aren’t looking at the empirical question of whether humans, specifically, converge. Only a subset of those sources are actually talking about humans specifically.
(^This isn’t disagreement. I agree with your main suggestion that humans probably don’t converge, although I do think they are at least describable by mono-modal distributions)
I’m not sure it’s even appropriate to use philosophy to answer this question. The philosophical problem here is “how do we apply idealized constructs like extrapolated preference and terminal values to flesh-and-blood animals?” Things like “should values which are not biologically ingrained count as terminal values?” and similar questions.
...and then, once we’ve developed constructs to the point that we’re ready to talk about the extent to which humans specifically converge if at all, it becomes an empirical question..
No Universally Compelling Arguments has been put to me as a decisive refutation of Moral Realism, by somebody who thought the LW line was anti-realist. It isn’t a decisive refutation because no (non strawman) realist thinks there are arguments that could compel an irrational person, an insane person, an very unintelligent person, and so on. Moral realists only need to argue that moral truths are independently discoverable by suitably motivated and equipped people, like mathematical truths (etc).
When you say “agents” here, did you mean to say “psychologically normal humans”? Because the general claim I think Eliezer would reject, based on what he says on No Universally Compelling Arguments.
Well, “No Universally Compelling Arguments” also applies to physics, but it is generally believed that all sufficiently intelligent agents would agree on the laws of physics.
True, but physics is discoverable via the scientific method, and ultimately, in the nastiest possible limit, via war. If we disagree on physics, all we have to do is highlight the disagreement and go to war over it: whichever one of us is closer to right will succeed in killing the other guy (and potentially a hell of a lot of other stuff).
Whereas if you try going to war over morality, everyone winds up dead and you’ve learned nothing, except possibly that almost everyone considers a Hobbesian war-of-all-against-all to be undesirable when it happens to him.
EDIT: Here is the other highly relevant post I was thinking of.
I think what he is talking about there is lack of disagreement in the sense of incommensurability, or orthogonality as it is locally known. Lack of disagreement int he sense of convergence or consensus is a very different thing.
When you say “agents” here, did you mean to say “psychologically normal humans”? Because the general claim I think Eliezer would reject, based on what he says on No Universally Compelling Arguments. But I do think he would accept the narrower claim about psychologically normal humans, or as he sometimes says “neurologically intact humans.” And the argument for that is found in places like The Psychological Unity of Humankind, though I think there’s an even better link for it somewhere—I seem to distinctly remember a post where he says something about how you should be very careful about attributing moral disagreements to fundamentally different values.
EDIT: Here is the other highly relevant post I was thinking of.
Yeah, I meant to remain ambiguous about how wide Eliezer means to cast the net around agents. Maybe it’s psychologically normal humans, maybe it’s wider or narrower than that.
I suppose ‘The psychological unity of humankind’ is sort of an argument that value convergence is likely at least among humans, though it’s more like a hand-wave. In response, I’d hand-wave toward Sobel (1999); Prinz (2007); Doring & Steinhoff (2009); Doring & Andersen (2009); Robinson (2009); Sotala (2010); Plunkett (2010); Plakias (2011); Egan (2012), all of which argue for pessimism about value convergence. Smith (1994) is the only philosophical work I know of that argues for optimism about value convergence, but there are probably others I just don’t know about.
Some of the sources you are hand waving towards are (quite rightly) pointing out that rational agents need not converge, but they aren’t looking at the empirical question of whether humans, specifically, converge. Only a subset of those sources are actually talking about humans specifically.
(^This isn’t disagreement. I agree with your main suggestion that humans probably don’t converge, although I do think they are at least describable by mono-modal distributions)
I’m not sure it’s even appropriate to use philosophy to answer this question. The philosophical problem here is “how do we apply idealized constructs like extrapolated preference and terminal values to flesh-and-blood animals?” Things like “should values which are not biologically ingrained count as terminal values?” and similar questions.
...and then, once we’ve developed constructs to the point that we’re ready to talk about the extent to which humans specifically converge if at all, it becomes an empirical question..
No Universally Compelling Arguments has been put to me as a decisive refutation of Moral Realism, by somebody who thought the LW line was anti-realist. It isn’t a decisive refutation because no (non strawman) realist thinks there are arguments that could compel an irrational person, an insane person, an very unintelligent person, and so on. Moral realists only need to argue that moral truths are independently discoverable by suitably motivated and equipped people, like mathematical truths (etc).
Well, “No Universally Compelling Arguments” also applies to physics, but it is generally believed that all sufficiently intelligent agents would agree on the laws of physics.
True, but physics is discoverable via the scientific method, and ultimately, in the nastiest possible limit, via war. If we disagree on physics, all we have to do is highlight the disagreement and go to war over it: whichever one of us is closer to right will succeed in killing the other guy (and potentially a hell of a lot of other stuff).
Whereas if you try going to war over morality, everyone winds up dead and you’ve learned nothing, except possibly that almost everyone considers a Hobbesian war-of-all-against-all to be undesirable when it happens to him.
I think what he is talking about there is lack of disagreement in the sense of incommensurability, or orthogonality as it is locally known. Lack of disagreement int he sense of convergence or consensus is a very different thing.