When you say “agents” here, did you mean to say “psychologically normal humans”? Because the general claim I think Eliezer would reject, based on what he says on No Universally Compelling Arguments.
Well, “No Universally Compelling Arguments” also applies to physics, but it is generally believed that all sufficiently intelligent agents would agree on the laws of physics.
True, but physics is discoverable via the scientific method, and ultimately, in the nastiest possible limit, via war. If we disagree on physics, all we have to do is highlight the disagreement and go to war over it: whichever one of us is closer to right will succeed in killing the other guy (and potentially a hell of a lot of other stuff).
Whereas if you try going to war over morality, everyone winds up dead and you’ve learned nothing, except possibly that almost everyone considers a Hobbesian war-of-all-against-all to be undesirable when it happens to him.
Well, “No Universally Compelling Arguments” also applies to physics, but it is generally believed that all sufficiently intelligent agents would agree on the laws of physics.
True, but physics is discoverable via the scientific method, and ultimately, in the nastiest possible limit, via war. If we disagree on physics, all we have to do is highlight the disagreement and go to war over it: whichever one of us is closer to right will succeed in killing the other guy (and potentially a hell of a lot of other stuff).
Whereas if you try going to war over morality, everyone winds up dead and you’ve learned nothing, except possibly that almost everyone considers a Hobbesian war-of-all-against-all to be undesirable when it happens to him.