If the king was a dictator and forced everyone to torture innocent people, it would still be against my morals to torture people, regardless of whether I had to do it or not. I can’t decide to adopt the king’s moral weights, no matter how much it may assuage my guilt. This is what I mean when I say it is not possible to get moral weights from an outside source. I may be playing by the king’s rules, but only because I value my life above all else, and it’s drowning out the rest of my utility function.
On a related note, is this an example of a intrapersonal utility monster? All my goals are being thrown under the bus except for one, which I value most highly.
Your example of the King who wants you to torture is extreme, and doesnt generalize … you have set up not torturing as a non-negotiable absolute imperative. A more steelmanned case would be compromising on negotiable principles at the behest of society at large.
If the king was a dictator and forced everyone to torture innocent people, it would still be against my morals to torture people, regardless of whether I had to do it or not. I can’t decide to adopt the king’s moral weights, no matter how much it may assuage my guilt. This is what I mean when I say it is not possible to get moral weights from an outside source. I may be playing by the king’s rules, but only because I value my life above all else, and it’s drowning out the rest of my utility function.
On a related note, is this an example of a intrapersonal utility monster? All my goals are being thrown under the bus except for one, which I value most highly.
Your example of the King who wants you to torture is extreme, and doesnt generalize … you have set up not torturing as a non-negotiable absolute imperative. A more steelmanned case would be compromising on negotiable principles at the behest of society at large.