I think the assumption it that human-like morality isn’t universally privileged.
Human morality has been shaped by evolution in the ancestral environment. Evolution in a different environment would create a mind with different structures and behaviours.
In other words, a full specification of human morality is sufficiently complex that it is unlikely to be spontaneously generated.
In other words, there is no compact specification of an AI that would do what humans want, even when on an alien world with no data about humanity. An AI could have a pointer at human morality with instructions to copy it. There are plenty of other parts of the universe it could be pointed to, so this is far from a default.
Imagine a device that looks like a calculator. When you type 2+2, you get 7. You could conclude its a broken calculator, or that arithmetic is subjective, or that this calculator is not doing addition at all. Its doing some other calculation.
Imagine a robot doing something immoral. You could conclude that its broken, or that morality is subjective, or that the robot isn’t thinking about morality at all.
These are just different ways to describe the same thing.
Addition has general rules. Like a+b=b+a. This makes it possible to reason about. Whatever the other calculator computes may follow this rule, or different rules, or no simple rules at all.
I think the assumption it that human-like morality isn’t universally privileged.
Human morality has been shaped by evolution in the ancestral environment. Evolution in a different environment would create a mind with different structures and behaviours.
In other words, a full specification of human morality is sufficiently complex that it is unlikely to be spontaneously generated.
In other words, there is no compact specification of an AI that would do what humans want, even when on an alien world with no data about humanity. An AI could have a pointer at human morality with instructions to copy it. There are plenty of other parts of the universe it could be pointed to, so this is far from a default.
But reasoning about morality? Is that a space with logic or with anything goes?
Imagine a device that looks like a calculator. When you type 2+2, you get 7. You could conclude its a broken calculator, or that arithmetic is subjective, or that this calculator is not doing addition at all. Its doing some other calculation.
Imagine a robot doing something immoral. You could conclude that its broken, or that morality is subjective, or that the robot isn’t thinking about morality at all.
These are just different ways to describe the same thing.
Addition has general rules. Like a+b=b+a. This makes it possible to reason about. Whatever the other calculator computes may follow this rule, or different rules, or no simple rules at all.
Not to the extent that there’s no difference at all...you can exclude some of them on further investigation.