Because Eliezer made the ingenious move of redefining “should” to mean “do what we prefer”.
He doesn’t define it indexically (like this) or in terms of humans; as I understand it, he defines it in terms of an objective computation that happens to be instantiated by humans (No License to be Human).
He doesn’t define it indexically (like this) or in terms of humans; as I understand it, he defines it in terms of an objective computation that happens to be instantiated by humans (No License to be Human).