Sure, existing people tend to have such preferences. But hypothetically it’s possible that they didn’t, and the mere possibility is enough to bring down an ethical theory if you can show that it would generate absurd results.
This might be one reason why Eliezer talks about morality as a fixed computation.
P.S. Also, doesn’t the being itself have a preference for not-suffering?
Sure, existing people tend to have such preferences. But hypothetically it’s possible that they didn’t, and the mere possibility is enough to bring down an ethical theory if you can show that it would generate absurd results.
This might be one reason why Eliezer talks about morality as a fixed computation.
P.S. Also, doesn’t the being itself have a preference for not-suffering?