Eliezer, you might do well to thoroughly understand and consider Fare’s criticisms of you. He seems to be one of the few readers here who isn’t in awe of you, and has therefore managed nailed some fundamental mistakes you’re making about human values. Mistakes that have been bugging me for some time too, but that I haven’t been able to articulate, possibly because I’m too enchanted by your cleverness.
We don’t value strangers on par with our friends and family, let alone freaky baby-eating or orgy-having aliens. Furthermore, I don’t want to be altered in such a manner as to make me so empathetic that I give equal moral standing to strangers and kin. I believe THAT would make me less human. If you or an FAI alters me into such a state, you are not correctly extrapolating my volition, nor of who knows how many other people like me. Do you have an estimate of how many people like that there are? How did you come by such an estimate?
So anyway, if this happened in any real future, I have no doubt some star would soon get supernova’d—current star, Huygens, Happy Homeworld, Eater Homeworld, or Sol, in that order of likelihood. For these idealized humans inhabiting the uncanny valley of empathy that creates the whole contrived dilemma in the first place, who knows? Maybe the fact that a nova was what brought them there, and now they’re contemplating creating a supernova is some kind of clue. Maybe the definition of “non-sentient baby” can be stretched to the point where the story ends as a blowjob joke, but I doubt it. Also, the mechanics of exactly how other people’s pain affects the Happies haven’t really been examined. It sounds like they’re merely extrapolating the pain they think others must be feeling… given that they’ve had no scruples against engineering other sources of discomfort out of existance, why not engineer that out of existance too?
Simon: “Eliezer tries to derive his morality from human values”
I would correct the above to “Eliezer tries to derive his morality from stated human values.”
That’s where many of his errors come from. Everyone is a selfish bastard. But Eliezer cannot bring himself to believe it, and a good fraction of the sorts of people whose opinions get taken seriously can’t bring themselves to admit it.