I suspect it gets worse. Eliezer seems to lean heavily on the psychological unity of humankind, but there’s a lot of room for variance within that human dot. My morality is a human morality, but that doesn’t mean I’d agree with a weighted sum across all possible extrapolated human moralities. So even if you preserve human morals and metamorals, you could still end up with a future we’d find horrifying (albeit better than a paperclip galaxy). It might be said that that’s only a Weirdtopia, that’s you’re horrified at first, but then you see that it’s actually for the best after all. But if “the utility function [really] isn’t up for grabs,” then I’ll be horrified for as long as I damn well please.
I’ll be horrified for as long as I damn well please.
Well, okay, but the Weirdtopia thesis under consideration makes the empirical falsifiable prediction that “as long as you damn well please” isn’t actually a very long time. Also, I call scope neglect: your puny human brain can model some aspects of your local environment, which is a tiny fraction of this Earth, but you’re simply not competent to judge the entire future, which is much larger.
I suspect it gets worse. Eliezer seems to lean heavily on the psychological unity of humankind, but there’s a lot of room for variance within that human dot. My morality is a human morality, but that doesn’t mean I’d agree with a weighted sum across all possible extrapolated human moralities. So even if you preserve human morals and metamorals, you could still end up with a future we’d find horrifying (albeit better than a paperclip galaxy). It might be said that that’s only a Weirdtopia, that’s you’re horrified at first, but then you see that it’s actually for the best after all. But if “the utility function [really] isn’t up for grabs,” then I’ll be horrified for as long as I damn well please.
Well, okay, but the Weirdtopia thesis under consideration makes the empirical falsifiable prediction that “as long as you damn well please” isn’t actually a very long time. Also, I call scope neglect: your puny human brain can model some aspects of your local environment, which is a tiny fraction of this Earth, but you’re simply not competent to judge the entire future, which is much larger.
I would like to point out that you’re probably replying to your past self. This gives me significant amusement.