Just to check, surely you’re not saying an extrapolated version of Archimedes and a thousand people who agreed with him wouldn’t have turned out OK?
It seems to me that we have some quite strong evidence against rotten bastards theory in that intelligent and well-informed people IRL seem to converge away from bastardly beliefs. Still, rotten bastards theory seems worth thinking about for negative-utilitarian reasons.
On second thought I suppose it could mean “being a jerk according to ‘ethics’”, where “ethics” is conceived not as something intrinsically moral but as a practical way for agents with different moralities to coordinate on a mutually acceptable solution.
“Being a jerk” here means “being a jerk according to other people’s notion of morality, but not according to my own notion of morality”, right?
I sort of take offense to “we’re the ancient Greeks”; I make sure to disagree with Western morality whenever it’s wrong, and I have no reason to believe the resulting distribution of errors is biased toward agreement with Western morality. If you meant to say “most of them are the ancient Greeks”, then sure.
Supposedly genetics allows for people of different ages or sexes to have different mental machinery, whereas individual genetic differences just represent low-complexity differences in tweaking. I’m not sure why that makes Eliezer’s point though, if the aforementioned differences in tweaking mean different complex machinery gets activated. Cultural differences I’d expect to wash out just through people learning about different cultures that they could have grown up in.
ZM, he means after volition-extrapolating.
I must be missing something—why would you advocate something that you know you can’t justify to anyone else?
I wonder if the distinction between 1) something implementing the same dynamic as a typical human but mistaken about what it says and 2) something implementing a completely different dynamic and not mistaken about anything, is the same as the distinction people normally make between 1) immoral and 2) amoral.
HA: To me this sums up to you picking positions for personal status enhancement rather than for solving the challenges we face. I understand why that would be salient for a non-anonymous blogger. I hope you at least do your best to address them anonymously. Otherwise we could be left with a tragedy of the future outcomes commons, with all the thinkers vying for status over maximizing our future outcomes.
If you were already blogging and started an anonymous blog, how would you avoid giving away your identity in your anonymous blog through things like your style of thinking, or the sort of background justifications you use? It doesn’t seem to me like it could be done.
There’s no disputing (we may suppose) that abortion is indeed prohibited by morality_Bob.
This isn’t the clearest example, because it seems like abortion is one of those things everyone would come to agree on if they knew and understood all the arguments. A clearer example is a pencil-maximizing AI vs a paperclip-maximizing AI. Do you think that these two necessarily disagree on any facts? I don’t.
Presumably anti-natalists would deny the need to carry on the species because they expect the negative value of future suffering to outweigh the positive value of future happiness, truth, etc.
It seems to me human life has value insofar as dead people can’t be happy, discover truth, and so on; but not beyond that.
Also I’d like to second TGGP’s question.
This looks like it estimates order-of-magnitude 100M deaths from full-scale US-Soviet nuclear war. I guess future arms races might be much worse though.
I’m always wary of claims of type “if we do otherwise moral thing X someone else may react by committing horrific crime Y, so we should not do X”; at the least we should be very explicit about where moral blame lies, and at the most we should discount Y because of game-theoretical incentives (example: not paying off kidnappers). Still, it seems like this doesn’t much affect Eliezer’s point here.
By the way, I’m always nervous about persuading people of impopular political positions when I agree with them on far more important non-political issues; they’ll probably lose social standing.
Do you devote a significant amount of your time and resources to making paperclips, given the possibility that you’re being simulated?
Keeping everyone alive would not take a significant amount of a paperclip maximizer’s time and resources. (Though for utilitarians this probably means it doesn’t count.) But the key difference is this: human-like goal systems seem like they will gain access to lots more simulation resources than paperclip maximizers or some other specific human-indifferent goal system (the set of all human-indifferent goal systems together is a different matter, but they’re not a coherent bloc).
I guess “game theory move” doesn’t make much sense; it should have read “given the possibility that it’s being simulated”.
Now that you mention it, I’m actually not 100% sure that a paperclip maximizer wouldn’t give humans some fraction of computing resources as some sort of very cheap game theory move.
Oh, and my ants say they’re offended.
Nick, it seems like if you discounted for algorithmic complexity that should imply some way to discount for extreme distances in space, yes. Not sure how exactly.
Carl, that’s not a feature of all bounded utility functions. If you discount exponentially for time (or more plausibly for the algorithmic complexity of locating an observer-moment in the universe, and I might want to do this for reasons independent of Pascal) then the value of something does not depend on how much of that something is already in existence, but it still seems like you couldn’t get to infinite utilities.
My argument for bounded utility is this: if a mind is so far into the future that specifying where it is takes as much information input as generating the mind from scratch, in what sense is it more present-in-the-world than the mind-from-scratch is?
I’m not Eliezer nor am I a pro, but I think I agree with Eliezer’s account, and as a first attempt I think it’s something like this...
When X judges that Y should Z, X is judging that Z is the solution to the problem W, where W is a rigid designator for the problem structure implicitly defined by the machinery shared by X and Y which they both use to make desirability judgments. (Or at least X is asserting that it’s shared.) Due to the nature of W, becoming informed will cause X and Y to get closer to the solution of W, but wanting-it-when-informed is not what makes that solution moral.