“Very Bad Sign #243: Many people all agree that a statement is definitely true but they can’t agree on why and/or what the statement means.”
Do you mean very bad for the person rejecting the statement? Such a consensus would seem to suggest that there are many reasons for affirming its truth, and many different ways it can be interpreted which are still true.
Michael Vassar: instead of arguing about the meaning of “honest” or “dishonest”, do you think it is possible for a person to know by introspection whether or not he has “really been trying” to get at the truth about something or not?
If it is, then people still shouldn’t disagree: the one who knows that he hasn’t been trying to get at the truth should just admit it, and accept the position of the other guy as more reasonable.
If it isn’t, then your account does not supply an argument against Robin Hanson (which I take it you thought that it does.)
Peter Turney: yes, I define Occam’s Razor in such a way that all orderings of the hypotheses are Occamian.
The razor still cuts, because in real life, a person must choose some particular ordering of the hypotheses. And once he has done this, the true hypothesis must fall relatively early in the series, namely after a finite number of other hypotheses, and before an infinite number of other hypotheses. The razor cuts away this infinite number of hypotheses and leaves a finite number.
In fact, an anti-Occam prior is impossible. As I’ve mentioned before, as long as you’re talking about anything that has any remote resemblance to something we might call simplicity, things can decrease in simplicity indefinitely, but there is a limit to increase. In other words, you can only get so simple, but you can always get more complicated. So if you assign a one-to-one correspondence between the natural numbers and potential claims, it follows of necessity that as the natural numbers go to infinity, the complexity of the corresponding claims goes to infinity as well. And if you assign a probability to each claim, while making your probabilities sum to 1, then the probability of the more and more complex claims will go to 0 in the limit.
In other words, Occam’s Razor is a logical necessity.
I don’t see why we shouldn’t admit that the more times the sun rises, the more likely it is not to rise the next time… In general life insurance agencies assume that the more days you have lived, the more likely you are to die the next day.
First interpretation is true; third interpretation is partially true. Second interpretation false, fourth interpretation mainly false (because people can execute a process which will change their desires in some unforeseeable manner.)
I’ve thought about Space Cannibals and the like before (i.e. creatures that kill one of the sexes during sexual reproduction). My suspicion is that even if such creatures evolved and survived, by the time they had a civilization, many would be saying to one another, “There really should be a better way...”
Evidence for this is the fact that even now, there are many human beings claiming it is wrong to kill other animals, despite the fact that humans evolved to kill and eat other animals. Likewise, in the ancestral environment, various tribes usually did kill each other rather than cooperate. But this didn’t stop them from beginning to cooperate. So I suspect that Space Cannibals would do something similar. And in any case, I would fully admit that murder couldn’t in fact be wrong for the Space Cannibals in the same way it is for us, even if there is an external moral truth.
In answer to Robin’s question, assuming that morality exists, it probably has a number of purposes. And if one of the purposes is to preserve things in existence (i.e. moral truths correspond roughly with what is necessary to preserve things), then of course there will be a selection pressure to perceive moral truth. The disclaimer should not be needed, but this is not in any way a claim that it is moral to maximize inclusive genetic fitness.
Eliezer: as you are aware yourself, we don’t know how to compute it, nor how to run a computation that computes how to compute it. If we leave it up to the superintelligence to decide how to interpret “helping” and “hurting,” it will be in a position no worse than our own, and possibly better, seeing that we are not superintelligent.
“Fuzzle” = “Morally right.”
Only in terms of how this actually gets into a human mind, there is a dynamic first: before anyone has any idea of fuzzleness, things are already being sent to the action system. Then we say, “Oh, these are things are fuzzle!”, i.e. these are the type of things that get sent to the action system. Then someone else tells us that something else is fuzzle, and right away it gets sent to the action system too.
There’s no particular need to renew the torture and dust specks debate, so I’ll just point out that GBM, Nominull, Ian C., and Manon de Gaillande have all made similar points: if you say, “if there is an external objective morality that says you should kill babies, why should you listen?” the question is the same as “if you should kill babies, why should you do it?”
Yes, and if 2 and 2 make 5, why should I admit it?
It isn’t in fact true that I should kill babies, just as 2 and 2 don’t make 5. But if I found out that 2 and 2 do make 5, of course I should admit it, and if I found out that I should kill babies, of course I should do it. As Nominull says, Eliezer’s objection to this is an objection to reason itself: if an argument establishes conclusively something you happen not to like, you should reject the conclusion.
“I mean… if an external objective morality tells you to kill babies, why should you even listen?”
This is an incredibly dangerous argument. Consider this : “I mean… if some moral argument, whatever the source, tells me to prefer 50 years of torture to any number of dust specks, why should I even listen?”
And we have seen many who literally made this argument.
I just had another idea: maybe I would begin to design an Unfriendly AI. After all, being an evil genius would at least be fun, and besides, it would be a way to get revenge on Eliezer for proving that morality doesn’t exist.
It seems people are interpreting the question in two different ways, one that we don’t have any desires any more, and therefore no actions, and the other in the more natural way, namely that “moral philosophy” and “moral claims” have no meaning or are all false. The first way of interpreting the question is useless, and I guess Eliezer intended the second.
Most commenters are saying that it would make no difference to them. My suspicion is that this is true, but mainly because they already believe that moral claims are meaningless or false.
Possibly (I am not sure of this) Eliezer hopes that everyone will answer in this way, so that he can say that morality is unnecessary.
Personally, I agree with Dynamically Linked. I would start out by stealing wallets and purses, and it would just go downhill from there. In other words, if I didn’t believe that such things were wrong, the bad feeling that results from doing them, and the idea that it hurts people, wouldn’t be strong enough to stop me, and once I got started, the feeling would go away too—this much I know from the experience of doing wrong. And once I had changed the way I feel about these things, the way I feel about other things (too horrible to mention at the moment) would begin to change too. So I can’t really tell where it would end, but it would be bad (according to my present judgment).
There are others who would follow or have followed the same course. TGGP says that over time his life did change after he ceased to believe in morality, and at one point he said that he would torture a stranger to avoid stubbing his toe, which presumably he would not have done when he believed in morality.
So if it is the case that Eliezer hoped that morality is unnecessary to prevent such things, his hope is in vain.
If the Neanderthals or some similar species had survived until the present day, presumably there would be a similar case. Probably Neanderthals had brainware quite different from ours (although quite possibly not as different as the difference between women and men, since this difference goes back a lot longer.)
TGGP, the evidence is that Eliezer suggested the reason to avoid this error is to avoid converting to Christianity. Presumably the real reason to avoid the error (if it is one, which he hasn’t shown convincingly yet) is to avoid turning the universe into paperclips.
In regard to AIXI: One should consider more carefully the fact that any self-modifying AI can be exactly modeled by a non-self modifying AI.
One should also consider the fact that no intelligent being can predict its own actions—this is one of those extremely rare universals. But this doesn’t mean that it can’t recognize itself in a mirror, despite its inability to predict its actions.
Just to be clear, as far as I can remember after reading every post on OB, no one else has posted specifically under the title “Unknown.” So there’s only one of me.
Prase, I think I would agree with that. But it seems Eliezer isn’t quite seeing is that even if mind-space in general is completely arbitrary, people programming an AI aren’t going to program something completely arbitrary. They’re going to program it to use assumptions and ways of argument that they find acceptable, and so it will also draw conclusions that they find acceptable, even if it does this better than they do themselves.
Also, Eliezer’s conclusion, “And then Wright converted to Christianity—yes, seriously. So you really don’t want to fall into this trap!” seems to suggest that a world where the AI converts everyone to Christianity is worse than a world that the AI fills with paperclips, by suggesting that converting to Christianity is the worst thing that can happen to you. I wonder if Eliezer really believes this, and would rather be made into paperclips than into a Christian?
Excuse me, that’s 42, not 49.
Poke, if anyone could give credence to the idea that “science is just another religion”, it would be you.