Wow, you are actually compounding the rudeness of abusing the edit feature to completely rewrite your comment by then analyzing my response to the original version as if it were responding to the edited version.
No. That statement is three comments above the comment in which you said I should acknowledge my error. It was already there when you wrote that comment. And I also acknowledged my misstatement in the comment you were replying to, and elaborated on what I had meant when I made the comment.
I acknowledge that sometimes randomness can be a good enough approximate substitute for the much harder strategy of actually understanding the implications of a probability distribution.
Good! We agree.
Since U0 is my current utility function, and therefore (2) describes my current wants, you will not be able to convince me that I should be persuaded by (1), which is a meaningless comparison. Adopting U as my utility function does not help me maximize U0.
Good! We agree again.
To the extent that trilobites can even be considered to have utility functions, my utility function is better than the trilobite utility function according to my values. The trilobites would disagree.
And we agree yet again!
Likewise, an optimal trilobite CEV would be a trilobite SUCCESS and a human FAIL. There is no absolute universal utilility function that says one of these is better than the others. It is my human values that cause me to say that the human SUCCESS is better.
And here is where we part ways.
Maybe there is no universal utility function. That’s a… I won’t say it’s a reasonable position, but I understand its appeal. I would call it an over-reasoned position, like when a philosopher announces that he has proved that he doesn’t exist. It’s time to go back to the drawing board when you come up with that conclusion. Or at least to take your own advice, and stop trying to change the world when you’ve already said it doesn’t matter how it changes.
But to believe that your utility function is nothing special, and still try to take over the universe and force your utility function on it for all time, is insane.
(Yes, yes, I know Eliezer has all sorts of disclaimers in the CEV document about how CEV should not try to take over the universe. I don’t believe that it’s logically possible; and I believe that his discussions of Friendly AI make it even clearer that his plans require complete control. Perhaps the theory is still vague enough that just maybe there’s a way around this; but I believe the burden of proof is on those who say there is a way around it.)
It would be consistent with the theory of utility functions if, in promoting CEV, you were acting on an inner drive that said, “Ooh, baby, I’m ensuring the survival of my utility function. Oh, God, yes! Yes! YES!” But that’s not what I see. I see people scribbling equations, studying the answers, and saying, “Hmm, it appears that my utility function is directing me to propagate itself. Oh, dear, I suppose I must, then.”
That’s just faking your utility function.
I think it’s key that the people I’m speaking of who believe utility functions are arbitrary, also believe they have no free will. And it’s probably also key that they assume their utility function must assign value to its own reproduction. They then use these two beliefs as an excuse to justify not following through on their belief about the arbitrariness of their utility function, because they think to do so would be logically impossible. “We can’t help ourselves! Our utility functions made us do it!” I don’t have a clean analysis, but there’s something circular, something wrong with this picture.
No. That statement is three comments above the comment in which you said I should acknowledge my error.
Let’s recap. You made a wrong claim. I responded to the wrong claim. You disputed my response. I refuted your disputation. You attempted to defend your claim. I responded to your defense. You edited your defense by replacing it with the acknowledgment of your mistake. You responded to my response still sort of defending your wrong claim, and attacking me for refuting your wrong claim. I defended my refutation, pointing out the you really did make the wrong claim and continued to defend it. And now you attack my defense, claiming that you did in fact acknowledge your mistake, and this should somehow negate your continued defense after the acknowledgement. Do you see how you are wrong here? When you acknowledge your claim is wrong, you should not at the same time criticize me for refuting your point.
But to believe that your utility function is nothing special, and still try to take over the universe and force your utility function on it for all time, is insane.
I do believe my utility function is special. I don’t expect the universe (outside of me, my fellow humans, and any optimizing processes we spawn off) to agree with me. But, like Eliezer says, “We’ll see which one of us is still standing when this is over.”
Let’s recap. You made a wrong claim. I responded to the wrong claim. You disputed my response. I refuted your disputation. You attempted to defend your claim. I responded to your defense. You edited your defense by replacing it with the acknowledgment of your mistake.
No, that isn’t what happened. I’m not sure which comment the last sentence is supposed to refer to, but I’m p > .8 it didn’t happen that way. If it’s referring to the statement, “Okay, you don’t actually need randomness,” I wrote that before I ever saw your first response to that comment. But that doesn’t match up with what you just described; there weren’t that many exchanges before that comment. It also doesn’t match up with anything after that comment, since I still don’t acknowledge any such mistake made after that comment.
When you acknowledge your claim is wrong, you should not at the same time criticize me for refuting your point.
We’re talking about 2 separate claims. The wrong claim that I made was in an early statement where I said that you “needed randomness” to explore the space of possible utility functions. The right claim that I made, at length, was that randomness is a useful tool. You are conflating my defense of that claim, with defending the initial wrong claim. You’ve also said that you agree that randomness is a useful tool, which suggests that what is happening is that you made a whole series of comments that I say were attacking claim 2, and that you believe were attacking claim 1.
I’m not planning to tile the universe with myself, I just want myself or something closely isomorphic to me to continue to exist. The two most obvious ways to ensure my own continued existence are avoidance of things that would destroy me, particularly intelligent agents which could devote significant resources to destroying me personally, and making redundant copies. My own ability to copy myself is limited, and an imperfect copy might compete with me for the same scarce resources, so option two is curtailed by option one. Actual destruction of enemies is just an extension of avoidance; that which no longer exists within my light-cone can no longer pose a threat.
Your characterization of my utility function as arbitrary is, itself, arbitrary. Deal with it.
No. That statement is three comments above the comment in which you said I should acknowledge my error. It was already there when you wrote that comment. And I also acknowledged my misstatement in the comment you were replying to, and elaborated on what I had meant when I made the comment.
Good! We agree.
Good! We agree again.
And we agree yet again!
And here is where we part ways.
Maybe there is no universal utility function. That’s a… I won’t say it’s a reasonable position, but I understand its appeal. I would call it an over-reasoned position, like when a philosopher announces that he has proved that he doesn’t exist. It’s time to go back to the drawing board when you come up with that conclusion. Or at least to take your own advice, and stop trying to change the world when you’ve already said it doesn’t matter how it changes.
But to believe that your utility function is nothing special, and still try to take over the universe and force your utility function on it for all time, is insane.
(Yes, yes, I know Eliezer has all sorts of disclaimers in the CEV document about how CEV should not try to take over the universe. I don’t believe that it’s logically possible; and I believe that his discussions of Friendly AI make it even clearer that his plans require complete control. Perhaps the theory is still vague enough that just maybe there’s a way around this; but I believe the burden of proof is on those who say there is a way around it.)
It would be consistent with the theory of utility functions if, in promoting CEV, you were acting on an inner drive that said, “Ooh, baby, I’m ensuring the survival of my utility function. Oh, God, yes! Yes! YES!” But that’s not what I see. I see people scribbling equations, studying the answers, and saying, “Hmm, it appears that my utility function is directing me to propagate itself. Oh, dear, I suppose I must, then.”
That’s just faking your utility function.
I think it’s key that the people I’m speaking of who believe utility functions are arbitrary, also believe they have no free will. And it’s probably also key that they assume their utility function must assign value to its own reproduction. They then use these two beliefs as an excuse to justify not following through on their belief about the arbitrariness of their utility function, because they think to do so would be logically impossible. “We can’t help ourselves! Our utility functions made us do it!” I don’t have a clean analysis, but there’s something circular, something wrong with this picture.
Let’s recap. You made a wrong claim. I responded to the wrong claim. You disputed my response. I refuted your disputation. You attempted to defend your claim. I responded to your defense. You edited your defense by replacing it with the acknowledgment of your mistake. You responded to my response still sort of defending your wrong claim, and attacking me for refuting your wrong claim. I defended my refutation, pointing out the you really did make the wrong claim and continued to defend it. And now you attack my defense, claiming that you did in fact acknowledge your mistake, and this should somehow negate your continued defense after the acknowledgement. Do you see how you are wrong here? When you acknowledge your claim is wrong, you should not at the same time criticize me for refuting your point.
I do believe my utility function is special. I don’t expect the universe (outside of me, my fellow humans, and any optimizing processes we spawn off) to agree with me. But, like Eliezer says, “We’ll see which one of us is still standing when this is over.”
No, that isn’t what happened. I’m not sure which comment the last sentence is supposed to refer to, but I’m p > .8 it didn’t happen that way. If it’s referring to the statement, “Okay, you don’t actually need randomness,” I wrote that before I ever saw your first response to that comment. But that doesn’t match up with what you just described; there weren’t that many exchanges before that comment. It also doesn’t match up with anything after that comment, since I still don’t acknowledge any such mistake made after that comment.
We’re talking about 2 separate claims. The wrong claim that I made was in an early statement where I said that you “needed randomness” to explore the space of possible utility functions. The right claim that I made, at length, was that randomness is a useful tool. You are conflating my defense of that claim, with defending the initial wrong claim. You’ve also said that you agree that randomness is a useful tool, which suggests that what is happening is that you made a whole series of comments that I say were attacking claim 2, and that you believe were attacking claim 1.
I’m not planning to tile the universe with myself, I just want myself or something closely isomorphic to me to continue to exist. The two most obvious ways to ensure my own continued existence are avoidance of things that would destroy me, particularly intelligent agents which could devote significant resources to destroying me personally, and making redundant copies. My own ability to copy myself is limited, and an imperfect copy might compete with me for the same scarce resources, so option two is curtailed by option one. Actual destruction of enemies is just an extension of avoidance; that which no longer exists within my light-cone can no longer pose a threat.
Your characterization of my utility function as arbitrary is, itself, arbitrary. Deal with it.