in fact, it could be an anti-rational agent with the opposite utility function.
These two people might look the same, the might be identical on a quantum level, but one of them is a largely rational agent, and the other is an anti-rational agent with the opposite utility function.
I think that calling something an anti-rational agent with the opposite utility function is a wierd description that doesn’t cut reality at its joints. The is a simple notion of a perfect sphere. There is also a simple notion of a perfect optimizer. Real world objects aren’t perfect spheres, but some are pretty close. Thus “sphere” is a useful approximation, and “sphere + error term” is a useful description. Real agents aren’t perfect optimisers, (ignoring contived goals like “1 for doing whatever you were going to do anyway, 0 else”) but some are pretty close, hence “utility function + biases” is a useful description. This makes the notion of an anti-rational agent with opposite utility function like an inside out sphere with its surface offset inwards by twice the radius. Its a cack handed description of a simple object in terms of a totally different simple object and a huge error term.
This is one of those circumstances where it is important to differentiate between you being in a situation and a simulation of you being in a situation.
I actually don’t think that there is a general procedure to tell what is you, and what is a simulation of you. Standard argument about slowly replacing neurons with nanomachines, slowly porting it to software, slowly abstracting and proving theorems about it rather than running it directly.
It is an entirely meaningful utility function to only care about copies of your algorithm that are running on certain kinds of hardware. That makes you a “biochemical brains running this algorithm” mazimizer. The paperclip maximizer doesn’t care about any copy of its algorithm. Humans worrying about whether the predictors simulation is detailed enough to really suffer is due to specific features of human morality. From the perspective of the paperclip maximizer doing decision theory, what we care about is logical correlation.
“I actually don’t think that there is a general procedure to tell what is you, and what is a simulation of you”—Let’s suppose I promise to sell you an autographed Michael Jackson CD. But then it turns out that the CD wasn’t signed by Michael, but by me. Now I’m really good at forgeries, so good in fact that my signature matches his atom to atom. Haven’t I still lied?
Imagine sitting outside the universe, and being given an exact description of everything that happened within the universe. From this perspective you can see who signed what.
You can also see whether your thoughts are happening in biology or silicon or whatever.
My point isn’t “you can’t tell whether or not your in a simulation so there is no difference”, my point is that there is no sharp cut off point between simulation and not simulation. We have a “know it when you see it” definition with ambiguous edge cases. Decision theory can’t have different rules for dealing with dogs and not dogs because some things are on the ambiguous edge of dogginess. Likewise decision theory can’t have different rules for you, copies of you and simulations of you as there is no sharp cut off. If you want to propose a continuous “simulatedness” parameter, and explain where that gets added to decision theory, go ahead. (Or propose some sharp cutoff)
Some people want to act as though a simulation of you is automatically you and my argument is that it is bad practise to assume this. I’m much more open to the idea that some simulations might be you in some sense than the claim that all are. This seems compatible with a fuzzy cut-off.
These two people might look the same, the might be identical on a quantum level, but one of them is a largely rational agent, and the other is an anti-rational agent with the opposite utility function.
I think that calling something an anti-rational agent with the opposite utility function is a wierd description that doesn’t cut reality at its joints. The is a simple notion of a perfect sphere. There is also a simple notion of a perfect optimizer. Real world objects aren’t perfect spheres, but some are pretty close. Thus “sphere” is a useful approximation, and “sphere + error term” is a useful description. Real agents aren’t perfect optimisers, (ignoring contived goals like “1 for doing whatever you were going to do anyway, 0 else”) but some are pretty close, hence “utility function + biases” is a useful description. This makes the notion of an anti-rational agent with opposite utility function like an inside out sphere with its surface offset inwards by twice the radius. Its a cack handed description of a simple object in terms of a totally different simple object and a huge error term.
I actually don’t think that there is a general procedure to tell what is you, and what is a simulation of you. Standard argument about slowly replacing neurons with nanomachines, slowly porting it to software, slowly abstracting and proving theorems about it rather than running it directly.
It is an entirely meaningful utility function to only care about copies of your algorithm that are running on certain kinds of hardware. That makes you a “biochemical brains running this algorithm” mazimizer. The paperclip maximizer doesn’t care about any copy of its algorithm. Humans worrying about whether the predictors simulation is detailed enough to really suffer is due to specific features of human morality. From the perspective of the paperclip maximizer doing decision theory, what we care about is logical correlation.
“I actually don’t think that there is a general procedure to tell what is you, and what is a simulation of you”—Let’s suppose I promise to sell you an autographed Michael Jackson CD. But then it turns out that the CD wasn’t signed by Michael, but by me. Now I’m really good at forgeries, so good in fact that my signature matches his atom to atom. Haven’t I still lied?
Imagine sitting outside the universe, and being given an exact description of everything that happened within the universe. From this perspective you can see who signed what.
You can also see whether your thoughts are happening in biology or silicon or whatever.
My point isn’t “you can’t tell whether or not your in a simulation so there is no difference”, my point is that there is no sharp cut off point between simulation and not simulation. We have a “know it when you see it” definition with ambiguous edge cases. Decision theory can’t have different rules for dealing with dogs and not dogs because some things are on the ambiguous edge of dogginess. Likewise decision theory can’t have different rules for you, copies of you and simulations of you as there is no sharp cut off. If you want to propose a continuous “simulatedness” parameter, and explain where that gets added to decision theory, go ahead. (Or propose some sharp cutoff)
Some people want to act as though a simulation of you is automatically you and my argument is that it is bad practise to assume this. I’m much more open to the idea that some simulations might be you in some sense than the claim that all are. This seems compatible with a fuzzy cut-off.