I waited to comment on this, to see what others would say. Right now Psy-Kosh seems to be right about anthropics; Wei Dai seems to be right about UDT; timtyler seems to be right about Boltzmann brains; byrnema seems to be mostly right about pointers; but I don’t understand why nobody latched on to the “reflective consistency” part. Surely the kind of consistency under observer-splitting that you describe is too strong a requirement in general: if two copies of you play a game, the correct behavior for both of them would be to try to win, regardless of what overall outcome you’d prefer before the copying. The paperclip formulation works around this problem, so the correct way to analyze this would be in terms of multiplayer game theory with chance moves, as Psy-Kosh outlined.
if two copies of you play a game, the correct behavior for both of them would be to try to win, regardless of what overall outcome you’d prefer before the copying
That doesn’t make sense to me, unless you’re assuming that the player isn’t capable of self-modification. If it was, wouldn’t it modify itself so that its copies won’t try to win individually, but cooperate to obtain the outcome that it prefers before the copying?
Yes, that’s right. I’ve shifted focus from correct program behavior to correct human behavior, because that’s what everyone else here seems to be talking about. If the problem is about programs, there’s no room for all this confusion in the first place. Just specify the inputs, outputs and goal function, then work out the optimal algorithm.
I waited to comment on this, to see what others would say. Right now Psy-Kosh seems to be right about anthropics; Wei Dai seems to be right about UDT; timtyler seems to be right about Boltzmann brains; byrnema seems to be mostly right about pointers; but I don’t understand why nobody latched on to the “reflective consistency” part. Surely the kind of consistency under observer-splitting that you describe is too strong a requirement in general: if two copies of you play a game, the correct behavior for both of them would be to try to win, regardless of what overall outcome you’d prefer before the copying. The paperclip formulation works around this problem, so the correct way to analyze this would be in terms of multiplayer game theory with chance moves, as Psy-Kosh outlined.
That doesn’t make sense to me, unless you’re assuming that the player isn’t capable of self-modification. If it was, wouldn’t it modify itself so that its copies won’t try to win individually, but cooperate to obtain the outcome that it prefers before the copying?
Yes, that’s right. I’ve shifted focus from correct program behavior to correct human behavior, because that’s what everyone else here seems to be talking about. If the problem is about programs, there’s no room for all this confusion in the first place. Just specify the inputs, outputs and goal function, then work out the optimal algorithm.
Unless the copies can modify themselves too.