if two copies of you play a game, the correct behavior for both of them would be to try to win, regardless of what overall outcome you’d prefer before the copying
That doesn’t make sense to me, unless you’re assuming that the player isn’t capable of self-modification. If it was, wouldn’t it modify itself so that its copies won’t try to win individually, but cooperate to obtain the outcome that it prefers before the copying?
Yes, that’s right. I’ve shifted focus from correct program behavior to correct human behavior, because that’s what everyone else here seems to be talking about. If the problem is about programs, there’s no room for all this confusion in the first place. Just specify the inputs, outputs and goal function, then work out the optimal algorithm.
That doesn’t make sense to me, unless you’re assuming that the player isn’t capable of self-modification. If it was, wouldn’t it modify itself so that its copies won’t try to win individually, but cooperate to obtain the outcome that it prefers before the copying?
Yes, that’s right. I’ve shifted focus from correct program behavior to correct human behavior, because that’s what everyone else here seems to be talking about. If the problem is about programs, there’s no room for all this confusion in the first place. Just specify the inputs, outputs and goal function, then work out the optimal algorithm.
Unless the copies can modify themselves too.