For the one shot PD, it seems like something similar is happening. Cooperating just doesn’t ‘seem’ right to me most of the time, but it’s only because I’d have a hard time believing the other guy was running the same algorithm.
Do you think that the other guy is thinking the same thing, and reasoning the same way? Or do you think that the other will probably decide to cooperate or defect on the PD using some unrelated algorithm?
My main reason for potentially defecting on the true PD against another human—note the sheer difficulty of obtaining this unless the partner is Hannibal with an imminently lethal wound—would be my doubts that they were actually calculating using a timeless decision theory, even counting someone thinking about Hofstadterian superrationality as TDT. Most people who’ve studied the matter in college have been taught that the right thing to do is to defect, and those who cooperate on instinct are running a different algorithm, that of being honorable.
But it’d be pretty damn hard in real life to put me into a literally one-shot, uncoordinated, no-communication, true PD where I’m running TDT, the other person is running honor with no inkling that I’m TDT, and the utilities at stake outweigh that which constrains me not to betray honorable people. It deserves a disclaimer to the effect of “This hypothetical problem is sufficiently different from the basic conditions of real life that no ethical advice should be taken from my hypothetical answer.”
Do you think that the other guy is thinking the same thing, and reasoning the same way? Or do you think that the other will probably decide to cooperate or defect on the PD using some unrelated algorithm?
The latter. I haven’t thought about this enough be comfortable knowing how similar his algorithm must be in order to cooperate, but if I ultimately decided to defect it’d be because I thought it qualified as sufficiently different.
So you fully expect in real life that you might defect and yet see the other person cooperate (with standard ethical disclaimers about how hard it is to true the PD such that you actually prefer to see that outcome).
Yes, that’s correct. I also currently see a significant probability of choosing to cooperate and finding out that the other guy defected on me. Should I take your response as evidence to reconsider? As I said before, I don’t claim to have this all sorted out.
As to your disclaimer, it seems like your impression says that it’s much harder to true PD than mine says. If you think you can make the thing truly one shot without reputational consequences (which may be the hard part, but it seems like you think its the other part), then it’s just a question of setting up the payoff table.
If you don’t have personal connections to the other party, it seems that you don’t care any more about him than the other 6 billion people on earth. If you can meet those conditions, even a small contribution to fighting existential risks (funded by your prize money) should outweigh anything you care about him.
But it’d be pretty damn hard in real life to put me into a literally one-shot, uncoordinated, no-communication, true PD where I’m running TDT, the other person is running honor with no inkling that I’m TDT, and the utilities at stake outweigh that which constrains me not to betray honorable people.
Mostly because of the “one-shot, uncoordinated, no-communication, true… utilities at stake outweigh” parts, I would think. The really relevant question conditions on those things.
Do you think that the other guy is thinking the same thing, and reasoning the same way? Or do you think that the other will probably decide to cooperate or defect on the PD using some unrelated algorithm?
My main reason for potentially defecting on the true PD against another human—note the sheer difficulty of obtaining this unless the partner is Hannibal with an imminently lethal wound—would be my doubts that they were actually calculating using a timeless decision theory, even counting someone thinking about Hofstadterian superrationality as TDT. Most people who’ve studied the matter in college have been taught that the right thing to do is to defect, and those who cooperate on instinct are running a different algorithm, that of being honorable.
But it’d be pretty damn hard in real life to put me into a literally one-shot, uncoordinated, no-communication, true PD where I’m running TDT, the other person is running honor with no inkling that I’m TDT, and the utilities at stake outweigh that which constrains me not to betray honorable people. It deserves a disclaimer to the effect of “This hypothetical problem is sufficiently different from the basic conditions of real life that no ethical advice should be taken from my hypothetical answer.”
The latter. I haven’t thought about this enough be comfortable knowing how similar his algorithm must be in order to cooperate, but if I ultimately decided to defect it’d be because I thought it qualified as sufficiently different.
So you fully expect in real life that you might defect and yet see the other person cooperate (with standard ethical disclaimers about how hard it is to true the PD such that you actually prefer to see that outcome).
Yes, that’s correct. I also currently see a significant probability of choosing to cooperate and finding out that the other guy defected on me. Should I take your response as evidence to reconsider? As I said before, I don’t claim to have this all sorted out.
As to your disclaimer, it seems like your impression says that it’s much harder to true PD than mine says. If you think you can make the thing truly one shot without reputational consequences (which may be the hard part, but it seems like you think its the other part), then it’s just a question of setting up the payoff table.
If you don’t have personal connections to the other party, it seems that you don’t care any more about him than the other 6 billion people on earth. If you can meet those conditions, even a small contribution to fighting existential risks (funded by your prize money) should outweigh anything you care about him.
Mostly because of the “one-shot, uncoordinated, no-communication, true… utilities at stake outweigh” parts, I would think. The really relevant question conditions on those things.