As Douglas_Knight shows, my comment wasn’t really well thought out. However, the idea is that a reflective decision theory agent considers the implications of the fact that whatever her decision is, similar agents will reach a similar decision. This makes them cooperate in Prisoner’s Dilemma—Tragedy of the Commons situations where “if all of us behaved so selfishly, we would be in big trouble”. The thing is sometimes called superrationality.
As Douglas_Knight shows, my comment wasn’t really well thought out. However, the idea is that a reflective decision theory agent considers the implications of the fact that whatever her decision is, similar agents will reach a similar decision. This makes them cooperate in Prisoner’s Dilemma—Tragedy of the Commons situations where “if all of us behaved so selfishly, we would be in big trouble”. The thing is sometimes called superrationality.