Anyone else worried by this result, or have I made a mistake?
To update my reply to Silas Barta, after a little reflection I would say this:
The various species are supposed to possess common knowledge of each other’s utility functions, and of each other’s epistemic beliefs about how these utility functions can be satsified.
Since the various species’ preferences are described by utility functions, we must assume that each species has self-modified collectively (or so the humans believe) such that they collectively obey the von Neumann-Morgenstern axioms—this eliminates much of the complexity that I had in mind when I wrote my reply to Barta.
However, one further item of common knowledge would be helpful: common knowledge of whether the various species are likely to be timeless decision theorists. If they possess this common knowledge, then the dilemma is just a disguised version of a standard Newcomblike problem: the agents possess common knowledge of all the relevant factors that might influence the abstract computation that they implement in determining which strategy to employ. This is no different to the scenario in which two AIs can read one another’s source code—except in this case they do it by magic (the scenario is entirely ridiculous, I’m afraid). And in that case co-operation in the prisoner’s dilemma is the optimal choice.
On the other hand if they don’t possess common knowledge of whether they are TDT-agents (rather than CDT-agents for example) then whether it is wise for humans to defect or co-operate depends on their probability estimate regarding whether the aliens are mostly TDT-agents, and their estimates of the aliens’ own estimates whether the other species are TDT-agents, et cetera. I don’t really know how that infinite regress would be resolved by the humans, and your premises give us little way of knowing what these probability estimates might be.
To update my reply to Silas Barta, after a little reflection I would say this:
The various species are supposed to possess common knowledge of each other’s utility functions, and of each other’s epistemic beliefs about how these utility functions can be satsified.
Since the various species’ preferences are described by utility functions, we must assume that each species has self-modified collectively (or so the humans believe) such that they collectively obey the von Neumann-Morgenstern axioms—this eliminates much of the complexity that I had in mind when I wrote my reply to Barta.
However, one further item of common knowledge would be helpful: common knowledge of whether the various species are likely to be timeless decision theorists. If they possess this common knowledge, then the dilemma is just a disguised version of a standard Newcomblike problem: the agents possess common knowledge of all the relevant factors that might influence the abstract computation that they implement in determining which strategy to employ. This is no different to the scenario in which two AIs can read one another’s source code—except in this case they do it by magic (the scenario is entirely ridiculous, I’m afraid). And in that case co-operation in the prisoner’s dilemma is the optimal choice.
On the other hand if they don’t possess common knowledge of whether they are TDT-agents (rather than CDT-agents for example) then whether it is wise for humans to defect or co-operate depends on their probability estimate regarding whether the aliens are mostly TDT-agents, and their estimates of the aliens’ own estimates whether the other species are TDT-agents, et cetera. I don’t really know how that infinite regress would be resolved by the humans, and your premises give us little way of knowing what these probability estimates might be.