Perhaps the crux here is whether we should expect all superintelligent agents to converge on the same decision procedure—and the agent themselves will expect this, such that they’ll coordinate by default? As sympathetic as I am to realism about rationality, I put a pretty nontrivial credence on the possibility that this convergence just won’t occur, and persistent disagreement (among well-informed people) about the fundamentals of what it means to “win” in decision theory thought experiments is evidence of this.
Perhaps the crux here is whether we should expect all superintelligent agents to converge on the same decision procedure—and the agent themselves will expect this, such that they’ll coordinate by default? As sympathetic as I am to realism about rationality, I put a pretty nontrivial credence on the possibility that this convergence just won’t occur, and persistent disagreement (among well-informed people) about the fundamentals of what it means to “win” in decision theory thought experiments is evidence of this.