To some extent, this is all already in Jozdien’s comment, but:
It seems that the closest thing to AIs debating alignment (or providing hopefully verifiable solutions) that we can observe is human debate about alignment (and perhaps also related questions about the future). Presumably John and Paul have similar views about the empirical difficulty of reaching agreement in the human debate about alignment, given that they both observe this debate a lot. (Perhaps they disagree about what people’s level of (in)ability to reach agreement / verify arguments implies for the probability of getting alignment right. Let’s ignore that possibility...) So I would have thought that even w.r.t. this fairly closely related debate, the disagreement is mostly about what happens as we move from human to superhuman-AI discussants. In particular, I would expect Paul to concede that the current level of disagreement in the alignment community is problematic and to argue that this will improve (enough) if we have superhuman debaters. If even this closely related form of debate/delegation/verification process isn’t taken to be very informative (by at least one of Paul and John), then it’s hard to imagine that much more distant delegation processes (such as those behind making computer monitors) are very informative to their disagreement.
To some extent, this is all already in Jozdien’s comment, but:
It seems that the closest thing to AIs debating alignment (or providing hopefully verifiable solutions) that we can observe is human debate about alignment (and perhaps also related questions about the future). Presumably John and Paul have similar views about the empirical difficulty of reaching agreement in the human debate about alignment, given that they both observe this debate a lot. (Perhaps they disagree about what people’s level of (in)ability to reach agreement / verify arguments implies for the probability of getting alignment right. Let’s ignore that possibility...) So I would have thought that even w.r.t. this fairly closely related debate, the disagreement is mostly about what happens as we move from human to superhuman-AI discussants. In particular, I would expect Paul to concede that the current level of disagreement in the alignment community is problematic and to argue that this will improve (enough) if we have superhuman debaters. If even this closely related form of debate/delegation/verification process isn’t taken to be very informative (by at least one of Paul and John), then it’s hard to imagine that much more distant delegation processes (such as those behind making computer monitors) are very informative to their disagreement.