It seems like you’ve ignored the possibility of importance sampling?
Ah, right, I agree. I forgot about that suggestion as I was writing. It seems likely some version of this would work.
(I feel like I’m restating what you said, I guess I’m confused why you interpret this as evidence that the simplicity of the setup is “hiding something”.)
Yep, sorry, I think you should take that as something-about-Scott’s-point-abram-didn’t-explain. I still disclaim myself as maybe missing part of Scott’s point. But: what the simpler setup is “hiding” is the complexity of comparing answers:
The complexity of determining whether two claims are “different”.
The complexity of determining whether two claims are mutually exclusive.
The complexity of comparing the quality of different arguments, when the different answers may be expressed in very different ontologies, and deal with very difficult-to-compare considerations.
Making the two sides defend entirely unrelated claims makes all this obvious. In addition, it makes the first two bullet points irrelevant, removing a “fake difficulty” from the setup.
Okay, that all makes sense. One maybe-caveat-or-disagreement:
The complexity of comparing the quality of different arguments, when the different answers may be expressed in very different ontologies, and deal with very difficult-to-compare considerations.
I do think that answering the same question does make it meaningfully easier to compare answers, though I agree it’s still not obvious that it’s easy on some absolute scale for the reasons you outline.
Ah, right, I agree. I forgot about that suggestion as I was writing. It seems likely some version of this would work.
Yep, sorry, I think you should take that as something-about-Scott’s-point-abram-didn’t-explain. I still disclaim myself as maybe missing part of Scott’s point. But: what the simpler setup is “hiding” is the complexity of comparing answers:
The complexity of determining whether two claims are “different”.
The complexity of determining whether two claims are mutually exclusive.
The complexity of comparing the quality of different arguments, when the different answers may be expressed in very different ontologies, and deal with very difficult-to-compare considerations.
Making the two sides defend entirely unrelated claims makes all this obvious. In addition, it makes the first two bullet points irrelevant, removing a “fake difficulty” from the setup.
Okay, that all makes sense. One maybe-caveat-or-disagreement:
I do think that answering the same question does make it meaningfully easier to compare answers, though I agree it’s still not obvious that it’s easy on some absolute scale for the reasons you outline.