Yeah, that’d work for investigating the hypothesis and is interestingly similar to the theoretical ideal of the peer review process. I was personally more curious about how to distinguish in the general case where reliable domain experts may not be recognizable or readily available but that question may be rationality-complete.
Something like: there are dozen groups of people believing they are experts on meditation, each of them believing the other groups are wrong—how do find the right ones?
In that case, perhaps instead of talking about “meditation”, we could define “meditation1″ as whatever the group1 believes, “meditation2” as whatever the group2 believes… and test independently which groups can be easily fooled by GPT-3.
Yeah, that’d work for investigating the hypothesis and is interestingly similar to the theoretical ideal of the peer review process.
I was personally more curious about how to distinguish in the general case where reliable domain experts may not be recognizable or readily available but that question may be rationality-complete.
Something like: there are dozen groups of people believing they are experts on meditation, each of them believing the other groups are wrong—how do find the right ones?
In that case, perhaps instead of talking about “meditation”, we could define “meditation1″ as whatever the group1 believes, “meditation2” as whatever the group2 believes… and test independently which groups can be easily fooled by GPT-3.