Clearly, if all people rely on the same data, but all use different (but quite sound) cognitive rituals, there is still only one data set.
I’d think that you should first compute the meaning of the data by averaging over all (apparently sound) ROC results, and then update based on that outcome. I.e. if only lunatics saw something, and they all say it means A, then that counts for nothing.
If a bunch of trustworthy bayesians see something and they all conclude B, then that counts like one trustworthy bayesian for B. If some trustworthy bayesians and a similar number of (apparently “winning” ⇒ implying sound ROC) aliens, who deny to update a la Bayes, say it’s B there is still one vote only.
If the aliens and the bayesians saw different data though, that’ll make two votes.
It’s a question of how much does the variance in data mess up your conclusions compared to the variance in ROC.
If all the variance is in the data, then sure, several valid interpretations of the same data barely outweighs an individual with a unique data set.
However, if the data is largely shared but it’s a tough problem, so people hack at it in wildly different ways (eg outside view vs inside view), then you care more about different valid ROC than another slightly different data set.
I intended (though probably failed) to convey the idea of noninteger numbers of votes depending on the degree of correlation between datasets/ROC. If the datasets are 90% overlapping, then you dont get a full vote for adding another. If your ROC are largely overlapping (eg two attempts at outside view), then you only get a small increase in voting power, but if its large (eg inside vs outside) you can get almost another full vote.
Why do you count (ROC, Data) Pairs?
Clearly, if all people rely on the same data, but all use different (but quite sound) cognitive rituals, there is still only one data set.
I’d think that you should first compute the meaning of the data by averaging over all (apparently sound) ROC results, and then update based on that outcome. I.e. if only lunatics saw something, and they all say it means A, then that counts for nothing. If a bunch of trustworthy bayesians see something and they all conclude B, then that counts like one trustworthy bayesian for B. If some trustworthy bayesians and a similar number of (apparently “winning” ⇒ implying sound ROC) aliens, who deny to update a la Bayes, say it’s B there is still one vote only.
If the aliens and the bayesians saw different data though, that’ll make two votes.
It’s a question of how much does the variance in data mess up your conclusions compared to the variance in ROC.
If all the variance is in the data, then sure, several valid interpretations of the same data barely outweighs an individual with a unique data set.
However, if the data is largely shared but it’s a tough problem, so people hack at it in wildly different ways (eg outside view vs inside view), then you care more about different valid ROC than another slightly different data set.
I intended (though probably failed) to convey the idea of noninteger numbers of votes depending on the degree of correlation between datasets/ROC. If the datasets are 90% overlapping, then you dont get a full vote for adding another. If your ROC are largely overlapping (eg two attempts at outside view), then you only get a small increase in voting power, but if its large (eg inside vs outside) you can get almost another full vote.