The problem is that there’s no reasonable way to grade the quoted rejects as false. If you aren’t a lawyer (Edit: but maybe if you are*), there’s really nothing about labeling John Roberts as Chief Justice of the Supreme Court that is more useful than labeling him as “the justice in charge of the Supreme Court.” The error is roughly on par with asking “what does 2+3 =” and accepting “V” but rejecting “IIIII”
In short, I have dramatically adjusted downward my belief the reliability of public-ignorance surveys.
On reflection, I think some of the answers could be considered wrong in a technical sense not relevant to the question being asked. For example, “in charge” implies a bit more power over Supreme Court decisions than Roberts actually possesses.
In the old version, I stated that the difference wouldn’t matter even to a lawyer.
I haven’t. I expected they were making mistakes like this one, and haven’t seen anything indicating they generally make mistakes in this direction rather than the other.
It makes sense to adjust downward your belief that they are reliable, if you thought they were very reliable before. But this shouldn’t be enough to indicate they’re reliably getting it wrong in a particular direction.
If the ultimate goal is to compare knowledge of the Supreme Court to knowledge of the Simpsons, I would expect the surveys to reliably be wrong in the more sensational direction.
The problem is that there’s no reasonable way to grade the quoted rejects as false. If you aren’t a lawyer (Edit: but maybe if you are*), there’s really nothing about labeling John Roberts as Chief Justice of the Supreme Court that is more useful than labeling him as “the justice in charge of the Supreme Court.” The error is roughly on par with asking “what does 2+3 =” and accepting “V” but rejecting “IIIII”
In short, I have dramatically adjusted downward my belief the reliability of public-ignorance surveys.
On reflection, I think some of the answers could be considered wrong in a technical sense not relevant to the question being asked. For example, “in charge” implies a bit more power over Supreme Court decisions than Roberts actually possesses.
In the old version, I stated that the difference wouldn’t matter even to a lawyer.
I haven’t. I expected they were making mistakes like this one, and haven’t seen anything indicating they generally make mistakes in this direction rather than the other.
It makes sense to adjust downward your belief that they are reliable, if you thought they were very reliable before. But this shouldn’t be enough to indicate they’re reliably getting it wrong in a particular direction.
If the ultimate goal is to compare knowledge of the Supreme Court to knowledge of the Simpsons, I would expect the surveys to reliably be wrong in the more sensational direction.