If you query Less Wrong, what is the probability that the median response is acceptably close to correct? Please provide confidence intervals, feel free to break out any classes of propositions if you feel that it would be unfair/poor form/not very fun at all to group all classes together but explain why.
We would give you our estimates, but they’re probably wrong.
Seriously: For practical real-world questions, my wild guess is that the most-upvoted answer will be “acceptably close to correct” in about two thirds of the questions that are asked. For more nebulous philosophical stuff like many-worlds and qualia, I’d put our accuracy much lower.
Related is the calibration question in the old survey, though I think the staggering accuracy here was a fluke:
Thomas Edison patented the lightbulb in 1880. I’ve never before been a firm believer in the wisdom of crowds, but it really came through in this case. Even though this was clearly not an easy question and many people got really far-off answers, the mean was 1879.3 and the median was 1880. The standard deviation was 36.1. Person who put “2172”, you probably thought you were screwing up the results, but in fact you managed to counterbalance the other person who put “1700″, allowing the mean to revert back to within one year of the correct value :P
The average person was 26.77% sure they got within 5 years of the correct answer on the lightbulb question. 30% of people did get within 5 years. I’m not sure how much to trust the result, because several people put the exact correct year down and gave it 100% confidence. Either they were really paying attention in history class, or they checked Wikipedia. There was a high correlation between high levels of confidence on the question and actually getting the question right, significant at the <.001 level.
If you query Less Wrong, what is the probability that the median response is acceptably close to correct? Please provide confidence intervals, feel free to break out any classes of propositions if you feel that it would be unfair/poor form/not very fun at all to group all classes together but explain why.
We would give you our estimates, but they’re probably wrong.
Seriously: For practical real-world questions, my wild guess is that the most-upvoted answer will be “acceptably close to correct” in about two thirds of the questions that are asked. For more nebulous philosophical stuff like many-worlds and qualia, I’d put our accuracy much lower.
Related is the calibration question in the old survey, though I think the staggering accuracy here was a fluke: