So what an expert rationalist should do to avoid this overconfidence trap? The seeming answer is that we should rely less on our own reasoning and more on the “wisdom of the crowds”.
I have another answer. If my beliefs differ from the majority, then I would take the crowd’s beliefs as potential counter-evidence. Rather than leading me to conform to the views of the crowd, it will lead me to scrutinize my beliefs extra hard for potential errors in reasoning. If I find some, then I might agree with the crowd. Otherwise, I will assume that the crowd is wrong.
It also depends on the level of speculation involved and how good the quality of evidence is. For dealing with a speculative question where I don’t have much evidence, the beliefs of the crowd might be some of the best quality evidence I have, so I will adjust my assessment in their direction. But if I have some better evidence that I know the crowd doesn’t have, if I can point out massive gaping holes in the crowd’s reasoning, or if I can see cognitive biases that would better explain the crowd’s views instead of those views being caused by truth, then the evidence of the crowd would not be a large factor in my thinking.
For instance, multiple studies have shown that investors who are more confident of their ability to beat the market receive lower returns on their investments. This overconfidence penalty applies even to the supposed experts, such as fund managers.
The ability of experts vs. the crowd varies in different domains. Yes, there are domains where the experts are overconfident, or even charlatans (finance is one of the best examples). Yet it’s also easy to find domains where the experts do know a lot better than the crowd. I really don’t trust what the crowd thinks about quantum mechanics or brain surgery.
Remember, IQ is normally distributed. Only a couple percent of people have IQ 2+ standard deviations from the mean (130). For certain types of problems that require high intelligence to solve, the views of the masses will indeed be worthless on average. Same thing with problems that require certain types of specialized knowledge.
In many cases, it really is possible that the crowd is just being stupid, again. To ignore this, and to fail to acknowledge that it can be true even in cases where you arrogantly disagree with the crowd, is not rationality but a weird form of cognitive hyper-humility. The crowd isn’t necessarily wrong because we are smarter than it, but I think it’s possible to make rational arguments that the crowd is wrong and we are right.
Consider that many important philosophical and scientific ideas that the crowd acknowledges as true today (e.g. heliocentrism) were once pioneered by an expert going against the crowd. I think history would have gone worse if guys like Galileo (sincerely) said, “hey guys, I kinda thought that the Earth went around the Sun, but since so many of you believe otherwise, I think I might be a bit too overconfident...”
Smart people going against the wisdom of the crowd is a necessary, but not sufficient condition of the advancement of human knowledge.
Would they be willing to shift their views to accommodate the chance that their own reasoning powers are insufficient to get the right answer?
No. I’m confident that there are no rational reasons to believe in God, and I’ve scrutinized my reasoning plenty, and the arguments for belief in God. I have much better explanations for the crowd’s beliefs rather than God existing and causing those beliefs. Although I would like to hear Newton or Einstein’s justifications for theism. I’m a fallibilist, so I recognize that I could be wrong.
I agree with you that there is a danger in over-confidence, including for the smart people, but I think there is also a danger in excessive cognitive humility, especially for smart people. To paraphrase a famous quote: All that is necessary for the triumph of stupidity is that smart people do nothing.
Thanks for the thoughtful reply. I must admit that even though I expected that some “rationalists” would be just as defensive as religious folks about their views, looking at my karma now I realize that I grossly underestimated their number. That’s a good lesson to me for lecturing other people about overconfidence.
″ For certain types of problems that require high intelligence to solve, the views of the masses will indeed be worthless on average. ”
I generally share your opinion in case the debated issues at stake are devoid of emotional charge. However, once we move to our deeply cherished views high IQ people can be just as good at self-deception as anybody else, if not better. As Orwell said, “some ideas are so stupid that only intellectuals could believe them.”
I have another answer. If my beliefs differ from the majority, then I would take the crowd’s beliefs as potential counter-evidence. Rather than leading me to conform to the views of the crowd, it will lead me to scrutinize my beliefs extra hard for potential errors in reasoning. If I find some, then I might agree with the crowd. Otherwise, I will assume that the crowd is wrong.
It also depends on the level of speculation involved and how good the quality of evidence is. For dealing with a speculative question where I don’t have much evidence, the beliefs of the crowd might be some of the best quality evidence I have, so I will adjust my assessment in their direction. But if I have some better evidence that I know the crowd doesn’t have, if I can point out massive gaping holes in the crowd’s reasoning, or if I can see cognitive biases that would better explain the crowd’s views instead of those views being caused by truth, then the evidence of the crowd would not be a large factor in my thinking.
The ability of experts vs. the crowd varies in different domains. Yes, there are domains where the experts are overconfident, or even charlatans (finance is one of the best examples). Yet it’s also easy to find domains where the experts do know a lot better than the crowd. I really don’t trust what the crowd thinks about quantum mechanics or brain surgery.
Remember, IQ is normally distributed. Only a couple percent of people have IQ 2+ standard deviations from the mean (130). For certain types of problems that require high intelligence to solve, the views of the masses will indeed be worthless on average. Same thing with problems that require certain types of specialized knowledge.
In many cases, it really is possible that the crowd is just being stupid, again. To ignore this, and to fail to acknowledge that it can be true even in cases where you arrogantly disagree with the crowd, is not rationality but a weird form of cognitive hyper-humility. The crowd isn’t necessarily wrong because we are smarter than it, but I think it’s possible to make rational arguments that the crowd is wrong and we are right.
Consider that many important philosophical and scientific ideas that the crowd acknowledges as true today (e.g. heliocentrism) were once pioneered by an expert going against the crowd. I think history would have gone worse if guys like Galileo (sincerely) said, “hey guys, I kinda thought that the Earth went around the Sun, but since so many of you believe otherwise, I think I might be a bit too overconfident...”
Smart people going against the wisdom of the crowd is a necessary, but not sufficient condition of the advancement of human knowledge.
No. I’m confident that there are no rational reasons to believe in God, and I’ve scrutinized my reasoning plenty, and the arguments for belief in God. I have much better explanations for the crowd’s beliefs rather than God existing and causing those beliefs. Although I would like to hear Newton or Einstein’s justifications for theism. I’m a fallibilist, so I recognize that I could be wrong.
I agree with you that there is a danger in over-confidence, including for the smart people, but I think there is also a danger in excessive cognitive humility, especially for smart people. To paraphrase a famous quote: All that is necessary for the triumph of stupidity is that smart people do nothing.
Thanks for the thoughtful reply. I must admit that even though I expected that some “rationalists” would be just as defensive as religious folks about their views, looking at my karma now I realize that I grossly underestimated their number. That’s a good lesson to me for lecturing other people about overconfidence.
″ For certain types of problems that require high intelligence to solve, the views of the masses will indeed be worthless on average. ”
I generally share your opinion in case the debated issues at stake are devoid of emotional charge. However, once we move to our deeply cherished views high IQ people can be just as good at self-deception as anybody else, if not better. As Orwell said, “some ideas are so stupid that only intellectuals could believe them.”