There is much more to expertise than forecasting. Also
Designing and building “things” of various kinds that work
Fixing “things”
“Things” could include social systems, people, business structures, advertising campaigns, not just machines of course.
A person may be a very good football coach in the sense of putting together a team that wins, or fixing a losing team, but may not be too good at making predictions. Doctors are notoriously bad a predicting patient outcomes but are often be very skilled at actually treating them.
I think to a degree you confuse assessing whether a group does have expertise with assessing whether they are *likely* to have expertise.
As far as factors that count against expertise being reliably or significantly present, to your politics I would add
1. Money. The medical literature is replete with studies showing huge effect sizes from “who paid the piper”. In pharmaceutical research this seems to result in a 4X different chance of a positive result. But there is more than this; the ability to offer speaking and consultancy fees, funding of future projects etc can have a powerful effect.
Another example is the problem alluded to in relation to the consensus about the historical Jesus. When a field is dominated by people whose livelihood depends on their continuing to espouse a certain belief, the effect goes beyond those individuals and infects the whole field.
2. The pernicious effects of “great men” who can suppress dissent against their out of date views. Is the field pluralistic, realistically, and is dissent allowed? Science advances funeral by funeral. Have a look at what happened to John “Pure white and deadly” Yudkin.
3. Politics beyond what we normally think of as politics. Academia is notoriously “political” in this wider sense. Amplifying your point about reality checks, if feedback is not accurate, rapid, and unambiguous, it is hard for people in the field to know who is right, if anyone.
4. “Publish” or perish. There are massive incentives to get published or to get publicity or a high profile. This leads to people claiming expertise, results, achievements that are bogus. Consider for example the case of Theranos, which seemed, if media reports are accurate, to have no useful ability to build systems that did pathology tests, yet apparently hoodwinked many into thinking that they did.
You make a good point that claims of expertise without evidence or, worse, in the face of adverse evidence, are really really bad. I would go as far as to say that if you claim expertise but cannot prove it, I have a strong prior that you don’t have it.
There are large groups of self-described experts who do not have expertise or at best have far less than they think. One should be alert to the possibility that “experts” aren’t.
Good post. To which I would add...
There is much more to expertise than forecasting. Also
Designing and building “things” of various kinds that work
Fixing “things”
“Things” could include social systems, people, business structures, advertising campaigns, not just machines of course.
A person may be a very good football coach in the sense of putting together a team that wins, or fixing a losing team, but may not be too good at making predictions. Doctors are notoriously bad a predicting patient outcomes but are often be very skilled at actually treating them.
I think to a degree you confuse assessing whether a group does have expertise with assessing whether they are *likely* to have expertise.
As far as factors that count against expertise being reliably or significantly present, to your politics I would add
1. Money. The medical literature is replete with studies showing huge effect sizes from “who paid the piper”. In pharmaceutical research this seems to result in a 4X different chance of a positive result. But there is more than this; the ability to offer speaking and consultancy fees, funding of future projects etc can have a powerful effect.
Another example is the problem alluded to in relation to the consensus about the historical Jesus. When a field is dominated by people whose livelihood depends on their continuing to espouse a certain belief, the effect goes beyond those individuals and infects the whole field.
2. The pernicious effects of “great men” who can suppress dissent against their out of date views. Is the field pluralistic, realistically, and is dissent allowed? Science advances funeral by funeral. Have a look at what happened to John “Pure white and deadly” Yudkin.
3. Politics beyond what we normally think of as politics. Academia is notoriously “political” in this wider sense. Amplifying your point about reality checks, if feedback is not accurate, rapid, and unambiguous, it is hard for people in the field to know who is right, if anyone.
4. “Publish” or perish. There are massive incentives to get published or to get publicity or a high profile. This leads to people claiming expertise, results, achievements that are bogus. Consider for example the case of Theranos, which seemed, if media reports are accurate, to have no useful ability to build systems that did pathology tests, yet apparently hoodwinked many into thinking that they did.
You make a good point that claims of expertise without evidence or, worse, in the face of adverse evidence, are really really bad. I would go as far as to say that if you claim expertise but cannot prove it, I have a strong prior that you don’t have it.
There are large groups of self-described experts who do not have expertise or at best have far less than they think. One should be alert to the possibility that “experts” aren’t.
Also to reinforce a very important point: even when experts are not very expert, they are probably a lot better than you+google+30minutes!