I have come to believe that people’s ability to come to correct opinions about important questions is in large part a result of whether their social and monetary incentives reward them when they have accurate models in a specific domain. This means a person can have extremely good opinions in one domain of reality, because they are subject to good incentives, while having highly inaccurate models in a large variety of other domains in which their incentives are not well optimized.
People’s rationality is much more defined by their ability to maneuver themselves into environments in which their external incentives align with their goals, than by their ability to have correct opinions while being subject to incentives they don’t endorse. This is a tractable intervention and so the best people will be able to have vastly more accurate beliefs than the average person, but it means that “having accurate beliefs in one domain” doesn’t straightforwardly generalize to “will have accurate beliefs in other domains”.
One is strongly predictive of the other, and that’s in part due to general thinking skills and broad cognitive ability. But another major piece of the puzzle is the person’s ability to build and seek out environments with good incentive structures.
Everyone is highly irrational in their beliefs about at least some aspects of reality, and positions of power in particular tend to encourage strong incentives that don’t tend to be optimally aligned with the truth. This means that highly competent people in positions of power often have less accurate beliefs than competent people who are not in positions of power.
The design of systems that hold people who have power and influence accountable in a way that aligns their interests with both forming accurate beliefs and the interests of humanity at large is a really important problem, and is a major determinant of the overall quality of the decision-making ability of a community. General rationality training helps, but for collective decision making the creation of accountability systems, the tracking of outcome metrics and the design of incentives is at least as big of a factor as the degree to which the individual members of the community are able to come to accurate beliefs on their own.
Hah, I was thinking of replying to say I was largely just repeating things you said in that post.
Nonetheless, thanks both Kaj and Eric, I might turn it into a little post. It’s not bad to have two posts saying the same thing (slightly differently).
This post of mine feels closely related: https://www.lesswrong.com/posts/xhE4TriBSPywGuhqi/integrity-and-accountability-are-core-parts-of-rationality
Hah, I was thinking of replying to say I was largely just repeating things you said in that post.
Nonetheless, thanks both Kaj and Eric, I might turn it into a little post. It’s not bad to have two posts saying the same thing (slightly differently).