In an environment with very low epistemic-instrumental separation of concerns, an especially intelligent individual actually has an incentive to insulate their own epistemics (become good at lying and acting), so that they can think. Optimizing for true beliefs then becomes highly instrumentally valuable (modulo the cost of keeping up the barrier / the downside risk if discovered).
Also, still thinking about the environment where the overarching social structure doesn’t support much separation of concerns, there’s still a value to be had in associating with individuals who often naively speak the truth (not the liar/actor type), because there’s a lot of misaligned preferences floating around. Your incentives for bending the truth are different from another person’s, so you prefer to associate with others who don’t bend the truth much (especially if you internally separate concerns). So, separation of concerns seems like a convergent instrumental goal which will be bubbling under the surface even in a dysfunctional superstructure.
Of course, both effects are limited in their ability encourage truth, and it’s a little like arguing that cooperation in the prisoner’s dilemma is a convergent instrumental subgoal bubbling under the surface of a defective equilibrium.
In an environment with very low epistemic-instrumental separation of concerns, an especially intelligent individual actually has an incentive to insulate their own epistemics (become good at lying and acting), so that they can think. Optimizing for true beliefs then becomes highly instrumentally valuable (modulo the cost of keeping up the barrier / the downside risk if discovered).
Also, still thinking about the environment where the overarching social structure doesn’t support much separation of concerns, there’s still a value to be had in associating with individuals who often naively speak the truth (not the liar/actor type), because there’s a lot of misaligned preferences floating around. Your incentives for bending the truth are different from another person’s, so you prefer to associate with others who don’t bend the truth much (especially if you internally separate concerns). So, separation of concerns seems like a convergent instrumental goal which will be bubbling under the surface even in a dysfunctional superstructure.
Of course, both effects are limited in their ability encourage truth, and it’s a little like arguing that cooperation in the prisoner’s dilemma is a convergent instrumental subgoal bubbling under the surface of a defective equilibrium.