Great post! I really enjoy your writing style. I agree with everything up to your last sentence of cooperative epistemics. It looks like a false equivalence between a community of perfect trust and a community based on mistrust. I’m thinking a community of “trust but verify” with a vague assumption of goodwill will capture all the benefits of mistrust without the risks of half rationalists or “half a forum of autists” going off the deep end and making a carrying error in their EV calculations to overly negative results.
Corrupted Hardware leads me to think we need to aim high to end up at an optimum level of honesty.
Edit: Thanks Cole and Shankar.
Great article, I found the decision theory behind, if they think I think they think etc very interesting. I’m a bit confused about the knight of faith. In my mental model, people who look like the knight of faith aren’t accepting the situation is hopeless, but rather powering on through some combination of mentally minimizing barriers, pinning hopes on small odds and wishful thinking.
For example lets put it in the context of flipping 10 coins.
Rationalist—I’m expecting 5 heads
My model of knight of faith—I’m expecting 10 heads because there’s a slight chance and I really need it to be true.
Knight of faith as described—I’m expecting 20 heads and I’m going to base my decisions on 20 heads actually happening.
Maybe that’s an obvious distinction but in that case why bring up the knight of faith and instead just focus on the power of wishful thinking in some situations.