I started writing this a few weeks ago. By now I have other posts that make these points more cleanly in the works, and I’m in the process of thinking through some new thoughts that might revise bits of this.
But I think it’s going to be awhile before I can articulate all that. So meanwhile, here’s a quick summary of the overall thesis I’m building towards (with the “Rationalization” and “Sitting Bolt Upright in Alarm” post, and other posts and conversations that have been in the works).
(By now I’ve had fairly extensive chats with Jessicata and Benquo and I don’t expect this to add anything that I didn’t discuss there, so this is more for other people who’re interested in staying up to speed. I’m separately working on a summary of my current epistemic state after those chats)
The rationalsphere isn’t great at applying rationality to its own internal politics
We don’t seem to do much better than average. This seems like something that’s at least pretty sad, even if it’s a true brute fact about the world.
There have been some efforts to fix this fact, but most of it has seemed (to me) to be missing key facts about game theory, common knowledge, theory of mind, and some other topics that I see as necessary to solve the problem.
Billions of dollars are at stake, which creates important distortions that need addressing
The rationality and EA communities are valuable, in large part, because there is an opportunity for important ideas to influence the world-stage, moving millions or billions of dollars (or causing millions of dollars worth of stuff happening). But, when billions of dollars are at stake, you start attract world-class opportunists trying to coopt you, (as well as community members start feeling pressure to conform to social reality on the world-stage), which demands world-class ability to handle subtle political pressures to preserve that value.
[epistemic status: I’m not sure whether I endorse the rhetoric here. Maybe you don’t need to be world class, but you probably need to be at least 75th percentile, and/or become more illegible to the forces that would try to coopt you]
By default, we don’t seem very good at allocating attention re: these issues. But, the attempts to address that I’ve seen seem doomy.
One of the default failure modes that I’ve seen is, when people don’t pay attention to a given call-for-clarity about “hey, we seem to be acting in ways that distort truth in predictable ways”, is to jump all the way to statements like “EA has a lying problem,” which I think is both untrue and anti-helpful for preserving a truthseeking space.
In that case Sarah later wrote up a followup post that was more reasonable and Benquo wrote up a post that articulated the problem more clearly. [Can’t find the links offhand]. But it was a giant red flag for me that getting people to pay attention required sensationalizing the problem. It seemed to me that this was following an incentive gradient identical to political news. This seemed roughly as bad for truthseeking as the original problem Sarah was trying to address was, both because:
“clickbaitiness” is a bad process for allocating attention
The “Rationalization/Sitting-bolt-upright” post was intended to provide an outlet for that sort of impulse that was at less counterproductive (in the interim before figuring out a more robust solution).
A first guess at a “robust solution” is something like “develop clear norms for good, fair practices to critiquing organizations.” If you meet those norms, posts on LessWrong that deal with local politics can get curated.
By default, people use language for both truthseeking and for politics. It takes special effort to keep things truth-focused
A primary lesson I learned from the sequences is that most people’s beliefs and statements are not about truth at all. (“Science as attire”, “Fable of Science and Politics”, etc. Most of the places where the rationalsphere seems most truth-tracking are where it sidesteps this issue, rather than really solving it. Attempting to directly jump towards “well we just use words for truth, not politics”, sound to me about as promising as writing the word ‘cold’ on a fridge.
Relatedly, I think people struggle to stay in a truthseeking frame when they are feeling defensive. One person being defensive makes it 2-30x harder to remain truth-oriented. Multiple people being defensive at least add up that difficulty linearly, and potentially compound in weirder ways. I think this is challenging enough that it requires joint effort to avoid.
A truthseeking space that can actually discuss politics sanely needs both individuals who are putting special effort to avoid being defensive, and conversation partners that practice avoiding unnecessarily* provoking defensiveness.
*where by “unnecessary” I mean: “if your subject matter is inherently difficult to hear, you shouldn’t avoid saying it. But you should avoid saying it with rhetoric that is especially liable to inflame the conversation. (i.e. “i think your project is net-harmful” is fine. “I think your project is stupid and can’t believe you wasted our time on it” is making the conversation 20x harder, unnecessarily.)
Yes, this is hard and doesn’t come naturally to everyone. But I think it’s at least approximately as hard as learning to avoid getting defensive is (and I would guess the low-hanging fruit is actually comparatively easy). I think if a truthseeking space doesn’t ask people to at least pick up the low-hanging fruit here, it will be less effective as a truthseeking space.
I don’t think this is necessary for all conversations, but it’s escalatingly important the less the participants trust each other and the higher the stakes.
If conversation participants are not making efforts to pass each other’s Ideological Turing Test, my default assumption is no progress will get made
Communicating between frames/aesthetics/ontologies are very hard
Common knowledge of ‘Double Crux’ has made it somewhat easier to resolve gnarly disagreements, but I still frequently observe rationalists (myself included) just completely talking past each other, not noticing, and then either getting really frustrated, or assuming bad faith when the actual problem is significantly different world models.
There’s something of a skill to identifying what framework you are working under, which is somewhat separate from the usual doublecrux process.
I also think there’s a skill to figuring out how to backpropagate facts into your frame/taste/and-or/ontology, which I think helpful for resolving major frame disagreements. (But dramatically more helpful if both parties are doing it)
Difficulty communicating between frames exacerbates the difficulty of discussing local politics sanely
Different frames have quite different takes on which rocks and hard places are more scary. By default, when the frames bump into each other, they see each other as trampling all over each other’s obvious needs.
Meanwhile, someone communicating in a different frame from you will seem to be missing the point, or subtly off, in a way that is hard to put your finger on, which makes the whole communicating process feel like moving through molasses.
I think having more people with the taste/ontology doublecrux skill would enable more trust that conversations across frames are happening in good faith
Counterfactual revolutions are super important. Real revolutions are mostly bad.
Despite all the above, we’re unlikely to reach a state where everyone can easily communicate across frames. Even if we did, it wouldn’t guarantee that people actually were talking in good faith – sometimes people actually are actively-deceptive, or stuck in a harmful pattern that they can’t escape from. This is particularly problematic when they’re in power.
I think we’re in a rare, precious world where it’s actually achievable for the major power centers in the EA space to communicate sanely to each other.
So, it’s simultaneously important to have a gameboard with rules that everyone respects, but simultaneously, it’s important that there be a real threat of people kicking the gameboard over if the game seems rigged.
In that case Sarah later wrote up a followup post that was more reasonable and Benquo wrote up a post that articulated the problem more clearly. [Can’t find the links offhand].
Thanks! I do still pretty* much endorse “Between Honesty and Perjury.”
*avoiding making a stronger claim here since I only briefly re-read it and haven’t re-thought-through each particular section and claim. But the overall spirit it’s pointing to is quite important.
[Edit: Ah, well, in the comments there I apparently expressed some specific agreements and disagreements that seems… similar in shape to my current agreement and disagreement with Ben. But I think in the intervening years I’ve updated a bit towards “EA’s epistemic standards should be closer to Ben’s standards than I thought in 2017.”]
Thank you for the effort and clarity of thought you’re putting into this. One thing you may already be considering, but I haven’t seen it addressed directly:
Hobbyists vs fanatics vs professionals (or core/periphery, or founders/followers/exploiters, or any other acknowledgement of different individual capabilities and motives). What parts of “the community” are you talking about when you address various issues? You hint at this in the money/distortion topic, but you’re in danger of abstracting “motivation” way too far, and missing the important details of individual variation.
Also, it’s possible that you’re overestimating the need for legibility of reasoning over correctness of action (in the rational sense, of furthering one’s true goals). I very much dispute “We don’t seem to do much better than average”, unless you’re seriously cherry-picking your reference set. We do _WAY_ better than average both in terms of impact and in terms of transparency of reasoning. I’d love to explore some benchmarks (and copy some behaviors) if you can identify groups with similar composition and similar difficult-to-quantify goals, that are far more effective
I started writing this a few weeks ago. By now I have other posts that make these points more cleanly in the works, and I’m in the process of thinking through some new thoughts that might revise bits of this.
But I think it’s going to be awhile before I can articulate all that. So meanwhile, here’s a quick summary of the overall thesis I’m building towards (with the “Rationalization” and “Sitting Bolt Upright in Alarm” post, and other posts and conversations that have been in the works).
(By now I’ve had fairly extensive chats with Jessicata and Benquo and I don’t expect this to add anything that I didn’t discuss there, so this is more for other people who’re interested in staying up to speed. I’m separately working on a summary of my current epistemic state after those chats)
The rationalsphere isn’t great at applying rationality to its own internal politics
We don’t seem to do much better than average. This seems like something that’s at least pretty sad, even if it’s a true brute fact about the world.
There have been some efforts to fix this fact, but most of it has seemed (to me) to be missing key facts about game theory, common knowledge, theory of mind, and some other topics that I see as necessary to solve the problem.
Billions of dollars are at stake, which creates important distortions that need addressing
The rationality and EA communities are valuable, in large part, because there is an opportunity for important ideas to influence the world-stage, moving millions or billions of dollars (or causing millions of dollars worth of stuff happening). But, when billions of dollars are at stake, you start attract world-class opportunists trying to coopt you, (as well as community members start feeling pressure to conform to social reality on the world-stage), which demands world-class ability to handle subtle political pressures to preserve that value.
[epistemic status: I’m not sure whether I endorse the rhetoric here. Maybe you don’t need to be world class, but you probably need to be at least 75th percentile, and/or become more illegible to the forces that would try to coopt you]
By default, we don’t seem very good at allocating attention re: these issues. But, the attempts to address that I’ve seen seem doomy.
One of the default failure modes that I’ve seen is, when people don’t pay attention to a given call-for-clarity about “hey, we seem to be acting in ways that distort truth in predictable ways”, is to jump all the way to statements like “EA has a lying problem,” which I think is both untrue and anti-helpful for preserving a truthseeking space.
In that case Sarah later wrote up a followup post that was more reasonable and Benquo wrote up a post that articulated the problem more clearly. [Can’t find the links offhand]. But it was a giant red flag for me that getting people to pay attention required sensationalizing the problem. It seemed to me that this was following an incentive gradient identical to political news. This seemed roughly as bad for truthseeking as the original problem Sarah was trying to address was, both because:
“lying” is a wrong characterization of what was happening (see here for my sense of why it’s useful to distinguish lies from motivated cognition or other forms of deception)
“clickbaitiness” is a bad process for allocating attention
The “Rationalization/Sitting-bolt-upright” post was intended to provide an outlet for that sort of impulse that was at less counterproductive (in the interim before figuring out a more robust solution).
A first guess at a “robust solution” is something like “develop clear norms for good, fair practices to critiquing organizations.” If you meet those norms, posts on LessWrong that deal with local politics can get curated.
By default, people use language for both truthseeking and for politics. It takes special effort to keep things truth-focused
A primary lesson I learned from the sequences is that most people’s beliefs and statements are not about truth at all. (“Science as attire”, “Fable of Science and Politics”, etc. Most of the places where the rationalsphere seems most truth-tracking are where it sidesteps this issue, rather than really solving it. Attempting to directly jump towards “well we just use words for truth, not politics”, sound to me about as promising as writing the word ‘cold’ on a fridge.
Relatedly, I think people struggle to stay in a truthseeking frame when they are feeling defensive. One person being defensive makes it 2-30x harder to remain truth-oriented. Multiple people being defensive at least add up that difficulty linearly, and potentially compound in weirder ways. I think this is challenging enough that it requires joint effort to avoid.
A truthseeking space that can actually discuss politics sanely needs both individuals who are putting special effort to avoid being defensive, and conversation partners that practice avoiding unnecessarily* provoking defensiveness.
*where by “unnecessary” I mean: “if your subject matter is inherently difficult to hear, you shouldn’t avoid saying it. But you should avoid saying it with rhetoric that is especially liable to inflame the conversation. (i.e. “i think your project is net-harmful” is fine. “I think your project is stupid and can’t believe you wasted our time on it” is making the conversation 20x harder, unnecessarily.)
Yes, this is hard and doesn’t come naturally to everyone. But I think it’s at least approximately as hard as learning to avoid getting defensive is (and I would guess the low-hanging fruit is actually comparatively easy). I think if a truthseeking space doesn’t ask people to at least pick up the low-hanging fruit here, it will be less effective as a truthseeking space.
I don’t think this is necessary for all conversations, but it’s escalatingly important the less the participants trust each other and the higher the stakes.
If conversation participants are not making efforts to pass each other’s Ideological Turing Test, my default assumption is no progress will get made
Communicating between frames/aesthetics/ontologies are very hard
Common knowledge of ‘Double Crux’ has made it somewhat easier to resolve gnarly disagreements, but I still frequently observe rationalists (myself included) just completely talking past each other, not noticing, and then either getting really frustrated, or assuming bad faith when the actual problem is significantly different world models.
There’s something of a skill to identifying what framework you are working under, which is somewhat separate from the usual doublecrux process.
I also think there’s a skill to figuring out how to backpropagate facts into your frame/taste/and-or/ontology, which I think helpful for resolving major frame disagreements. (But dramatically more helpful if both parties are doing it)
Difficulty communicating between frames exacerbates the difficulty of discussing local politics sanely
Different frames have quite different takes on which rocks and hard places are more scary. By default, when the frames bump into each other, they see each other as trampling all over each other’s obvious needs.
Meanwhile, someone communicating in a different frame from you will seem to be missing the point, or subtly off, in a way that is hard to put your finger on, which makes the whole communicating process feel like moving through molasses.
I think having more people with the taste/ontology doublecrux skill would enable more trust that conversations across frames are happening in good faith
Counterfactual revolutions are super important. Real revolutions are mostly bad.
Despite all the above, we’re unlikely to reach a state where everyone can easily communicate across frames. Even if we did, it wouldn’t guarantee that people actually were talking in good faith – sometimes people actually are actively-deceptive, or stuck in a harmful pattern that they can’t escape from. This is particularly problematic when they’re in power.
I think we’re in a rare, precious world where it’s actually achievable for the major power centers in the EA space to communicate sanely to each other.
So, it’s simultaneously important to have a gameboard with rules that everyone respects, but simultaneously, it’s important that there be a real threat of people kicking the gameboard over if the game seems rigged.
“Reply to Criticism on my EA Post”, “Between Honesty and Perjury”
Thanks! I do still pretty* much endorse “Between Honesty and Perjury.”
*avoiding making a stronger claim here since I only briefly re-read it and haven’t re-thought-through each particular section and claim. But the overall spirit it’s pointing to is quite important.
[Edit: Ah, well, in the comments there I apparently expressed some specific agreements and disagreements that seems… similar in shape to my current agreement and disagreement with Ben. But I think in the intervening years I’ve updated a bit towards “EA’s epistemic standards should be closer to Ben’s standards than I thought in 2017.”]
Thank you for the effort and clarity of thought you’re putting into this. One thing you may already be considering, but I haven’t seen it addressed directly:
Hobbyists vs fanatics vs professionals (or core/periphery, or founders/followers/exploiters, or any other acknowledgement of different individual capabilities and motives). What parts of “the community” are you talking about when you address various issues? You hint at this in the money/distortion topic, but you’re in danger of abstracting “motivation” way too far, and missing the important details of individual variation.
Also, it’s possible that you’re overestimating the need for legibility of reasoning over correctness of action (in the rational sense, of furthering one’s true goals). I very much dispute “We don’t seem to do much better than average”, unless you’re seriously cherry-picking your reference set. We do _WAY_ better than average both in terms of impact and in terms of transparency of reasoning. I’d love to explore some benchmarks (and copy some behaviors) if you can identify groups with similar composition and similar difficult-to-quantify goals, that are far more effective