I had written up an overview of my current approach/worldview on my shortform page. Reposting here for ease of reference. (I think this is sort of covered in the OP, but it’s interwoven with various disagreement-untanglings that I think make it harder to parse)
The rationalsphere isn’t great at applying rationality to its own internal politics
We don’t seem to do much better than average. This seems like something that’s at least pretty sad, even if it’s a true brute fact about the world.
There have been some efforts to fix this fact, but most of it has seemed (to me) to be missing key facts about game theory, common knowledge, theory of mind, and some other topics that I see as necessary to solve the problem.
Billions of dollars are at stake, which creates important distortions that need addressing
The rationality and EA communities are valuable, in large part, because there is an opportunity for important ideas to influence the world-stage, moving millions or billions of dollars (or causing millions of dollars worth of stuff happening). But, when billions of dollars are at stake, you start attract world-class opportunists trying to coopt you, (as well as community members start feeling pressure to conform to social reality on the world-stage), which demands world-class ability to handle subtle political pressures to preserve that value.
[epistemic status: I’m not sure whether I endorse the rhetoric here. Maybe you don’t need to be world class, but you probably need to be at least 75th percentile, and/or become more illegible to the forces that would try to coopt you]
By default, we don’t seem very good at allocating attention re: these issues. But, the attempts to address that I’ve seen seem doomy.
One of the default failure modes that I’ve seen is, when people don’t pay attention to a given call-for-clarity about “hey, we seem to be acting in ways that distort truth in predictable ways”, is to jump all the way to statements like “EA has a lying problem,” which I think is both untrue and anti-helpful for preserving a truthseeking space.
In that case Sarah later wrote up a followup post that was more reasonable and Benquo wrote up a post that articulated the problem more clearly. [Can’t find the links offhand]. But it was a giant red flag for me that getting people to pay attention required sensationalizing the problem. It seemed to me that this was following an incentive gradient identical to political news. This seemed roughly as bad for truthseeking as the original problem Sarah was trying to address was, both because:
“clickbaitiness” is a bad process for allocating attention
The “Rationalization/Sitting-bolt-upright” post was intended to provide an outlet for that sort of impulse that was at less counterproductive (in the interim before figuring out a more robust solution).
A first guess at a “robust solution” is something like “develop clear norms for good, fair practices to critiquing organizations.” If you meet those norms, posts on LessWrong that deal with local politics can get curated.
By default, people use language for both truthseeking and for politics. It takes special effort to keep things truth-focused
A primary lesson I learned from the sequences is that most people’s beliefs and statements are not about truth at all. (“Science as attire”, “Fable of Science and Politics”, etc. Most of the places where the rationalsphere seems most truth-tracking are where it sidesteps this issue, rather than really solving it. Attempting to directly jump towards “well we just use words for truth, not politics”, sound to me about as promising as writing the word ‘cold’ on a fridge.
Relatedly, I think people struggle to stay in a truthseeking frame when they are feeling defensive. One person being defensive makes it 2-30x harder to remain truth-oriented. Multiple people being defensive at least add up that difficulty linearly, and potentially compound in weirder ways. I think this is challenging enough that it requires joint effort to avoid.
A truthseeking space that can actually discuss politics sanely needs both individuals who are putting special effort to avoid being defensive, and conversation partners that practice avoiding unnecessarily* provoking defensiveness.
*where by “unnecessary” I mean: “if your subject matter is inherently difficult to hear, you shouldn’t avoid saying it. But you should avoid saying it with rhetoric that is especially liable to inflame the conversation. (i.e. “i think your project is net-harmful” is fine. “I think your project is stupid and can’t believe you wasted our time on it” is making the conversation 20x harder, unnecessarily.)
Yes, this is hard and doesn’t come naturally to everyone. But I think it’s at least approximately as hard as learning to avoid getting defensive is (and I would guess the low-hanging fruit is actually comparatively easy). I think if a truthseeking space doesn’t ask people to at least pick up the low-hanging fruit here, it will be less effective as a truthseeking space.
I don’t think this is necessary for all conversations, but it’s escalatingly important the less the participants trust each other and the higher the stakes.
If conversation participants are not making efforts to pass each other’s Ideological Turing Test, my default assumption is no progress will get made
Communicating between frames/aesthetics/ontologies are very hard
Common knowledge of ‘Double Crux’ has made it somewhat easier to resolve gnarly disagreements, but I still frequently observe rationalists (myself included) just completely talking past each other, not noticing, and then either getting really frustrated, or assuming bad faith when the actual problem is significantly different world models.
There’s something of a skill to identifying what framework you are working under, which is somewhat separate from the usual doublecrux process.
I also think there’s a skill to figuring out how to backpropagate facts into your frame/taste/and-or/ontology, which I think helpful for resolving major frame disagreements. (But dramatically more helpful if both parties are doing it)
Difficulty communicating between frames exacerbates the difficulty of discussing local politics sanely
Different frames have quite different takes on which rocks and hard places are more scary. By default, when the frames bump into each other, they see each other as trampling all over each other’s obvious needs.
Meanwhile, someone communicating in a different frame from you will seem to be missing the point, or subtly off, in a way that is hard to put your finger on, which makes the whole communicating process feel like moving through molasses.
I think having more people with the taste/ontology doublecrux skill would enable more trust that conversations across frames are happening in good faith
Counterfactual revolutions are super important. Real revolutions are mostly bad.
Despite all the above, we’re unlikely to reach a state where everyone can easily communicate across frames. Even if we did, it wouldn’t guarantee that people actually were talking in good faith – sometimes people actually are actively-deceptive, or stuck in a harmful pattern that they can’t escape from. This is particularly problematic when they’re in power.
I think we’re in a rare, precious world where it’s actually achievable for the major power centers in the EA space to communicate sanely to each other.
So, it’s simultaneously important to have a gameboard with rules that everyone respects, but simultaneously, it’s important that there be a real threat of people kicking the gameboard over if the game seems rigged.
I had written up an overview of my current approach/worldview on my shortform page. Reposting here for ease of reference. (I think this is sort of covered in the OP, but it’s interwoven with various disagreement-untanglings that I think make it harder to parse)
The rationalsphere isn’t great at applying rationality to its own internal politics
We don’t seem to do much better than average. This seems like something that’s at least pretty sad, even if it’s a true brute fact about the world.
There have been some efforts to fix this fact, but most of it has seemed (to me) to be missing key facts about game theory, common knowledge, theory of mind, and some other topics that I see as necessary to solve the problem.
Billions of dollars are at stake, which creates important distortions that need addressing
The rationality and EA communities are valuable, in large part, because there is an opportunity for important ideas to influence the world-stage, moving millions or billions of dollars (or causing millions of dollars worth of stuff happening). But, when billions of dollars are at stake, you start attract world-class opportunists trying to coopt you, (as well as community members start feeling pressure to conform to social reality on the world-stage), which demands world-class ability to handle subtle political pressures to preserve that value.
[epistemic status: I’m not sure whether I endorse the rhetoric here. Maybe you don’t need to be world class, but you probably need to be at least 75th percentile, and/or become more illegible to the forces that would try to coopt you]
By default, we don’t seem very good at allocating attention re: these issues. But, the attempts to address that I’ve seen seem doomy.
One of the default failure modes that I’ve seen is, when people don’t pay attention to a given call-for-clarity about “hey, we seem to be acting in ways that distort truth in predictable ways”, is to jump all the way to statements like “EA has a lying problem,” which I think is both untrue and anti-helpful for preserving a truthseeking space.
In that case Sarah later wrote up a followup post that was more reasonable and Benquo wrote up a post that articulated the problem more clearly. [Can’t find the links offhand]. But it was a giant red flag for me that getting people to pay attention required sensationalizing the problem. It seemed to me that this was following an incentive gradient identical to political news. This seemed roughly as bad for truthseeking as the original problem Sarah was trying to address was, both because:
“lying” is a wrong characterization of what was happening (see here for my sense of why it’s useful to distinguish lies from motivated cognition or other forms of deception)
“clickbaitiness” is a bad process for allocating attention
The “Rationalization/Sitting-bolt-upright” post was intended to provide an outlet for that sort of impulse that was at less counterproductive (in the interim before figuring out a more robust solution).
A first guess at a “robust solution” is something like “develop clear norms for good, fair practices to critiquing organizations.” If you meet those norms, posts on LessWrong that deal with local politics can get curated.
By default, people use language for both truthseeking and for politics. It takes special effort to keep things truth-focused
A primary lesson I learned from the sequences is that most people’s beliefs and statements are not about truth at all. (“Science as attire”, “Fable of Science and Politics”, etc. Most of the places where the rationalsphere seems most truth-tracking are where it sidesteps this issue, rather than really solving it. Attempting to directly jump towards “well we just use words for truth, not politics”, sound to me about as promising as writing the word ‘cold’ on a fridge.
Relatedly, I think people struggle to stay in a truthseeking frame when they are feeling defensive. One person being defensive makes it 2-30x harder to remain truth-oriented. Multiple people being defensive at least add up that difficulty linearly, and potentially compound in weirder ways. I think this is challenging enough that it requires joint effort to avoid.
A truthseeking space that can actually discuss politics sanely needs both individuals who are putting special effort to avoid being defensive, and conversation partners that practice avoiding unnecessarily* provoking defensiveness.
*where by “unnecessary” I mean: “if your subject matter is inherently difficult to hear, you shouldn’t avoid saying it. But you should avoid saying it with rhetoric that is especially liable to inflame the conversation. (i.e. “i think your project is net-harmful” is fine. “I think your project is stupid and can’t believe you wasted our time on it” is making the conversation 20x harder, unnecessarily.)
Yes, this is hard and doesn’t come naturally to everyone. But I think it’s at least approximately as hard as learning to avoid getting defensive is (and I would guess the low-hanging fruit is actually comparatively easy). I think if a truthseeking space doesn’t ask people to at least pick up the low-hanging fruit here, it will be less effective as a truthseeking space.
I don’t think this is necessary for all conversations, but it’s escalatingly important the less the participants trust each other and the higher the stakes.
If conversation participants are not making efforts to pass each other’s Ideological Turing Test, my default assumption is no progress will get made
Communicating between frames/aesthetics/ontologies are very hard
Common knowledge of ‘Double Crux’ has made it somewhat easier to resolve gnarly disagreements, but I still frequently observe rationalists (myself included) just completely talking past each other, not noticing, and then either getting really frustrated, or assuming bad faith when the actual problem is significantly different world models.
There’s something of a skill to identifying what framework you are working under, which is somewhat separate from the usual doublecrux process.
I also think there’s a skill to figuring out how to backpropagate facts into your frame/taste/and-or/ontology, which I think helpful for resolving major frame disagreements. (But dramatically more helpful if both parties are doing it)
Difficulty communicating between frames exacerbates the difficulty of discussing local politics sanely
Different frames have quite different takes on which rocks and hard places are more scary. By default, when the frames bump into each other, they see each other as trampling all over each other’s obvious needs.
Meanwhile, someone communicating in a different frame from you will seem to be missing the point, or subtly off, in a way that is hard to put your finger on, which makes the whole communicating process feel like moving through molasses.
I think having more people with the taste/ontology doublecrux skill would enable more trust that conversations across frames are happening in good faith
Counterfactual revolutions are super important. Real revolutions are mostly bad.
Despite all the above, we’re unlikely to reach a state where everyone can easily communicate across frames. Even if we did, it wouldn’t guarantee that people actually were talking in good faith – sometimes people actually are actively-deceptive, or stuck in a harmful pattern that they can’t escape from. This is particularly problematic when they’re in power.
I think we’re in a rare, precious world where it’s actually achievable for the major power centers in the EA space to communicate sanely to each other.
So, it’s simultaneously important to have a gameboard with rules that everyone respects, but simultaneously, it’s important that there be a real threat of people kicking the gameboard over if the game seems rigged.