It’s kind of funny to me to see this one nominated. It’s sort of peak “Val is weird on LW”.
The point of this post wasn’t to offer claims for people to examine. I still agree with the claims I see myself having tried to make! But the point wasn’t to offer ideas for discussion. It was to light a path out of Hell.
Because of that purpose, the style of this post really doesn’t fit LW culture. I think it’s fair to call it a mind spell. I get the impression that LWers in particular find mind spells unnerving: they’re a symmetric tech that can do an end-run around the parts of cognition that rationalists heavily rely on to feel safe. Hence tripping the “cult”/”guru” immune reaction.
(To me it’s dead obvious that this highlights a gap in the LW rationality toolbox. The reaction of “Lock down, distrust, get cynical, burn it with fire” actually makes you more susceptible to skillful bad actors — like going rigid in response to a judo master grabbing a hold of you. IMO, a mature Art of Rationality would necessarily include learning to navigate cognition-jamming (or cognition-incompatible!) spaces with grace. But I get the sense LW collectively doesn’t want to build that skillset. Which is fine, but I find it a bit disappointing.)
I picked up some of the language & framing of this post from Perri Chase. I now talk about this stuff a little differently. And more kindly, I think. I suspect I could write a version of this spell today that would be less of a problem for the LW memetic immune system. Partly because I’m better at slipping through immune systems! (I’m sure that’s comforting!) But mostly because I’ve learned how to work with such systems instead of needing to step around them to have the “real” conversation.
That said, I don’t regret writing this post. I got a lot of feedback (including in quite a few PMs across many different media) from people who found this relieving, validating, soothing, deeply helpful, kind, orienting. I’m okay with some people being upset with me if that’s the price for enacting this kindness. I went in expecting that price, really.
I think there’s a post possible that would be something like a LW-compatible rewrite of this one. It’d remove the “spell” nature and try to lay out some claims & implications for folk to consider. A bit like dissecting a once-living specimen and laying out its organs for examination.
I probably won’t write that post. I don’t see it doing hardly any good beyond being kind of interesting.
I might write a related post sometime on the nature of Hell as a psychosocial attractor state. AFAICT it’s utterly essential study for real Defense Against the Dark Arts. It’s also very tricky to talk about in a way that’s kind to the listeners or the speaker. But if LW were to learn to take it seriously without falling into it harder, I think that awareness would transform a lot of what “rationality” means here, and it would soften a lot of the sharp edges that can meaningfully hurt people here.
I don’t plan on rewriting any of this post for the review. The spell worked great. I want to leave it here as is.
(Though if someone understands the spellcraft and wants to suggest some edits, I’m open to receiving those suggestions! I’m not putting up a wall here. I’m just sharing where I’m at with this post right now, for the sake of the 2022 review.)
To me it’s dead obvious that this highlights a gap in the LW rationality toolbox. The reaction of “Lock down, distrust, get cynical, burn it with fire” actually makes you more susceptible to skillful bad actors — like going rigid in response to a judo master grabbing a hold of you. IMO, a mature Art of Rationality would necessarily include learning to navigate cognition-jamming (or cognition-incompatible!) spaces with grace. But I get the sense LW collectively doesn’t want to build that skillset. Which is fine, but I find it a bit disappointing.
Mm, this sounds to me like saying “a master rationalist could surround themself with con artists and frauds and other epistemically adversarial actors who were gaming the rationalist, and still have perfectly true beliefs”, and that may be true, but I think another pretty good option is “a master rationalist would definitely avoid surrounding themselves with con artists and frauds and other adversarial actors”.
I do think there are real skills you are pointing to, but to some extent I prefer the world where I don’t have those skills and in place of that my allies and I coordinate to identify and exclude people who are using the dark arts.
(I don’t say this as the ‘last word’ on the subject and expect you would produce a substantive and interesting counterargument if you chose to engage on this, I nonetheless thought I’d share that I currently disagree with what I perceive this paragraph to be saying.)
…I think another pretty good option is “a master rationalist would definitely avoid surrounding themselves with con artists and frauds and other adversarial actors”.
I think that’s a great option. I’d question a “master rationalist’s” skills if they couldn’t avoid such adversarial actors, or notice them if they slip through the cracks.
I do think there are real skills you are pointing to, but to some extent I prefer the world where I don’t have those skills and in place of that my allies and I coordinate to identify and exclude people who are using the dark arts.
I like your preference. I’ll say some things, but I want to start by emphasizing that I don’t think you’re making a wrong or bad choice.
I want to talk about what I think the Art could be, kind of for aesthetic reasons. This isn’t to assert anything about what you or any given individual should or shouldn’t be doing in any kind of moral sense.
So with that said, here are three points:
(1) I think there’s a strong analogy here to studying combat and war. Yes, if you can be in a pacifist cluster and just exclude folk who are really into applied competitive strategy, then you have something kind of like a cooperate/cooperate equilibrium. But if that’s the whole basis of your culture, it’s extremely vulnerable, the way cooperate-bot is vulnerable in prisoners’ dilemmas. You need military strength, the way a walled garden needs walls. Otherwise folk who have military strength can just come take your resources, even if you try to exclude them at first.
At the risk of using maybe an unfair example, I think what happened with FTX last year maybe illustrates the point.
Clearer examples in my mind are Ziz and Brent. The point not being “These people are bad!” But rather, these people were psychologically extremely potent and lots of folk in the community could neither (a) adequately navigate their impact (myself included!) nor (b) rally ejection/exclusion power until well after they’d already had their impact.
Maybe, you might hope, you can make the ejection/exclusion sensitivity refined enough to work earlier. But if you don’t do that by studying the Dark Arts, and becoming intimately familiar with them, then what you get is a kind of naïve allergic response that Dark Artists can weaponize.
Again, I don’t mean that you in particular or even rationalists in general need to address this. There’s nothing wrong with a hobby. I’m saying that as an Art, it seems like rationality is seriously vulnerable if it doesn’t include masterful familiarity with the Dark Arts. Kind of like, there’s nothing wrong with practicing aikido as a sport, but you’re not gonna get the results you hope for if you train in aikido for self-defense. That art is inadequate for that purpose and needs exposure to realistic combat to matter that way.
(2) …and I think that if the Art of Rationality were to include intimate familiarity with the Dark Arts, it would work way way better.
Things like the planning fallacy or confirmation bias are valuable to track. I could stand to improve my repertoire here for sure.
But the most potent forms of distorted thinking aren’t about sorting out the logic. I think they look more like reaching deep down and finding ways to become immune to things like frame control.
Frame control is an amazing example in my mind precisely because of the hydra-like nature of the beast. How do you defend against frame control without breaking basic things about culture and communication and trust? How do you make it so your cultural and individual defenses don’t themselves become the manual that frame controllers use to get their desired effects?
And this barely begins to touch on the kind of impact that I’d want to call “spiritual”. By which I don’t mean anything supernatural; I’m talking about the deep psychological stuff that (say) conversing with someone deep in a psilocybin trip can do to the tripper. That’s not just frame control. That’s something way deeper, like editing someone’s basic personality operating system code. And sometimes it reaches deeper even than that. And it turns out, you don’t need psychedelics to reach that deep; those chemical tools just open a door that you can open other ways, voluntarily or otherwise, sometimes just by having a conversation.
The standard rationalist defense I’ve noticed against this amounts to mental cramping. Demand everything go through cognition, and anything that seems to try to route around cognition gets a freakout/shutdown/”shame it into oblivion” kind of response. The stuff that disables this immune response is really epistemically strange — things like prefacing with “Here’s a fake framework, it’s all baloney, don’t believe anything I’m saying.” Or doing a bunch of embodied stuff to act low-status and unsure. A Dark Artist who wanted to deeply mess with this community wouldn’t have to work very hard to do some serious damage before getting detected, best as I can tell (and as community history maybe illustrates).
If this community wanted to develop the Art to actually be skillful in these areas… well, it’s hard to predict exactly what that’d create, but I’m pretty sure it’d be glorious. If I think of the Sequences as retooling skeptical materialism, I think we’d maybe see something like a retooling of the best of Buddhist psychotechnology. I think folk here might tend to underestimate how potent that could really be.
(3) It also seems relevant to me that “Dark Arts” is maybe something of a fake category. I’m not sure it even forms a coherent cluster.
Like, is being charismatic a Dark Art? It certainly can be! It can act as a temptation. It seems to be possible to cultivate charisma. But the issue isn’t that charisma is a Dark Art. It’s that charisma is mostly symmetric. So if someone has a few slightly anti-epistemic social strategies in them, and they’re charismatic, this can have a net Dark effect that’s even strategic. But this is a totally normal level of epistemic noise!
Or how about something simpler, like someone using confirmation bias in a way that benefits their beliefs? Astrology is mostly this. Is astrology a Dark Art? Is talking about astrology a Dark Art? It seems mostly just epistemically hazardous… but where’s the line between that and Dark Arts?
How about more innocent things, like when someone is trying to understand systemic racism? Is confirmation bias a helpful pattern-recognizer, or a Dark Art? Maybe it’s potentially in service to Dark Arts, but is a necessary risk to learn the patterns?
I think Vervaeke makes this point really well. The very things that allow us to notice relevance are precisely the things that allow us to be fooled. Rationality (and he explicitly cites this — even the Keith Stanovich stuff) is a literally incomputable practice of navigating both Type I and Type II errors in this balancing act between relevance realization and being fooled.
When I think of central examples of Dark Arts, I think mostly of agents who exploit this ambiguity in order to extract value from others.
…which brings me back to point (1), about this being more a matter of skill in war. The relevant issue isn’t that there are “Dark Arts”. It’s that there are unaligned agents who are trying to strategically fool you. The skill isn’t to detect a Dark toolset; it’s to detect intelligent intent to deceive and extract value.
All of which is to say:
I think a mature Art of Rationality would most definitely include something like skillful navigation of manipulation.
I don’t think every practitioner needs to master every aspect of a mature Art. Much like not all cooks need to know how to make a roux.
But an Art that has detection, exclusion, & avoidance as its only defense against Dark Artists is a much poorer & more vulnerable Art. IMO.
Thanks for the comment. I’m gonna err on the side of noting disagreements and giving brief descriptions of my perspective rather than writing something I think has a good chance of successfully persuading you of my perspective, primarily so as to actually write a reply in a timely fashion.
I don’t want to create an expectation that if you reply then you will reply to each point; rather I’d encourage you if you reply to simply reply to whichever points seem interesting or cruxy to you.
———
1) You make the analogy to having non-violent states. I concur that presently one cannot have states without militaries. I don’t see this as showing that in all domains one must maintain high offensive capabilities in order to have good defenses. I agree one needs defenses, but sometimes good defenses don’t look like “Training thousands of people how to carry out a targeted kill-strike” and instead look like “Not being tempted to reply by rude comments online” or “Checking whether a factual claim someone makes is accurate”.
You say that for LaSota and Brent that folks “could neither (a) adequately navigate their impact (myself included!) nor (b) rally ejection/exclusion power until well after they’d already had their impact” and “Maybe, you might hope, you can make the ejection/exclusion sensitivity refined enough to work earlier”.
I don’t share the sense of difficulty I read in the second of those quotes. I think the Bay Area rationalists (and most other rationalists globally) had some generally extreme lack of boundaries of any sort. The ~only legible boundaries that the Bay Area rationality scene had were (a) are you an employee at one of CFAR/MIRI, and (b) are you invited to CFAR events. MIRI didn’t have much to do with these two individuals, and I think CFAR was choosing a strategy of “we’re not really doing social policing, we’re primarily just selecting on people who have interesting ideas about rationality”. Everything else was highly social and friend-based and it was quite dramatic to ban people from your social events. The REACH was the only communityspace and if I recall correctly explicitly had no boundaries on who could be there. This is an environment where people with lots of red flags will be able to move around with much more ease than in the rest of the world.
I think these problems aren’t that hard once you have community spaces that are willing to enforce boundaries. Over the last few years I’ve run many events and spaces, and often gotten references for people who want to enter the spaces, and definitely chosen to not invite people due to concerns about ethics and responsible behavior. I don’t believe I would’ve accepted these two people into the spaces more than once or twice at most. It’s unpleasant work to enforce boundaries and I’ve made mistakes, but overall I think that there were just not many strong boundaries in these people’s way initially, and they would have been pushed back and dissuaded much earlier if there were.
2) You write:
But the most potent forms of distorted thinking aren’t about sorting out the logic. I think they look more like reaching deep down and finding ways to become immune to things like frame control.
My position is that most thinking isn’t really about reality and isn’t truth-tracking, but that if you are doing that thinking then a lot of important questions are surprisingly easy to answer. Generally doing a few fermi estimates with a few datapoints can get you pretty in touch with the relevant part of reality.
I think there’s a ton of adversarial stuff going on as well, but the primary reason that people haven’t noticed that AI is an x-risk isn’t because people are specifically trying to trick them about the domain, but because the people are not really asking themselves the question and checking.
(And also something about people not having any conception of what actions to take in the fact of a civilizational-scale problem that most of the rest of civilization is not thinking about.)
(I think there’s some argument to be made here that the primary reason people don’t think for themselves is because civilization is trying to make them go crazy, which is interesting, though I still think the solution is primarily “just make a space where you can actually think about the object level”.)
I acknowledge that there are people who are very manipulative and adversarial in illegible ways that are hard to pin down. There’s a whole discussion about offense/defense here and how it plays out. I currently expect that there are simple solutions here. As a pointer, someone I know and respect along with their partner, makes lists of people they know for whom they would not be surprised to later find out that the person did something quite manipulative/bad/unethical, and I think they’ve had some success with this. Also personally I have repeatedly kicked myself thinking “I knew that person was suspicious, why didn’t I say so earlier?” I don’t think these problems are particularly intractable and I do think people know things and I think probably there are good ways to help that info rise up and get shared (I do not claim to have solved this problem). I don’t think it requires you yourself being very skilled at engaging with manipulative people.
The standard rationalist defense I’ve noticed against this amounts to mental cramping. Demand everything go through cognition, and anything that seems to try to route around cognition gets a freakout/shutdown/”shame it into oblivion” kind of response.
Yeah I’ve seen this, and done it somewhat. I think it works in some situations, but there’s a bunch of adversarial situations it definitely doesn’t work. I do agree it seems like a false hope to think that this can be remotely sufficient.
3) I do sometimes look at people who think they’re at war a lot more than me, and they seem very paranoid and to spend so many cognitive cycles modeling ghosts and attacks that aren’t there. It seems so tiring! I suspect you and I disagree about the extent to which we are at war with people epistemically.
Another potentially relevant point here is that I tend to see large groups and institutions as the primary forces deceiving me and tricking me, and much less so individuals. I’m much more scared of Twitter winding its way into my OODA loop than I am of a selfish scheming frame controlling individual. I think it’s much easier for me to keep boundaries against individuals than I am against these much bigger and broader forces.
4)
All of which is to say:
I think a mature Art of Rationality would most definitely include something like skillful navigation of manipulation.
I don’t think every practitioner needs to master every aspect of a mature Art. Much like not all cooks need to know how to make a roux.
But an Art that has detection, exclusion, & avoidance as its only defense against Dark Artists is a much poorer & more vulnerable Art. IMO.
My perspective on these.
Personally I would like to know two or three people who have successfully navigated being manipulated, and hopefully have them write up their accounts of that.
I think aspiring rationalists should maneuver themselves into an environment where they can think clearly and be productive and live well, and maintain that, and not try to learn to survive being manipulated without a clear and present threat that they think they have active reason to move toward rather than away from.
I agree with your last claim. I note that when I read your comment I’m not sure whether you’re saying “this is an important area of improvement” or “this should be central to the art”, which are very different epistemic states.
I’m gonna err on the side of noting disagreements and giving brief descriptions of my perspective rather than writing something I think has a good chance of successfully persuading you of my perspective, primarily so as to actually write a reply in a timely fashion.
Acknowledged.
I don’t see this as showing that in all domains one must maintain high offensive capabilities in order to have good defenses.
Oh, uh, I didn’t mean to imply that. I meant to say that rejecting attention to military power is a bad strategy for defense. A much, much better defensive strategy is to study offense. But that doesn’t need to mean getting good at offense!
(Although I do think it means interacting with offense. Most martial arts fail spectacularly on this point for instance. Pragmatically speaking, you have to have practice actually defending yourself in order to get skillful at defense. And in cases like MMA, that does translate to getting skilled at attack! But that’s incidental. I think you could design good self-defense training systems that have most people never practicing offense.)
I think these problems aren’t that hard once you have community spaces that are willing to enforce boundaries. Over the last few years I’ve run many events and spaces, and often gotten references for people who want to enter the spaces, and definitely chosen to not invite people due to concerns about ethics and responsible behavior. I don’t believe I would’ve accepted these two people into the spaces more than once or twice at most.
Nice. And I agree, boundaries like this can be great for a large range of things.
I don’t think this helps the Art much though.
And it’s hard to know how much your approach doesn’t work.
I also wonder how much this lesson about boundaries arose because of the earlier Dark exploits. In which case it’s actually, ironically, an example of exactly the kind of thing I’m talking about! Only with lessons learned much more painfully than I think was necessary due to their not being sought out.
But also, maybe this is good enough for what you care about. Again, I don’t mean to pressure that you should do anything differently.
I’m mostly pushing back against the implication I read that “Nah, our patches are fine, we’ve got the Dark Arts distanced enough that they’re not an issue.” You literally can’t know that.
My position is that most thinking isn’t really about reality and isn’t truth-tracking, but that if you are doing that thinking then a lot of important questions are surprisingly easy to answer.
Totally agree. And this is a major defense against a lot of the stuff that bamboozles most folk.
I think there’s a ton of adversarial stuff going on as well, but the primary reason that people haven’t noticed that AI is an x-risk isn’t because people are specifically trying to trick them about the domain, but because the people are not really asking themselves the question and checking.
I agree — and I’m not sure why you felt this was relevant to say? I think maybe you thought I was saying something I wasn’t trying to.
(I think there’s some argument to be made here that the primary reason people don’t think for themselves is because civilization is trying to make them go crazy, which is interesting, though I still think the solution is primarily “just make a space where you can actually think about the object level”.)
This might be a crux between us. I’m not sure. But I think you might be seriously underestimating what’s involved in that “just” part (“just make a space…”). Attention on the object-level is key, I 100% agree there. But what defines the space? What protects its boundaries? If culture wants to grab you by the epistemic throat, but you don’t know how it tries to do so, and you just try to “make a space”… you’re going to end up way more confident of the clarity of your thinking than is true.
I acknowledge that there are people who are very manipulative and adversarial in illegible ways that are hard to pin down. […] …I think probably there are good ways to help that info rise up and get shared…. I don’t think it requires you yourself being very skilled at engaging with manipulative people.
I think there’s maybe something of a communication impasse happening here. I agree with what you’re saying here. I think it’s probably good enough for most cases you’re likely to care about, for some reasonable definition of “most”. It also strikes me as obvious that (a) it’s unlikely to cover all the cases you’re likely to care about, and (b) the Art would be deeply enriched by learning how one would skillfully engage with manipulative people. I don’t think everyone who wants to benefit from that enrichment needs to do that engagement, just like not everyone who wants to train in martial arts needs to get good at realistic self-defense.
I’ve said this several times, and you seem to keep objecting to my implied claim of not-that. I’m not sure what’s going on there. Maybe I’m missing your point?
I do sometimes look at people who think they’re at war a lot more than me, and they seem very paranoid and to spend so many cognitive cycles modeling ghosts and attacks that aren’t there. It seems so tiring!
I agree. I think it’s dumb.
I suspect you and I disagree about the extent to which we are at war with people epistemically.
Another potentially relevant point here is that I tend to see large groups and institutions as the primary forces deceiving me and tricking me, and much less so individuals.
Oh! I’m really glad you said this. I didn’t realize we were miscommunicating about this point.
I totally agree. This is what I mean when I’m talking about agents. I’m using adversarial individuals mostly as case studies & training data. The thing I actually care about is the multipolar war going on with already-present unaligned superintelligences. Those are the Dark forces I want to know how to be immune to.
I’m awfully suspicious of someone’s ability to navigate hostile psychofauna if literally their only defense against (say) a frame controller is “Sus, let’s exclude them.” You can’t exclude Google or wokism or collective anxiety the same way.
Having experienced frame control clawing at my face, and feeling myself become immune without having to brace… and noticing how that skill generalized to some of the tactics that the psychofauna use…
…it just seems super obvious to me that this is really core DADA. Non-cognitive, very deep, very key.
Personally I would like to know two or three people who have successfully navigated being manipulated, and hopefully have them write up their accounts of that.
Ditto!
I think aspiring rationalists should maneuver themselves into an environment where they can think clearly and be productive and live well, and maintain that, and not try to learn to survive being manipulated without a clear and present threat that they think they have active reason to move toward rather than away from.
Totally agree with the first part. I think the whole thing is a fine choice. I notice my stance of “Epistemic warriors would still be super useful” is totally unmoved thus far though. (And I’m reminded of your caveat at the very beginning!)
I’m reminded of the John Adams quote: “I must study Politicks and War that my sons may have liberty to study Mathematicks and Philosophy. My sons ought to study mathematics and Philosophy, Geography, natural History, naval Architecture, navigation, Commerce and Agriculature, in order to give their Children a right to study Painting, Poetry, Musick, Architecture, Statuary, Tapestry and Porcelaine.”
I note that when I read your comment I’m not sure whether you’re saying “this is an important area of improvement” or “this should be central to the art”, which are very different epistemic states.
Oh, I don’t know what should or shouldn’t be central to the Art.
It just strikes me that rationality currently is in a similar state as aikido.
Aikido claims to be an effective form of self-defense. (Or at least it used to! Maybe it’s been embarrassed out of saying that anymore?) It’s a fine practice, it has immense value… it’s just not what it says on the tin.
If it wanted to be what it claims, it needs to do things like add pressure testing. Realistic combat. Going into MMA tournaments and coming back with refinements to what it’s doing.
And that could be done in a way that honors its spirit! It can add the constraints that are key to its philosophy, like “Protect everyone involved, including the attacker.”
But maybe it doesn’t care about that. Maybe it just wants to be a sport and discipline.
That’s totally fine!
It does seem weird for it to continue claiming to be effective self-defense though. Like it needs its fake meaning to be something its practitioners believe in.
I think rationality is in a similar state. It has some really good stuff in it. Really good. It’s a great domain.
But I just don’t see it mattering for the power plays. I think rationalists don’t understand power, the same way aikido practitioners don’t understand fighting. And they seem to be in a similar epistemic state about it: they think they basically do, but they don’t pressure-test their understanding to check, best as I can tell.
So of your two options, it’s more like “important area for improvement”… roughly like pressure-testing could be an important area of improvement for aikido. It’d probably become a kind of central if it were integrated! But I don’t know.
And, I think the current state of rationality is fine.
Just weak in one axis it sometimes claims to care about.
COVID was one of the MMA-style arenas for different egregores to see which might come out ‘on top’ in an epistemically unfriendly environment.
I have a lot of opinions on this that are more controversial than I’m willing to go into right now. But I wonder what else will work as one of these “testing arenas.”
The standard rationalist defense I’ve noticed against this amounts to mental cramping. Demand everything go through cognition, and anything that seems to try to route around cognition gets a freakout/shutdown/”shame it into oblivion” kind of response. The stuff that disables this immune response is really epistemically strange — things like prefacing with “Here’s a fake framework, it’s all baloney, don’t believe anything I’m saying.” Or doing a bunch of embodied stuff to act low-status and unsure. A Dark Artist who wanted to deeply mess with this community wouldn’t have to work very hard to do some serious damage before getting detected, best as I can tell (and as community history maybe illustrates).
Can you spell this out a little more? Did Brent and LaSota employ baloney-disclaimers and uncertainty-signaling in order to bypass people’s defenses?
Can you spell this out a little more? Did Brent and LaSota employ baloney-disclaimers and uncertainty-signaling in order to bypass people’s defenses?
I think Brent did something different from what I’m describing — a bit more like judo plus DOS attacks.
I’m not as familiar with LaSota’s methods. I talked with them several times, but mostly before I learned to detect the level of psychological impact I’m talking about with any detail. Thinking back to those interactions, I remember it feeling like LaSota was confidently asserting moral and existential things that threatened to make me feel inadequate and immoral if I didn’t go along with what they were saying and seek out the brain hemisphere hacking stuff they were talking about. And maybe even then I’d turn out to be innately “non-good”.
(Implied here is a type of Dark hack I find most folk don’t have good defenses against other than refusing to reason and blankly shutting down. It works absurdly well on people who believe they should do what they intellectually conclude makes sense to do.)
The thing I was referring to is something I personally stumbled across. IME rationalists on the whole are generally more likely to take in something said in a low-status way. It’s like the usual analyze-and-scrutinize machinery kind of turns off.
One of the weirder examples is, just ending sentences as though they’re questions? I’m guessing it’s because ending each thing with confidence as a statement is a kind of powerful assertion. But, I mean, if the person talking is less confident then maybe what they’re saying is pretty safe to consider?
(I’m demoing back & forth in that paragraph, in case that wasn’t clear.)
I think LaSota might have been doing something like this too, but I’m not sure.
(As a maybe weird example: Notice how that last sentence is in fact caveated, but it’s still confident. I’m quite sure this is my supposition. I’m sure I’m not sure of the implied conclusion. I feel solid in all of this. My impression is, this kind of solidity is a little (sometimes a lot) disturbing to many rationalists (with some exceptions I don’t understand very well — like how Zvi and Eliezer can mostly get away with brazen confidence without much pushback). By my models, the content of the above sentence would have been easier to receive if rewritten along the lines of, “I’m really not sure, but based on my really shaky memories, I kinda wonder if LaSota might have been doing something like this too — but don’t believe me too much!”)
Notice how that last sentence is in fact caveated, but it’s still confident. I’m quite sure this is my supposition. I’m sure I’m not sure of the implied conclusion. I feel solid in all of this.
As an aside, looking over the way some of my comments were downvoted in the discussion section:
I think LW could stand to have a clearer culture around what karma downvotes are for.
Now that downvote is separable from disagreement vote, I read a downvote as “This comment shouldn’t have been posted / doesn’t belong on LW.”
But it’s clear that some of what I said was heavily downvoted because I took a stance people didn’t like. Saying things like “Yep, I could have phrased this post in a more epistemically accurate way… but for this post in particular I really don’t care.”
Would you really rather I didn’t share the fact that I didn’t care?
I’m guessing the intention was to punish me for not caring.
…which is terrible collective rationality, by the way! It’s an attempt to use social-emotional force to change how my mind works without dialoguing with the reasons I’m making the choices I am.
(Which is ironic given the nature of the complaints about this post in particular!)
I’d argue that the right and good function of downvoting is to signal an opinion that a post or comment does not belong here.
That’s how I use it. And until I’m given good reason otherwise, that’s how I plan to continue using it.
I’d also really like to see a return of the old LW cultural thing of, if you downvote then you explain why. There are some downvotes on my comments that I’m left scratching my head about and going “Okay, whatever.” It’s hard for downvotes to improve culture if the feedback amounts to “Bad.”
(But this really is an aside. It doesn’t matter at all for the 2022 review. It’s not really about this particular post either. It just has some very loud-to-me examples of the downvote behavior I think is unhealthy.)
I’d also really like to see a return of the old LW cultural thing of, if you downvote then you explain why. There are some downvotes on my comments that I’m left scratching my head about and going “Okay, whatever.” It’s hard for downvotes to improve culture if the feedback amounts to “Bad.”
For instance, my review has been pretty heavily downvoted. Why? I can think of several reasons. But the net effect is to convey that LW would rather not have seen such a review.
Now why would that be?
I notice that there’s also a −16 on the agree/disagree voting, with just three votes. So I’m guessing that what I said seriously irked a few people who probably heavy-downvoted the karma too.
But if it’s really a distributed will, it’s curious. Do you really want me not to have shared more context? Not to have reflected on where I’m at with the post? Or is it that you want me to feel differently about the post than I do?
I guess I don’t get to know!
It’s worth remembering that karma downvoting has a technical function. Serious negative karma makes a comment invisible by default. A user who gets a lot of negative karma in a short period of time can’t post comments for a while (I think?). A user who has low karma overall can’t post articles (unless that’s changed?).
So a karma downvote amounts to saying “Shut up.”
And a strong-downvote amounts to saying “Shut the fuck up.”
If that’s really the only communication the whole culture encourages for downvotes… that doesn’t really foster clarity.
It seems dead obvious to me that this aspect of conversation culture here is quite bad.
I’d also really like to see a return of the old LW cultural thing of, if you downvote then you explain why.
I wholeheartedly agree with you on this, but unfortunately, the current site culture, moderation policies, etc., actively discourage such explanations.
…the current site culture, moderation policies, etc., actively discourage such explanations.
How so? What’s the discouragement? I could see people feeling like they don’t want to bother, but you make it sound like there’s some kind of punishment for doing so…?
Well, a downvote implies that I didn’t like the post or comment for some reason, right? Maybe I think it’s wrong, or poorly written, or such things shouldn’t be posted to Less Wrong in the first place, etc.—all the usual stuff.
But comments that say such things are discouraged. You’re supposed to post “constructive” things, to not be “negative”, to not be “confrontational”, etc. I, personally, have gotten punishment by the moderation team, for… well, sometimes not even explaining downvotes, exactly, but even just writing comments in lieu of downvotes.
And just think of how your (and my!) preferred cultural norm interacts with the “author can ban commenters from their posts” feature! Suppose that someone writes a post, I downvote it, I try to write a comment that explains my downvote, but oops—I’ve been banned from the post! (Or, the explanatory comment gets me banned from the post. Because the author doesn’t want to experience negativity, you see.)
Indeed, it’s entirely possible to read someone’s post, agree with it, read the comments to that post, see some foolish and poorly-considered criticism of the OP, downvote that comment, try to write an explanation for the downvote—and find out that the OP has banned you from their posts. Oops!
The whole system, both technically and in terms of policy, is set up to shield authors from “negativity”, and allow them to avoid seeing harsh criticism. We know this, because the admins/mods have told us. Well, of course that ends up discouraging explanations of downvotes. How can it possibly not?
It has also been pointed out before that the asymmetry of voting and commenting is most of what enables vote-rings and other invisible manipulation on link aggregator websites. If entities are manipulating a site by leaving comments, then this is almost by definition visible. If entities are manipulating via voting but not commenting, then they are invisible except to possibly administrators with relatively high-powered analysis tools designed for network/graph analysis. For example, one could manipulate a site by registering many accounts and then steering by downvoting one type of comment and upvoting the opposite type. Anyone who sticks their head out with a good comment opposed to the manipulation gets punished (and depending on the site mechanics may in fact eventually be banned or lose voting powers etc), while counter-voters at least don’t suffer.
But it’s clear that some of what I said was heavily downvoted because I took a stance people didn’t like. Saying things like “Yep, I could have phrased this post in a more epistemically accurate way… but for this post in particular I really don’t care.”
Well, that particular comment had a lot of other stuff going on, and yes I think it’s a kind of comment that doesn’t belong here and no I don’t particularly feel like explaining that.
But also, yeah, I do kinda feel like “downvoting people when they admit they did something bad” is a thing we sometimes do here and that’s not great incentives. If someone wants to avoid that kind of downvote, “stop admitting to the bad thing” seems like an obvious strategy. Oops! And like, I remember times when I asked someone a question and they got downvoted for their answer, and I did think it was a bad answer that in a vacuum deserved downvotes, but I still upvoted as thanks for answering.
I’m not sure it’s so bad though. Some things that mitigate it as a strategy:
“This person strategically fails to answer certain questions” is a thing it’s possible for someone to notice and point out.
Someone might not have realized the thing they did was bad-according-to-LW, and the downvotes help signal that. (Maybe better to instead upvote the admission and downvote the thing they did? But that’s not always a thing that can be downvoted, or downvotes might not be specifically targetable to make it clear “this thing you did was bad”.)
If someone did a bad thing and doesn’t care, maybe we just don’t want them here. Downvotes probably marginally push them away, as well as marginally push them towards not-admitting-things. Notably, I feel like we’re more likely to downvote “I did a bad thing and don’t care” than “I did a bad thing, oops, sorry”.
Sometimes someone might take “not being able to say a thing” as a cost, and prefer the downvotes over the silence.
In general it seems like a hard problem, and it’s not clear to me that downvoting this kind of thing is a mistake.
I’d also really like to see a return of the old LW cultural thing of, if you downvote then you explain why. There are some downvotes on my comments that I’m left scratching my head about and going “Okay, whatever.” It’s hard for downvotes to improve culture if the feedback amounts to “Bad.”
I think there’s currently too many things that deserve downvotes for that to be realistic.
That’s really not a central example of what I meant. I meant more like this one. Or this one.
But also, yeah, I do kinda feel like “downvoting people when they admit they did something bad” is a thing we sometimes do here and that’s not great incentives. If someone wants to avoid that kind of downvote, “stop admitting to the bad thing” seems like an obvious strategy. Oops! And like, I remember times when I asked someone a question and they got downvoted for their answer, and I did think it was a bad answer that in a vacuum deserved downvotes, but I still upvoted as thanks for answering.
Yep. This is messy and unfortunate, I agree.
Someone might not have realized the thing they did was bad-according-to-LW, and the downvotes help signal that.
It’s not possible to take the downvotes as a signal of this if downvotes get used for a wide range of things. If the same signal gets used for
“This was written in bad form, but if you’d written it differently it would have been welcome”
and
“Your attitude doesn’t belong on this website, and you should change it or leave”
and
“I don’t like your vibe, so I’m just gonna downvote”
then the feedback isn’t precise enough to be helpful in shaping behavior.
If someone did a bad thing and doesn’t care, maybe we just don’t want them here.
True.
Although if the person disagrees with whether it was bad, and the answer to that disagreement is to try to silence them… then that seems to me like a pretty anti-epistemic norm. At least locally.
I’d also really like to see a return of the old LW cultural thing of, if you downvote then you explain why. There are some downvotes on my comments that I’m left scratching my head about and going “Okay, whatever.” It’s hard for downvotes to improve culture if the feedback amounts to “Bad.”
I think there’s currently too many things that deserve downvotes for that to be realistic.
I have a hard time believing this claim. It’s not what I see when I look around.
The dynamic would be pretty simple:
After I downvote, I skim the replies to see if someone else already explained what had me do the downvote. If so, I upvote that explanation and agree-vote it too.
If there’s no such explanation, I write one.
Easy peasy. I seriously doubt the number of things needing downvotes on this site is so utterly overwhelming that this approach is untenable. The feedback would be very rich, the culture well-defined and transparent.
I don’t know why LW stopped doing this. Once upon a time it used to cost karma to downvote, so people took downvotes more seriously. I assume there was some careful thought put into changing that system to the current one. I haven’t put more than a sum total of maybe ten minutes of thinking into this. So I’m probably missing something.
But without knowing what that something is, and without a lot of reason for me to invest a ton more time into figuring it out… my tentative but clear impression is that what I’m describing would be way better for culture here by a long shot.
It’s not possible to take the downvotes as a signal of this if downvotes get used for a wide range of things.
Perhaps not in general, but I think it’s often pretty clear. Like you’ve already said “I’m guessing the intention was to punish me for not caring”, and yes, I think you’re right. Seems to me the signal was recieved as intended.
Although if the person disagrees with whether it was bad, and the answer to that disagreement is to try to silence them… then that seems to me like a pretty anti-epistemic norm. At least locally.
Well, if someone comes here arguing for flat-earthism, I’m probably going to downvote without bothering to read their arguments. Is that anti-epistemic? Maybe, I guess? Certainly yes, if it turns out that the earth is flat (and that their arguments are correct). And “this practice isn’t anti-epistemic as long as we only dismiss false ideas” is, um. Nevertheless, I endorse that practice.
If someone comes around here calling people names, and we downvote that rather than checking in “hey are you doing this because you think name calling is good actually? Would you like to dialogue about that?” is that anti-epistemic? Again, maybe yes? But I endorse it anyway.
The dynamic would be pretty simple:
After I downvote, I skim the replies to see if someone else already explained what had me do the downvote. If so, I upvote that explanation and agree-vote it too.
If there’s no such explanation, I write one.
Easy peasy.
I do not consider writing these explanations to be easy.
I seriously doubt the number of things needing downvotes on this site is so utterly overwhelming that this approach is untenable.
I can think of a few places we might disagree here: how many things deserve downvotes, how costly it is to explain them, how realistic it is for people to pay those costs. I’m not super enthusiastic about trying to drill down into this, though.
But I also think I’m less optimistic than you about the benefits of doing it. I can think of multiple conversations I’ve had where I wanted people to change what they’re doing, I explained why I thought they were doing something bad, and they just keep on doing it. You yourself seem to understand what it is that many people dislike in many of your posts and comments, and yet you keep doing the thing. Surely there are cases where it does help, but I think they’re a minority. (It seems plausible to me that the helpful cases actually do get explained more often than others. E.g. if someone explicitly asks why they’re getting downvoted, that’s evidence they’re interested in improving, and also it makes them more likely to get an explanation.)
Another thing worth mentioning is that reacts reduce the cost of explaining downvotes. I dunno how much they’re used, since I mostly use GreaterWrong which doesn’t (yet?) support them. I believe they were only added to this post later, so they wouldn’t have been helpful at the time. But yeah, if a comment gets downvoted a bunch with not even any reacts explaining why, that seems not ideal.
After I downvote, I skim the replies to see if someone else already explained what had me do the downvote. If so, I upvote that explanation and agree-vote it too.
If there’s no such explanation, I write one.
Easy peasy. I seriously doubt the number of things needing downvotes on this site is so utterly overwhelming that this approach is untenable. The feedback would be very rich, the culture well-defined and transparent.
I don’t know why LW stopped doing this. Once upon a time it used to cost karma to downvote, so people took downvotes more seriously. I assume there was some careful thought put into changing that system to the current one. I haven’t put more than a sum total of maybe ten minutes of thinking into this. So I’m probably missing something.
But without knowing what that something is, and without a lot of reason for me to invest a ton more time into figuring it out… my tentative but clear impression is that what I’m describing would be way better for culture here by a long shot.
I agree with you that what you propose would be better for LW’s culture. However, I think I can answer the “why did LW stop doing this” question:
An increased prevalence, in those social circles which influence decisions made by the LW admin team, of people who have a strong aversion to open conflict.
You write a post or a comment. Someone writes a reply explaining why they downvoted—in other words, a critical reply. This is open conflict—confrontation.
You reply to them to dispute their criticism, to question their characterization, to argue—more open conflict. Encouraging downvote explanations is nothing more nor less than encouraging critical comments, after all! More critical comments—more open conflict.
Some people can’t stand open conflict. So, they use their influence to cause to be enacted such policies, and to be built such structures, as will prevent confrontation, explicit disagreement, direct criticism. (This is usually couched in euphemisms, of course, as calling such things by their simple names also invites confrontation.)
It’s kind of funny to me to see this one nominated. It’s sort of peak “Val is weird on LW”.
The point of this post wasn’t to offer claims for people to examine. I still agree with the claims I see myself having tried to make! But the point wasn’t to offer ideas for discussion. It was to light a path out of Hell.
Because of that purpose, the style of this post really doesn’t fit LW culture. I think it’s fair to call it a mind spell. I get the impression that LWers in particular find mind spells unnerving: they’re a symmetric tech that can do an end-run around the parts of cognition that rationalists heavily rely on to feel safe. Hence tripping the “cult”/”guru” immune reaction.
(To me it’s dead obvious that this highlights a gap in the LW rationality toolbox. The reaction of “Lock down, distrust, get cynical, burn it with fire” actually makes you more susceptible to skillful bad actors — like going rigid in response to a judo master grabbing a hold of you. IMO, a mature Art of Rationality would necessarily include learning to navigate cognition-jamming (or cognition-incompatible!) spaces with grace. But I get the sense LW collectively doesn’t want to build that skillset. Which is fine, but I find it a bit disappointing.)
I picked up some of the language & framing of this post from Perri Chase. I now talk about this stuff a little differently. And more kindly, I think. I suspect I could write a version of this spell today that would be less of a problem for the LW memetic immune system. Partly because I’m better at slipping through immune systems! (I’m sure that’s comforting!) But mostly because I’ve learned how to work with such systems instead of needing to step around them to have the “real” conversation.
That said, I don’t regret writing this post. I got a lot of feedback (including in quite a few PMs across many different media) from people who found this relieving, validating, soothing, deeply helpful, kind, orienting. I’m okay with some people being upset with me if that’s the price for enacting this kindness. I went in expecting that price, really.
I think there’s a post possible that would be something like a LW-compatible rewrite of this one. It’d remove the “spell” nature and try to lay out some claims & implications for folk to consider. A bit like dissecting a once-living specimen and laying out its organs for examination.
I probably won’t write that post. I don’t see it doing hardly any good beyond being kind of interesting.
I might write a related post sometime on the nature of Hell as a psychosocial attractor state. AFAICT it’s utterly essential study for real Defense Against the Dark Arts. It’s also very tricky to talk about in a way that’s kind to the listeners or the speaker. But if LW were to learn to take it seriously without falling into it harder, I think that awareness would transform a lot of what “rationality” means here, and it would soften a lot of the sharp edges that can meaningfully hurt people here.
I don’t plan on rewriting any of this post for the review. The spell worked great. I want to leave it here as is.
(Though if someone understands the spellcraft and wants to suggest some edits, I’m open to receiving those suggestions! I’m not putting up a wall here. I’m just sharing where I’m at with this post right now, for the sake of the 2022 review.)
Mm, this sounds to me like saying “a master rationalist could surround themself with con artists and frauds and other epistemically adversarial actors who were gaming the rationalist, and still have perfectly true beliefs”, and that may be true, but I think another pretty good option is “a master rationalist would definitely avoid surrounding themselves with con artists and frauds and other adversarial actors”.
I do think there are real skills you are pointing to, but to some extent I prefer the world where I don’t have those skills and in place of that my allies and I coordinate to identify and exclude people who are using the dark arts.
(I don’t say this as the ‘last word’ on the subject and expect you would produce a substantive and interesting counterargument if you chose to engage on this, I nonetheless thought I’d share that I currently disagree with what I perceive this paragraph to be saying.)
I think that’s a great option. I’d question a “master rationalist’s” skills if they couldn’t avoid such adversarial actors, or notice them if they slip through the cracks.
I like your preference. I’ll say some things, but I want to start by emphasizing that I don’t think you’re making a wrong or bad choice.
I want to talk about what I think the Art could be, kind of for aesthetic reasons. This isn’t to assert anything about what you or any given individual should or shouldn’t be doing in any kind of moral sense.
So with that said, here are three points:
(1) I think there’s a strong analogy here to studying combat and war. Yes, if you can be in a pacifist cluster and just exclude folk who are really into applied competitive strategy, then you have something kind of like a cooperate/cooperate equilibrium. But if that’s the whole basis of your culture, it’s extremely vulnerable, the way cooperate-bot is vulnerable in prisoners’ dilemmas. You need military strength, the way a walled garden needs walls. Otherwise folk who have military strength can just come take your resources, even if you try to exclude them at first.
At the risk of using maybe an unfair example, I think what happened with FTX last year maybe illustrates the point.
Clearer examples in my mind are Ziz and Brent. The point not being “These people are bad!” But rather, these people were psychologically extremely potent and lots of folk in the community could neither (a) adequately navigate their impact (myself included!) nor (b) rally ejection/exclusion power until well after they’d already had their impact.
Maybe, you might hope, you can make the ejection/exclusion sensitivity refined enough to work earlier. But if you don’t do that by studying the Dark Arts, and becoming intimately familiar with them, then what you get is a kind of naïve allergic response that Dark Artists can weaponize.
Again, I don’t mean that you in particular or even rationalists in general need to address this. There’s nothing wrong with a hobby. I’m saying that as an Art, it seems like rationality is seriously vulnerable if it doesn’t include masterful familiarity with the Dark Arts. Kind of like, there’s nothing wrong with practicing aikido as a sport, but you’re not gonna get the results you hope for if you train in aikido for self-defense. That art is inadequate for that purpose and needs exposure to realistic combat to matter that way.
(2) …and I think that if the Art of Rationality were to include intimate familiarity with the Dark Arts, it would work way way better.
Things like the planning fallacy or confirmation bias are valuable to track. I could stand to improve my repertoire here for sure.
But the most potent forms of distorted thinking aren’t about sorting out the logic. I think they look more like reaching deep down and finding ways to become immune to things like frame control.
Frame control is an amazing example in my mind precisely because of the hydra-like nature of the beast. How do you defend against frame control without breaking basic things about culture and communication and trust? How do you make it so your cultural and individual defenses don’t themselves become the manual that frame controllers use to get their desired effects?
And this barely begins to touch on the kind of impact that I’d want to call “spiritual”. By which I don’t mean anything supernatural; I’m talking about the deep psychological stuff that (say) conversing with someone deep in a psilocybin trip can do to the tripper. That’s not just frame control. That’s something way deeper, like editing someone’s basic personality operating system code. And sometimes it reaches deeper even than that. And it turns out, you don’t need psychedelics to reach that deep; those chemical tools just open a door that you can open other ways, voluntarily or otherwise, sometimes just by having a conversation.
The standard rationalist defense I’ve noticed against this amounts to mental cramping. Demand everything go through cognition, and anything that seems to try to route around cognition gets a freakout/shutdown/”shame it into oblivion” kind of response. The stuff that disables this immune response is really epistemically strange — things like prefacing with “Here’s a fake framework, it’s all baloney, don’t believe anything I’m saying.” Or doing a bunch of embodied stuff to act low-status and unsure. A Dark Artist who wanted to deeply mess with this community wouldn’t have to work very hard to do some serious damage before getting detected, best as I can tell (and as community history maybe illustrates).
If this community wanted to develop the Art to actually be skillful in these areas… well, it’s hard to predict exactly what that’d create, but I’m pretty sure it’d be glorious. If I think of the Sequences as retooling skeptical materialism, I think we’d maybe see something like a retooling of the best of Buddhist psychotechnology. I think folk here might tend to underestimate how potent that could really be.
(…and I also think that it’s maybe utterly critical for sorting out AI alignment. But while I think that’s a very important point, it’s not needed for my main message for this exchange.)
(3) It also seems relevant to me that “Dark Arts” is maybe something of a fake category. I’m not sure it even forms a coherent cluster.
Like, is being charismatic a Dark Art? It certainly can be! It can act as a temptation. It seems to be possible to cultivate charisma. But the issue isn’t that charisma is a Dark Art. It’s that charisma is mostly symmetric. So if someone has a few slightly anti-epistemic social strategies in them, and they’re charismatic, this can have a net Dark effect that’s even strategic. But this is a totally normal level of epistemic noise!
Or how about something simpler, like someone using confirmation bias in a way that benefits their beliefs? Astrology is mostly this. Is astrology a Dark Art? Is talking about astrology a Dark Art? It seems mostly just epistemically hazardous… but where’s the line between that and Dark Arts?
How about more innocent things, like when someone is trying to understand systemic racism? Is confirmation bias a helpful pattern-recognizer, or a Dark Art? Maybe it’s potentially in service to Dark Arts, but is a necessary risk to learn the patterns?
I think Vervaeke makes this point really well. The very things that allow us to notice relevance are precisely the things that allow us to be fooled. Rationality (and he explicitly cites this — even the Keith Stanovich stuff) is a literally incomputable practice of navigating both Type I and Type II errors in this balancing act between relevance realization and being fooled.
When I think of central examples of Dark Arts, I think mostly of agents who exploit this ambiguity in order to extract value from others.
…which brings me back to point (1), about this being more a matter of skill in war. The relevant issue isn’t that there are “Dark Arts”. It’s that there are unaligned agents who are trying to strategically fool you. The skill isn’t to detect a Dark toolset; it’s to detect intelligent intent to deceive and extract value.
All of which is to say:
I think a mature Art of Rationality would most definitely include something like skillful navigation of manipulation.
I don’t think every practitioner needs to master every aspect of a mature Art. Much like not all cooks need to know how to make a roux.
But an Art that has detection, exclusion, & avoidance as its only defense against Dark Artists is a much poorer & more vulnerable Art. IMO.
Thanks for the comment. I’m gonna err on the side of noting disagreements and giving brief descriptions of my perspective rather than writing something I think has a good chance of successfully persuading you of my perspective, primarily so as to actually write a reply in a timely fashion.
I don’t want to create an expectation that if you reply then you will reply to each point; rather I’d encourage you if you reply to simply reply to whichever points seem interesting or cruxy to you.
———
1) You make the analogy to having non-violent states. I concur that presently one cannot have states without militaries. I don’t see this as showing that in all domains one must maintain high offensive capabilities in order to have good defenses. I agree one needs defenses, but sometimes good defenses don’t look like “Training thousands of people how to carry out a targeted kill-strike” and instead look like “Not being tempted to reply by rude comments online” or “Checking whether a factual claim someone makes is accurate”.
You say that for LaSota and Brent that folks “could neither (a) adequately navigate their impact (myself included!) nor (b) rally ejection/exclusion power until well after they’d already had their impact” and “Maybe, you might hope, you can make the ejection/exclusion sensitivity refined enough to work earlier”.
I don’t share the sense of difficulty I read in the second of those quotes. I think the Bay Area rationalists (and most other rationalists globally) had some generally extreme lack of boundaries of any sort. The ~only legible boundaries that the Bay Area rationality scene had were (a) are you an employee at one of CFAR/MIRI, and (b) are you invited to CFAR events. MIRI didn’t have much to do with these two individuals, and I think CFAR was choosing a strategy of “we’re not really doing social policing, we’re primarily just selecting on people who have interesting ideas about rationality”. Everything else was highly social and friend-based and it was quite dramatic to ban people from your social events. The REACH was the only community space and if I recall correctly explicitly had no boundaries on who could be there. This is an environment where people with lots of red flags will be able to move around with much more ease than in the rest of the world.
I think these problems aren’t that hard once you have community spaces that are willing to enforce boundaries. Over the last few years I’ve run many events and spaces, and often gotten references for people who want to enter the spaces, and definitely chosen to not invite people due to concerns about ethics and responsible behavior. I don’t believe I would’ve accepted these two people into the spaces more than once or twice at most. It’s unpleasant work to enforce boundaries and I’ve made mistakes, but overall I think that there were just not many strong boundaries in these people’s way initially, and they would have been pushed back and dissuaded much earlier if there were.
2) You write:
My position is that most thinking isn’t really about reality and isn’t truth-tracking, but that if you are doing that thinking then a lot of important questions are surprisingly easy to answer. Generally doing a few fermi estimates with a few datapoints can get you pretty in touch with the relevant part of reality.
I think there’s a ton of adversarial stuff going on as well, but the primary reason that people haven’t noticed that AI is an x-risk isn’t because people are specifically trying to trick them about the domain, but because the people are not really asking themselves the question and checking.
(And also something about people not having any conception of what actions to take in the fact of a civilizational-scale problem that most of the rest of civilization is not thinking about.)
(I think there’s some argument to be made here that the primary reason people don’t think for themselves is because civilization is trying to make them go crazy, which is interesting, though I still think the solution is primarily “just make a space where you can actually think about the object level”.)
I acknowledge that there are people who are very manipulative and adversarial in illegible ways that are hard to pin down. There’s a whole discussion about offense/defense here and how it plays out. I currently expect that there are simple solutions here. As a pointer, someone I know and respect along with their partner, makes lists of people they know for whom they would not be surprised to later find out that the person did something quite manipulative/bad/unethical, and I think they’ve had some success with this. Also personally I have repeatedly kicked myself thinking “I knew that person was suspicious, why didn’t I say so earlier?” I don’t think these problems are particularly intractable and I do think people know things and I think probably there are good ways to help that info rise up and get shared (I do not claim to have solved this problem). I don’t think it requires you yourself being very skilled at engaging with manipulative people.
Yeah I’ve seen this, and done it somewhat. I think it works in some situations, but there’s a bunch of adversarial situations it definitely doesn’t work. I do agree it seems like a false hope to think that this can be remotely sufficient.
3) I do sometimes look at people who think they’re at war a lot more than me, and they seem very paranoid and to spend so many cognitive cycles modeling ghosts and attacks that aren’t there. It seems so tiring! I suspect you and I disagree about the extent to which we are at war with people epistemically.
Another potentially relevant point here is that I tend to see large groups and institutions as the primary forces deceiving me and tricking me, and much less so individuals. I’m much more scared of Twitter winding its way into my OODA loop than I am of a selfish scheming frame controlling individual. I think it’s much easier for me to keep boundaries against individuals than I am against these much bigger and broader forces.
4)
My perspective on these.
Personally I would like to know two or three people who have successfully navigated being manipulated, and hopefully have them write up their accounts of that.
I think aspiring rationalists should maneuver themselves into an environment where they can think clearly and be productive and live well, and maintain that, and not try to learn to survive being manipulated without a clear and present threat that they think they have active reason to move toward rather than away from.
I agree with your last claim. I note that when I read your comment I’m not sure whether you’re saying “this is an important area of improvement” or “this should be central to the art”, which are very different epistemic states.
Acknowledged.
Oh, uh, I didn’t mean to imply that. I meant to say that rejecting attention to military power is a bad strategy for defense. A much, much better defensive strategy is to study offense. But that doesn’t need to mean getting good at offense!
(Although I do think it means interacting with offense. Most martial arts fail spectacularly on this point for instance. Pragmatically speaking, you have to have practice actually defending yourself in order to get skillful at defense. And in cases like MMA, that does translate to getting skilled at attack! But that’s incidental. I think you could design good self-defense training systems that have most people never practicing offense.)
Nice. And I agree, boundaries like this can be great for a large range of things.
I don’t think this helps the Art much though.
And it’s hard to know how much your approach doesn’t work.
I also wonder how much this lesson about boundaries arose because of the earlier Dark exploits. In which case it’s actually, ironically, an example of exactly the kind of thing I’m talking about! Only with lessons learned much more painfully than I think was necessary due to their not being sought out.
But also, maybe this is good enough for what you care about. Again, I don’t mean to pressure that you should do anything differently.
I’m mostly pushing back against the implication I read that “Nah, our patches are fine, we’ve got the Dark Arts distanced enough that they’re not an issue.” You literally can’t know that.
Totally agree. And this is a major defense against a lot of the stuff that bamboozles most folk.
I agree — and I’m not sure why you felt this was relevant to say? I think maybe you thought I was saying something I wasn’t trying to.
This might be a crux between us. I’m not sure. But I think you might be seriously underestimating what’s involved in that “just” part (“just make a space…”). Attention on the object-level is key, I 100% agree there. But what defines the space? What protects its boundaries? If culture wants to grab you by the epistemic throat, but you don’t know how it tries to do so, and you just try to “make a space”… you’re going to end up way more confident of the clarity of your thinking than is true.
I think there’s maybe something of a communication impasse happening here. I agree with what you’re saying here. I think it’s probably good enough for most cases you’re likely to care about, for some reasonable definition of “most”. It also strikes me as obvious that (a) it’s unlikely to cover all the cases you’re likely to care about, and (b) the Art would be deeply enriched by learning how one would skillfully engage with manipulative people. I don’t think everyone who wants to benefit from that enrichment needs to do that engagement, just like not everyone who wants to train in martial arts needs to get good at realistic self-defense.
I’ve said this several times, and you seem to keep objecting to my implied claim of not-that. I’m not sure what’s going on there. Maybe I’m missing your point?
I agree. I think it’s dumb.
Oh! I’m really glad you said this. I didn’t realize we were miscommunicating about this point.
I totally agree. This is what I mean when I’m talking about agents. I’m using adversarial individuals mostly as case studies & training data. The thing I actually care about is the multipolar war going on with already-present unaligned superintelligences. Those are the Dark forces I want to know how to be immune to.
I’m awfully suspicious of someone’s ability to navigate hostile psychofauna if literally their only defense against (say) a frame controller is “Sus, let’s exclude them.” You can’t exclude Google or wokism or collective anxiety the same way.
Having experienced frame control clawing at my face, and feeling myself become immune without having to brace… and noticing how that skill generalized to some of the tactics that the psychofauna use…
…it just seems super obvious to me that this is really core DADA. Non-cognitive, very deep, very key.
Ditto!
Totally agree with the first part. I think the whole thing is a fine choice. I notice my stance of “Epistemic warriors would still be super useful” is totally unmoved thus far though. (And I’m reminded of your caveat at the very beginning!)
I’m reminded of the John Adams quote: “I must study Politicks and War that my sons may have liberty to study Mathematicks and Philosophy. My sons ought to study mathematics and Philosophy, Geography, natural History, naval Architecture, navigation, Commerce and Agriculature, in order to give their Children a right to study Painting, Poetry, Musick, Architecture, Statuary, Tapestry and Porcelaine.”
Oh, I don’t know what should or shouldn’t be central to the Art.
It just strikes me that rationality currently is in a similar state as aikido.
Aikido claims to be an effective form of self-defense. (Or at least it used to! Maybe it’s been embarrassed out of saying that anymore?) It’s a fine practice, it has immense value… it’s just not what it says on the tin.
If it wanted to be what it claims, it needs to do things like add pressure testing. Realistic combat. Going into MMA tournaments and coming back with refinements to what it’s doing.
And that could be done in a way that honors its spirit! It can add the constraints that are key to its philosophy, like “Protect everyone involved, including the attacker.”
But maybe it doesn’t care about that. Maybe it just wants to be a sport and discipline.
That’s totally fine!
It does seem weird for it to continue claiming to be effective self-defense though. Like it needs its fake meaning to be something its practitioners believe in.
I think rationality is in a similar state. It has some really good stuff in it. Really good. It’s a great domain.
But I just don’t see it mattering for the power plays. I think rationalists don’t understand power, the same way aikido practitioners don’t understand fighting. And they seem to be in a similar epistemic state about it: they think they basically do, but they don’t pressure-test their understanding to check, best as I can tell.
So of your two options, it’s more like “important area for improvement”… roughly like pressure-testing could be an important area of improvement for aikido. It’d probably become a kind of central if it were integrated! But I don’t know.
And, I think the current state of rationality is fine.
Just weak in one axis it sometimes claims to care about.
Musings:
COVID was one of the MMA-style arenas for different egregores to see which might come out ‘on top’ in an epistemically unfriendly environment.
I have a lot of opinions on this that are more controversial than I’m willing to go into right now. But I wonder what else will work as one of these “testing arenas.”
Can you spell this out a little more? Did Brent and LaSota employ baloney-disclaimers and uncertainty-signaling in order to bypass people’s defenses?
I think Brent did something different from what I’m describing — a bit more like judo plus DOS attacks.
I’m not as familiar with LaSota’s methods. I talked with them several times, but mostly before I learned to detect the level of psychological impact I’m talking about with any detail. Thinking back to those interactions, I remember it feeling like LaSota was confidently asserting moral and existential things that threatened to make me feel inadequate and immoral if I didn’t go along with what they were saying and seek out the brain hemisphere hacking stuff they were talking about. And maybe even then I’d turn out to be innately “non-good”.
(Implied here is a type of Dark hack I find most folk don’t have good defenses against other than refusing to reason and blankly shutting down. It works absurdly well on people who believe they should do what they intellectually conclude makes sense to do.)
The thing I was referring to is something I personally stumbled across. IME rationalists on the whole are generally more likely to take in something said in a low-status way. It’s like the usual analyze-and-scrutinize machinery kind of turns off.
One of the weirder examples is, just ending sentences as though they’re questions? I’m guessing it’s because ending each thing with confidence as a statement is a kind of powerful assertion. But, I mean, if the person talking is less confident then maybe what they’re saying is pretty safe to consider?
(I’m demoing back & forth in that paragraph, in case that wasn’t clear.)
I think LaSota might have been doing something like this too, but I’m not sure.
(As a maybe weird example: Notice how that last sentence is in fact caveated, but it’s still confident. I’m quite sure this is my supposition. I’m sure I’m not sure of the implied conclusion. I feel solid in all of this. My impression is, this kind of solidity is a little (sometimes a lot) disturbing to many rationalists (with some exceptions I don’t understand very well — like how Zvi and Eliezer can mostly get away with brazen confidence without much pushback). By my models, the content of the above sentence would have been easier to receive if rewritten along the lines of, “I’m really not sure, but based on my really shaky memories, I kinda wonder if LaSota might have been doing something like this too — but don’t believe me too much!”)
Does that answer what you’d hoped?
Perhaps relevant: Nate Soares does this too, based on one of his old essays. And I think it works very well for him.
As an aside, looking over the way some of my comments were downvoted in the discussion section:
I think LW could stand to have a clearer culture around what karma downvotes are for.
Now that downvote is separable from disagreement vote, I read a downvote as “This comment shouldn’t have been posted / doesn’t belong on LW.”
But it’s clear that some of what I said was heavily downvoted because I took a stance people didn’t like. Saying things like “Yep, I could have phrased this post in a more epistemically accurate way… but for this post in particular I really don’t care.”
Would you really rather I didn’t share the fact that I didn’t care?
I’m guessing the intention was to punish me for not caring.
…which is terrible collective rationality, by the way! It’s an attempt to use social-emotional force to change how my mind works without dialoguing with the reasons I’m making the choices I am.
(Which is ironic given the nature of the complaints about this post in particular!)
I’d argue that the right and good function of downvoting is to signal an opinion that a post or comment does not belong here.
That’s how I use it. And until I’m given good reason otherwise, that’s how I plan to continue using it.
I’d also really like to see a return of the old LW cultural thing of, if you downvote then you explain why. There are some downvotes on my comments that I’m left scratching my head about and going “Okay, whatever.” It’s hard for downvotes to improve culture if the feedback amounts to “Bad.”
(But this really is an aside. It doesn’t matter at all for the 2022 review. It’s not really about this particular post either. It just has some very loud-to-me examples of the downvote behavior I think is unhealthy.)
For instance, my review has been pretty heavily downvoted. Why? I can think of several reasons. But the net effect is to convey that LW would rather not have seen such a review.
Now why would that be?
I notice that there’s also a −16 on the agree/disagree voting, with just three votes. So I’m guessing that what I said seriously irked a few people who probably heavy-downvoted the karma too.
But if it’s really a distributed will, it’s curious. Do you really want me not to have shared more context? Not to have reflected on where I’m at with the post? Or is it that you want me to feel differently about the post than I do?
I guess I don’t get to know!
It’s worth remembering that karma downvoting has a technical function. Serious negative karma makes a comment invisible by default. A user who gets a lot of negative karma in a short period of time can’t post comments for a while (I think?). A user who has low karma overall can’t post articles (unless that’s changed?).
So a karma downvote amounts to saying “Shut up.”
And a strong-downvote amounts to saying “Shut the fuck up.”
If that’s really the only communication the whole culture encourages for downvotes… that doesn’t really foster clarity.
It seems dead obvious to me that this aspect of conversation culture here is quite bad.
But this isn’t a hill I intend to die on.
I wholeheartedly agree with you on this, but unfortunately, the current site culture, moderation policies, etc., actively discourage such explanations.
How so? What’s the discouragement? I could see people feeling like they don’t want to bother, but you make it sound like there’s some kind of punishment for doing so…?
Well, a downvote implies that I didn’t like the post or comment for some reason, right? Maybe I think it’s wrong, or poorly written, or such things shouldn’t be posted to Less Wrong in the first place, etc.—all the usual stuff.
But comments that say such things are discouraged. You’re supposed to post “constructive” things, to not be “negative”, to not be “confrontational”, etc. I, personally, have gotten punishment by the moderation team, for… well, sometimes not even explaining downvotes, exactly, but even just writing comments in lieu of downvotes.
And just think of how your (and my!) preferred cultural norm interacts with the “author can ban commenters from their posts” feature! Suppose that someone writes a post, I downvote it, I try to write a comment that explains my downvote, but oops—I’ve been banned from the post! (Or, the explanatory comment gets me banned from the post. Because the author doesn’t want to experience negativity, you see.)
Indeed, it’s entirely possible to read someone’s post, agree with it, read the comments to that post, see some foolish and poorly-considered criticism of the OP, downvote that comment, try to write an explanation for the downvote—and find out that the OP has banned you from their posts. Oops!
The whole system, both technically and in terms of policy, is set up to shield authors from “negativity”, and allow them to avoid seeing harsh criticism. We know this, because the admins/mods have told us. Well, of course that ends up discouraging explanations of downvotes. How can it possibly not?
It has also been pointed out before that the asymmetry of voting and commenting is most of what enables vote-rings and other invisible manipulation on link aggregator websites. If entities are manipulating a site by leaving comments, then this is almost by definition visible. If entities are manipulating via voting but not commenting, then they are invisible except to possibly administrators with relatively high-powered analysis tools designed for network/graph analysis. For example, one could manipulate a site by registering many accounts and then steering by downvoting one type of comment and upvoting the opposite type. Anyone who sticks their head out with a good comment opposed to the manipulation gets punished (and depending on the site mechanics may in fact eventually be banned or lose voting powers etc), while counter-voters at least don’t suffer.
Well, that particular comment had a lot of other stuff going on, and yes I think it’s a kind of comment that doesn’t belong here and no I don’t particularly feel like explaining that.
But also, yeah, I do kinda feel like “downvoting people when they admit they did something bad” is a thing we sometimes do here and that’s not great incentives. If someone wants to avoid that kind of downvote, “stop admitting to the bad thing” seems like an obvious strategy. Oops! And like, I remember times when I asked someone a question and they got downvoted for their answer, and I did think it was a bad answer that in a vacuum deserved downvotes, but I still upvoted as thanks for answering.
I’m not sure it’s so bad though. Some things that mitigate it as a strategy:
“This person strategically fails to answer certain questions” is a thing it’s possible for someone to notice and point out.
Someone might not have realized the thing they did was bad-according-to-LW, and the downvotes help signal that. (Maybe better to instead upvote the admission and downvote the thing they did? But that’s not always a thing that can be downvoted, or downvotes might not be specifically targetable to make it clear “this thing you did was bad”.)
If someone did a bad thing and doesn’t care, maybe we just don’t want them here. Downvotes probably marginally push them away, as well as marginally push them towards not-admitting-things. Notably, I feel like we’re more likely to downvote “I did a bad thing and don’t care” than “I did a bad thing, oops, sorry”.
Sometimes someone might take “not being able to say a thing” as a cost, and prefer the downvotes over the silence.
In general it seems like a hard problem, and it’s not clear to me that downvoting this kind of thing is a mistake.
I think there’s currently too many things that deserve downvotes for that to be realistic.
That’s really not a central example of what I meant. I meant more like this one. Or this one.
Yep. This is messy and unfortunate, I agree.
It’s not possible to take the downvotes as a signal of this if downvotes get used for a wide range of things. If the same signal gets used for
“This was written in bad form, but if you’d written it differently it would have been welcome”
and
“Your attitude doesn’t belong on this website, and you should change it or leave”
and
“I don’t like your vibe, so I’m just gonna downvote”
then the feedback isn’t precise enough to be helpful in shaping behavior.
True.
Although if the person disagrees with whether it was bad, and the answer to that disagreement is to try to silence them… then that seems to me like a pretty anti-epistemic norm. At least locally.
I have a hard time believing this claim. It’s not what I see when I look around.
The dynamic would be pretty simple:
After I downvote, I skim the replies to see if someone else already explained what had me do the downvote. If so, I upvote that explanation and agree-vote it too.
If there’s no such explanation, I write one.
Easy peasy. I seriously doubt the number of things needing downvotes on this site is so utterly overwhelming that this approach is untenable. The feedback would be very rich, the culture well-defined and transparent.
I don’t know why LW stopped doing this. Once upon a time it used to cost karma to downvote, so people took downvotes more seriously. I assume there was some careful thought put into changing that system to the current one. I haven’t put more than a sum total of maybe ten minutes of thinking into this. So I’m probably missing something.
But without knowing what that something is, and without a lot of reason for me to invest a ton more time into figuring it out… my tentative but clear impression is that what I’m describing would be way better for culture here by a long shot.
Perhaps not in general, but I think it’s often pretty clear. Like you’ve already said “I’m guessing the intention was to punish me for not caring”, and yes, I think you’re right. Seems to me the signal was recieved as intended.
Well, if someone comes here arguing for flat-earthism, I’m probably going to downvote without bothering to read their arguments. Is that anti-epistemic? Maybe, I guess? Certainly yes, if it turns out that the earth is flat (and that their arguments are correct). And “this practice isn’t anti-epistemic as long as we only dismiss false ideas” is, um. Nevertheless, I endorse that practice.
If someone comes around here calling people names, and we downvote that rather than checking in “hey are you doing this because you think name calling is good actually? Would you like to dialogue about that?” is that anti-epistemic? Again, maybe yes? But I endorse it anyway.
I do not consider writing these explanations to be easy.
I can think of a few places we might disagree here: how many things deserve downvotes, how costly it is to explain them, how realistic it is for people to pay those costs. I’m not super enthusiastic about trying to drill down into this, though.
But I also think I’m less optimistic than you about the benefits of doing it. I can think of multiple conversations I’ve had where I wanted people to change what they’re doing, I explained why I thought they were doing something bad, and they just keep on doing it. You yourself seem to understand what it is that many people dislike in many of your posts and comments, and yet you keep doing the thing. Surely there are cases where it does help, but I think they’re a minority. (It seems plausible to me that the helpful cases actually do get explained more often than others. E.g. if someone explicitly asks why they’re getting downvoted, that’s evidence they’re interested in improving, and also it makes them more likely to get an explanation.)
Another thing worth mentioning is that reacts reduce the cost of explaining downvotes. I dunno how much they’re used, since I mostly use GreaterWrong which doesn’t (yet?) support them. I believe they were only added to this post later, so they wouldn’t have been helpful at the time. But yeah, if a comment gets downvoted a bunch with not even any reacts explaining why, that seems not ideal.
I agree with you that what you propose would be better for LW’s culture. However, I think I can answer the “why did LW stop doing this” question:
An increased prevalence, in those social circles which influence decisions made by the LW admin team, of people who have a strong aversion to open conflict.
You write a post or a comment. Someone writes a reply explaining why they downvoted—in other words, a critical reply. This is open conflict—confrontation.
You reply to them to dispute their criticism, to question their characterization, to argue—more open conflict. Encouraging downvote explanations is nothing more nor less than encouraging critical comments, after all! More critical comments—more open conflict.
Some people can’t stand open conflict. So, they use their influence to cause to be enacted such policies, and to be built such structures, as will prevent confrontation, explicit disagreement, direct criticism. (This is usually couched in euphemisms, of course, as calling such things by their simple names also invites confrontation.)
Hence, the Less Wrong of today.