Epistemic Status: 13 years working as a therapist for a wide variety of populations, 5 of them working with rationalists and EA clients. 7 years teaching and directing at over 20 rationality camps and workshops. This is an extremely short and colloquially written form of points that could be expanded on to fill a book, and there is plenty of nuance to practically everything here, but I am extremely confident of the core points in this frame, and have used it to help many people break out of or avoid manipulative practices.

TL;DR: Your wants and preferences are not invalidated by smarter or more “rational” people’s preferences. What feels good or bad to someone is not a monocausal result of how smart or stupid they are.

Alternative titles to this post are “Two people are enough to form a cult” and “Red flags if dating rationalists,” but this stuff extends beyond romance and far beyond LW-Rationalism.

I saw forms of it as a college student among various intellectual subcultures. I saw forms of it growing up around non-intellectuals who still ascribed clear positives and negatives to the words “smart” and “stupid.” I saw forms of it as a therapist working with people from a variety of nationalities. And of course, my various roles in the rationalist and EA communities have exposed me to a number of people who have been subject to some form of it from friends, romantic partners, or family… hell, most of the time I’ve heard it coming from someone’s parents.

What I’m here to argue against is, put simply, the notion that what feels good or bad to someone is a monocausal result of how smart or stupid they are. There are a lot of false beliefs downstream of that notion, but the main one I’m focusing on here is the idea that your wants or preferences might be invalid because someone “more rational” than you said so.

Because while I’ve taught extensively about how to defend against “dark arts” emotional manipulation in a variety of flavors, I especially dislike seeing “rationality” used as an authoritative word to shame others into self-coercive narratives.

Rationality, as I use the word, refers to an epistemology that minimizes false beliefs and a course of action that maximally fulfills one’s goals.

If someone else tells you that something you’re doing or thinking is irrational, they need to first demonstrate that they understand your goals, and second demonstrate that they have information you don’t, which may inform predictions of why your actions will fail to achieve those goals.

If they can’t do that, criticizing your feelings or preferences is not the same thing as offering reasonable critique of your beliefs and actions. Feelings and preferences are not assertions that can be wrong; they’re experiences.

And I think no one should feel ashamed of their own experiential qualia, but I especially don’t want people to ignore their preferences because they’re worried about not being “rational” enough.

1) You are not too stupid to know what you want.

Ash and Bryce have just had dinner together, and on the way home Ash spots an ice cream shop.

“Oo, I want to stop in for some ice cream!”

“Why?”

“Because it’ll be tasty!”

“But it’s just a superstimulus of sugar and dairy that you evolved to find enjoyable. It might make make you gain weight, becoming less attractive, lowering your expected income, and shortening your lifespan. Also that money can be better spent on malaria nets.”

“Sure, but I still want it.”

“That’s stupid.”

I hope that most people would recognize that there’s something wrong in the above conversation. Maybe Ash is perfectly happy for Bryce to talk at length about the downsides of having ice cream when they express a preference for it, or maybe they understand that Bryce has good intentions and won’t take it to heart. Either way, most people would flag “that’s stupid,” or worse, “you’re stupid,” as bad communication coming from a friend or partner.

But what about “That’s irrational?”

Dun dun duuuuun...

Cue defensiveness, self-doubt, internal conflict. We don’t want to be irrational, right? That’s like, our whole thing!

Now, there are some people out there who might use this opportunity to say “Exactly, you can’t be rational literally all the time! Everyone needs to be irrational sometimes, especially in pursuit of happiness!”

To them I say, “You want people to enjoy their ice cream even if it’s ‘irrational.’ I want people to know that preferences are never ‘irrational.’ We are not the same.”^[1]

If you think the ice cream example is too easy, what about career choice? What about being monogamous vs open vs poly? What about how you receive feedback?

The key is recognizing that the preference itself is completely independent from rationality or intelligence.

If you want your partner to do the dishes sometimes, even if they make 5x more than you and their time is “worth more?”

Nothing to do with rationality.

If you wish your partner would do the dishes sometimes so you don’t always do them, or as a signal that they value your time too?

Still nothing to do with rationality!

If you refuse any other solution to the dish situation and insist that they must do the dishes sometimes or else they don’t love you...

Now we have stepped away from pure “wants” or preferences. Now the examination of what’s rational makes sense. Your feelings can be understandable and valid, while the thoughts that arise from them can sometimes be false or unjustified… which is to say, (epistemologically) irrational.

And your assertions or ultimatums as a result can sometimes be self-defeating to your broader values or preferences… which is what I mean when I say someone is being (instrumentally) irrational.

But the fundamental preferences you have are not about rationality. Inconsistent actions can be irrational if they’re self-defeating, but “inconsistent preferences” only makes sense if you presume you’re a monolithic entity, or believe your “parts” need to all be in full agreement all the time… which I think very badly misunderstands how human brains work.

If you try different solutions and listen to arguments and ultimately decide you do not want to be with someone who does not help clean the dishes sometimes, no matter how good the relationship is otherwise, many people may find that unusual, but “irrational” presumes that you have the same values and wants as others, and why should that presumption be any more true for this than it is for ice cream flavors?

Wants are not beliefs. They are things you feel.

And no one is too stupid or irrational to know what they feel. Too disembodied or disassociated, yes! Many such cases. But knowing what you feel is not a test of epistemology!^[2]

Pause and consider whether you have ever felt irrational for wanting or not wanting something. Where did that belief come from? Why do you believe it?

Because what I’ve seen, over and over, is that it comes from the expectation, from yourself or others, that if you want something, all your beliefs about it must be true, or all the plans that arise to get it must be followed.

And that’s obviously nonsense, on both counts.

Wants are emergent, complex forms of pain and pleasure. They are either felt or they are not felt, and reason only comes in at the stage of deciding what to do about them.

So stop judging your wants, and stop listening to other people who judge them either.

2) Feeling hurt is not a sign of irrationality.

“I don’t see what you’re so upset about,” a hypothetical Bryce may say. “I’m just being honest. I don’t want you to get fat, because I care about you, and eating lots of ice cream may increase the odds that you end up at a higher weight set-point. You want me to be honest, right? I wouldn’t be a true friend if I didn’t point out when I thought you were making an error.”

I think honesty is a virtue. I genuinely believe that people who do their best to be as truthful as possible, not just in direct words but also to avoid misleading others, are demonstrating a sort of deep Goodness. More than that, I think dishonesty consequentially leads to worse worlds in almost every circumstance.

This virtue is not extended to people who repeatedly volunteer “brutal honesty” to their friends or partners while knowing it is likely to be painful.

My steelman of the speaker who says “you shouldn’t feel hurt by this” probably means something like “it would be better for you, suffering-wise or consequentially, to not get hurt about this, and perceive it as truth untouched by stuck priors.”

But that’s not how humans work.

It is instrumentally valuable to be careful how and when you offer criticism. It is not just a sign of caring about others, but also understanding the role of emotion in our cognition. “You should not be hurt by that” or “you should not find that offensive” are words said by people who want humans to work a certain way more than they actually care to understand how human psychology actually functions.

Our feelings are lossy heuristics and blunt motivations for things we evolved or have been conditioned to care about. They are not things to which the word “should” makes sense to attach.

Some people genuinely don’t care if they are physically unattractive. Some people genuinely don’t care if they are perceived as smart or competent. Some people genuinely don’t care if they are “likeable.”

But people who do not care about any of those things are extremely rare, because each is a dimension of evolutionary fitness; it is not a sign of strength or “rationality” to not care about them, it is a sign of either extreme abundance, or being extremely neuroatypical.

(And if someone is trying to explicitly hold their neuroatypical traits up as a blanket flex on neurotypical norms, I have a different essay in mind for that sort of compartmentalized blindspot).

It isn’t impossible to sometimes “reason” your way into feelings of security and abundance, but that’s because our feelings of insecurity and scarcity are, again, lossy heuristics. They are not always accurate, and there are some ways to explicitly think your way through and around the catastrophization and fear that makes certain words hurtful.

But they nearly all require slow, careful exploration. I have never seen them dissolve from brute force, and if someone wields “truth” like a mallet to try and force another person’s behavior or feelings into the shape better suited to their own preferences or ideals, this is fundamentally a hostile act, whether they intend it as such or not. It’s the same as shouting in someone’s face after they tell you to stop because you think what you’re saying is just too important for them to ignore.

To be clear, if someone invites truthful claims or evaluations or feedback, there is nothing wrong with their partner or friend being honest, even if it might be painful to hear. If the listener gets upset after inviting honesty, that is not the speaker’s fault.

But I have seen too many people perpetuating emotionally abusive relationships justify their behavior with “I’m just being honest” to let this pass without clear and unpressured signs that the person hearing constant harsh truths actually wants to hear them that way.

Even worse, if someone takes another person’s pain as evidence that they must have said something true or valuable for them to hear, they are again fundamentally misunderstanding how emotions work.

If someone accuses me of thinking or feeling something hostile that I didn’t think or feel, it might bother me. It might even hurt, if it’s someone I consider a friend. But that hurt would come from feeling unseen or uncared for, completely independent of how accurate their perception was (which by default they certainly do not have better evidence of than I do).

And of course, I should emphasize that no one is perfect. People make mistakes. Grace and forgiveness are important for people to improve, and build robust and healthy relationships. I don’t mean to bash anyone who finds learning inhibition difficult, and I think “filtering” is genuinely not equally easy for everyone.

But if you express to a friend or partner or parent or child or even a stranger a clear preference against this sort of communication, and make it clear that you find it hurtful and not helpful, I think effort on their part to try and learn better ways to communicate is an integral signal of caring about you and the relationship.

If they decide they’d rather not, that it’s too stifling… that is their right. Maybe the relationship just isn’t meant to be, and by all means, people should self-select into the friend groups and romantic partnerships that work for them.

But if they try to justify it as a matter of virtue, or “rationality?” If you ask them to stop and they start quoting the Litany of Gendlin at you and insisting that if you were smarter you’d be thankful that they’re willing to honestly tell you how unattractive or stupid you are?

That’s not rationality.

I’m gatekeeping the term away from that sort of high-school negging bullshit, and if they continue to insist it is, I will suspect they are gaining negative-sum value from this form of communication which they do not want to lose, and advise anyone I care about to avoid close association.

3) Illegible preferences are not invalid.

“Well at least explain why you want the ice cream,” an increasingly frustrated Bryce may say. “You have to have a reason for it, right?”

“You just want me to give a reason?”

“Yeah, it doesn’t make sense to me.”

“The reason is it tastes good and will make me happy.”

“Those don’t seem like actual reasons to have ice cream specifically. If I find you something tasty but healthier, you’d have that instead, right?”

“Maybe? But I actually just want the ice cream right now.”

“Okay, but let’s look at this logically...”

There’s a great discussion to be had around the ways knowledge can be legible or illegible, explicit or implicit, as well as situations where putting too much weight on legibility can lead to ignoring or discounting implicit knowledge. By default, most things we know or think we know are hard to make legible to others. For the speaker, language is a lossy medium through which to compress experiences and concepts, and for the listener, attention and memory have many imperfections that can cause further corruption of transferred knowledge/meaning.

I’ve got the skeleton of a book written called When to Think and When to Feel, and my hope is that it will help people understand what feelings are “for,” what logical thought is “for,” and why it’s a bad idea to ignore either or confuse one for the other. The header above is how I would sum up about ¹⁄₅ of it.

Your preferences do not need to be legible to be valid, and you should never feel like you need to justify or defend them to others.

Models and predictions, certainly! Actions even more so, if they affect other people!

But again, things you want are just extensions of positive and negative valence qualia. “I like vanilla ice cream” and “I dislike chocolate ice cream” are not a matter of rationality or irrationality, and neither are any other preferences you have.

Accept them for what they are, and ignore people who want you not to.

“But what about—”

Yes, yes. Of course it is often helpful, for you or others, if your preferences are legible! Of course your preferences may be contradictory at times! Of course your preferences may not, in fact, all be things you endorse acting on at all times!

But they are not individually invalid just because you have other preferences, or reasons against acting on them, and they are certainly not invalid because someone else doesn’t understand them. Legibility is for planning and coordination. It is not for justification.

If someone says “let’s be logical about this” or tries to otherwise use “reason” to dissuade, or worse shame, a preference you have, they are not likely to be actually using reason, logic, or “rationality” to point to true things.

And if they do it specifically in a context where they are trying to change your preference to match their own, they may just be using authoritative words to manipulate you.

Please don’t get mad at “logic” or “legibility,” get mad at people attempting to gaslight you out of your preferences! It’s not a different monster than guilt-tripping, it’s the same monster wearing a “reason” skinsuit.

“Because I don’t want to” is always, always, always a sufficient reason to not do something or not accept something others do that affects you.

This is not a get-out-of-jail-free card! Others may respond accordingly! Your wants do not by default override theirs! Maybe people will judge you for not being able to make your reasoning more legible to them, or feel like they can’t then cooperate with you in some circumstances. Social or legal consequences exist independent of you being true to yourself.

And, of course, you may have other preferences that could make you feel compelled to do things you predict would be unpleasant. But in every case, the first step that makes the most difference is accepting (and being willing to defend) what you want and don’t want.

From there you become much more resistant to manipulations of all sorts, and can actually figure out what actions balance all your preferences, both short-term and long-term.

4) Your preferences do not need to fully match your community’s.

Every culture and subculture has norms and expectations, things that get socially reinforced or punished, often through status. This is not mostly not a conscious action, but the simple flow of how much people like each other; communities having norms is important, as it’s the fundamental thing that makes their subculture more enjoyable for the sort of people who formed and gravitated to it in the first place.

But every norm can also act as a mechanism to pressure others into conformity, and if you let your sense of what’s right or wrong be determined by a group, not only are you inviting blindspots, you’re at risk of being manipulated, even by people with good intentions.

Sometimes people will come into a subculture or community and say “Hey, this place is great and all, but it would be much better if it did X instead of Y.” Then maybe things get discussed and changed, or maybe most people in the community say “No thanks, we made this subculture in part to not have that sort of thing in it. We’d rather keep it as it is.” If there’s enough fundamental preference mismatches there, then it might make sense to part ways with the community as a whole.^[3]

These sorts of dialogue and outcomes are perfectly fine. But one of the surest signs of a High Control Group is discouraging communication with outsiders, whether by punishment or more subtly undermining the judgment of those outside the subculture. A lot of cultures have a norm of “don’t air dirty laundry from the family to outsiders,” and this helps hide a lot of abuse around the world.

“Don’t ask your family/friends about this, they’re not ‘one of us,’ they wouldn’t understand… here, let me call our friend/priest/village elder, they can act as a neutral party!”

Is it always a red flag? Not necessarily; it is often true that subcultures have norms that work well for some people in them but not for everyone, and unfortunately larger cultures will often pathologize those differences by default. So an “inside” third-person perspective can be useful!

But both people should have a say in who gets asked, and if either side starts to insist that only their picks are smart or rational or wise enough to be trusted to have a good take, that’s pretty sus. Additionally, it doesn’t make any sense to do it for disagreeing preferences, as opposed to specific strategies for fulfilling preferences! Any time someone tries, substitute with “He’ll tell us if I’m right for preferring chocolate ice cream over vanilla” and see if that helps.

If instead they try to insist that you just don’t understand all the information needed to agree with them, guess what? Convincing you is their job as the supposedly more rational or better informed person. And that happens not by making you feel worse about yourself, nor by dressing up a browbeating with rationalist shibboleths, but by understanding your model of the world, identifying the different beliefs or values, finding cruxes, and so on.

If they can’t do that, why on earth should you give up on your preferences? In what bizarro world would that sort of acquiescence to someone else’s self-claimed authority be “rational?”

By all means, work to find solutions or compromises where you can, but do not let anyone else tell you that your wants are any more irrational than theirs. Not an entire community of people, nor an entire world of them. If your preferences are inherently self-destructive or self-defeating, the territory should demonstrate that, and there should be compelling examples to point to.

Sometimes preferences are too far out of reach from each other for a solution or compromise to be reached in reasonable timeframes. That sucks, but it happens. If so, the appropriate response may be to grieve, alone or together, and move on. Maybe just changing the nature of the relationship will be enough, or adjusting how much of each person’s life is spent with one another.

And sometimes communities do in fact have explicit “preferences” that will cost people status just by having different ones. It might even be costly to find out what those diffuse preferences are, and especially daunting for people new to a community. But if you ultimately do discover that your preferences don’t fully line up with those, hopefully you can find others in the community who differ in the same way.

Meanwhile, if someone insists you’re just “not being rational enough” to concede to some preference or community norm? Separate entirely from all of the above, please be sure to check how representative they are in the community… and don’t just trust people they selected to tell you.

Final Thoughts

As I said, this topic contains a lot of nuance that I don’t have time to get into here. The frame above is one that treats values and preferences and desires as entirely separate from what’s “rational” in the spirit of the orthogonality thesis, distinguishing goals from capability-to-achieve-goals. There are other frames we could use, like one that highlights the ways our preferences, being the result of natural selection and behavioral conditioning, are thus are fully rational expressions of our genetic predispositions given the complexity and superstimuli of the modern world.

My main intent here is to push back against narratives that try to shove “rationality” into particular value sets, rather than keeping it a pure expression of epistemic rigor and effective action evaluation. Believing that preferences indicate intellectual inadequacy is a subtle, and common, extension of that.

Different brains find different things enjoyable, and I wouldn’t be surprised if there is some correlation between intelligence and what sorts of activities a person finds enjoyable for how long before they get bored or frustrated… but there is no value-gain in judging yourself or anyone else by generalities, or putting up with others who do.

Even if your literal job is to record preference correlations to form some Bayesian prior by which to evaluate applicants for a job or something, thinking that someone is less smart or less rational because of a preference is like judging someone’s cooking skills by whether they enjoy a fast food burger.

^
If “I’m allowed to be irrational sometimes” is what helps you live a better, less stressful life, feel free to ignore all this and go about your day. My point is that, by my definition of rational, there are many, many circumstances where it is completely rational to decide to eat ice cream (or whatever) rather than donate to malaria nets (or whatever).
^
I don’t mean to oversimplify this, and could talk for hours on the topic. Wants are often influenced by beliefs! Feelings are often hard to put into legible words! Some people have trouble feeling anything they want as physical sensations, but even in such cases it’s important to note how they’re ultimately the result of genetics and conditioning first and foremost. Beliefs and predictions that influence wants may be false or miscalibrated, but the feeling itself, the want itself, just is what it is, the same way sensations of hunger or heat just are what they are.
^
Though the implications of what it means to “leave” a community should itself be unpacked. It implies that being part of a culture is all-or-nothing, which is more how cults operate than communities. Part of why it’s “healthier” to be part of multiple communities is that it creates a robustness against single-culture pressures. While some people might find only one community that deeply matches their ideals and preferences, it’s still extremely unlikely for one community to match every preference someone has. Instead, a healthy community should include multiple slightly different sub-cultures, so that people can shift from one to another if they feel their preferences aren’t sufficiently matched.

You are not too “irrational” to know your preferences.

1) You are not too stupid to know what you want.

2) Feeling hurt is not a sign of irrationality.

3) Illegible preferences are not invalid.

4) Your preferences do not need to fully match your community’s.

Final Thoughts