You are not too “irrational” to know your preferences.
Epistemic Status: 13 years working as a therapist for a wide variety of populations, 5 of them working with rationalists and EA clients. 7 years teaching and directing at over 20 rationality camps and workshops. This is an extremely short and colloquially written form of points that could be expanded on to fill a book, and there is plenty of nuance to practically everything here, but I am extremely confident of the core points in this frame, and have used it to help many people break out of or avoid manipulative practices.
TL;DR: Your wants and preferences are not invalidated by smarter or more “rational” people’s preferences. What feels good or bad to someone is not a monocausal result of how smart or stupid they are.
Alternative titles to this post are “Two people are enough to form a cult” and “Red flags if dating rationalists,” but this stuff extends beyond romance and far beyond LW-Rationalism.
I saw forms of it as a college student among various intellectual subcultures. I saw forms of it growing up around non-intellectuals who still ascribed clear positives and negatives to the words “smart” and “stupid.” I saw forms of it as a therapist working with people from a variety of nationalities. And of course, my various roles in the rationalist and EA communities have exposed me to a number of people who have been subject to some form of it from friends, romantic partners, or family… hell, most of the time I’ve heard it coming from someone’s parents.
What I’m here to argue against is, put simply, the notion that what feels good or bad to someone is a monocausal result of how smart or stupid they are. There are a lot of false beliefs downstream of that notion, but the main one I’m focusing on here is the idea that your wants or preferences might be invalid because someone “more rational” than you said so.
Because while I’ve taught extensively about how to defend against “dark arts” emotional manipulation in a variety of flavors, I especially dislike seeing “rationality” used as an authoritative word to shame others into self-coercive narratives.
Rationality, as I use the word, refers to an epistemology that minimizes false beliefs and a course of action that maximally fulfills one’s goals.
If someone else tells you that something you’re doing or thinking is irrational, they need to first demonstrate that they understand your goals, and second demonstrate that they have information you don’t, which may inform predictions of why your actions will fail to achieve those goals.
If they can’t do that, criticizing your feelings or preferences is not the same thing as offering reasonable critique of your beliefs and actions. Feelings and preferences are not assertions that can be wrong; they’re experiences.
And I think no one should feel ashamed of their own experiential qualia, but I especially don’t want people to ignore their preferences because they’re worried about not being “rational” enough.
1) You are not too stupid to know what you want.
Ash and Bryce have just had dinner together, and on the way home Ash spots an ice cream shop.
“Oo, I want to stop in for some ice cream!”
“Why?”
“Because it’ll be tasty!”
“But it’s just a superstimulus of sugar and dairy that you evolved to find enjoyable. It might make make you gain weight, becoming less attractive, lowering your expected income, and shortening your lifespan. Also that money can be better spent on malaria nets.”
“Sure, but I still want it.”
“That’s stupid.”
I hope that most people would recognize that there’s something wrong in the above conversation. Maybe Ash is perfectly happy for Bryce to talk at length about the downsides of having ice cream when they express a preference for it, or maybe they understand that Bryce has good intentions and won’t take it to heart. Either way, most people would flag “that’s stupid,” or worse, “you’re stupid,” as bad communication coming from a friend or partner or family member.
But what about “That’s irrational?”
Dun dun duuuuun...
Cue defensiveness, self-doubt, internal conflict. We don’t want to be irrational, right? That’s like, our whole thing!
Now, there are some people out there who might use this opportunity to say “Exactly, you can’t be rational literally all the time! Everyone needs to be irrational sometimes, especially in pursuit of happiness!”
To them I say, “You want people to enjoy their ice cream even if it’s ‘irrational.’ I want people to know that preferences are never ‘irrational.’ We are not the same.”[1]
If you think the ice cream example is too easy, what about career choice? What about being monogamous vs open vs poly? What about how you receive feedback?
The key is recognizing that the preference itself is completely independent from rationality or intelligence.
If you want a partner who helps with housework, even if they make 5x more than you and their time is “worth more?”
Nothing to do with rationality.
If you wish your partner would do the dishes sometimes so you don’t always do them, or as a signal that they value your time too?
Still nothing to do with rationality!
If you refuse any other solution to the dish situation and insist that they must do the dishes sometimes or else they don’t love you...
Now we have stepped away from pure “wants” or preferences. Now the examination of what’s rational makes sense. Your feelings can be understandable and valid, while the thoughts that arise from them can sometimes be false or unjustified… which is to say, (epistemologically) irrational.
And your assertions or ultimatums as a result can sometimes be self-defeating to your broader values or preferences… which is what I mean when I say someone is being (instrumentally) irrational.
But the fundamental preferences you have are not about rationality. Inconsistent actions can be irrational if they’re self-defeating, but “inconsistent preferences” only makes sense if you presume you’re a monolithic entity, or believe your “parts” need to all be in full agreement all the time… which I think very badly misunderstands how human brains work.
If you try different solutions and listen to arguments and ultimately decide you do not want to be with someone who does not help clean the dishes sometimes, no matter how good the relationship is otherwise, many people may find that unusual, but “irrational” presumes that you have the same values and wants as others, and why should that presumption be any more true for this than it is for ice cream flavors?
Wants are not beliefs. They are things you feel.
And no one is too stupid or irrational to know what they feel. Too disembodied or disassociated, yes! Many such cases. But knowing what you feel is not a test of epistemology.[2]
Pause and consider whether you have ever believed you were being stupid or irrational for wanting or not wanting something. Where did that belief come from? Why do you believe it?
Because what I’ve seen, over and over, is that it comes from the expectation, from yourself or others, that if you want something, all your beliefs about it must be true, or all the plans that arise to get it must be followed.
And that’s obviously nonsense, on both counts.
Wants are emergent, complex forms of predicted pain and pleasure. They are either felt or they are not felt, and reason only comes in at the stage of deciding what to do about them.
So stop judging your wants, and stop listening to other people who judge them either.
2) Feeling hurt is not a sign of irrationality.
“I don’t see what you’re so upset about,” a hypothetical Bryce may say. “I’m just being honest. I don’t want you to get fat, because I care about you, and eating lots of ice cream may increase the odds that you end up at a higher weight set-point. You want me to be honest, right? I wouldn’t be a true friend if I didn’t point out when I thought you were making an error.”
I think honesty is a virtue. I genuinely believe that people who do their best to be as truthful as possible, not just in direct words but also to avoid misleading others, are demonstrating a sort of deep Goodness. More than that, I think dishonesty consequentially leads to worse worlds in almost every circumstance.
This virtue is not extended to people who repeatedly volunteer “brutal honesty” to their friends or partners or children while knowing it is likely to be painful.
My steelman of the speaker who says “you shouldn’t feel hurt by this” probably means something like “it would be better for you, suffering-wise or consequentially, to not get hurt about this, and perceive it as truth untouched by stuck priors.”
But that’s not how humans work.
It is instrumentally valuable to be careful how and when you offer criticism. It is not just a sign of caring about others, but also understanding the role of emotion in our cognition. “You should not be hurt by that” or “you should not find that offensive” are words said by people who want humans to work a certain way more than they actually care to understand how human psychology actually functions.
Our feelings are lossy heuristics and blunt motivations for things we evolved or have been conditioned to care about.[3] They are not things to which the word “should” makes sense to attach.
Some people genuinely don’t care if they are physically unattractive. Some people genuinely don’t care if they are perceived as smart or competent. Some people genuinely don’t care if they are “likeable.”
But people who do not care about any of those things are extremely rare, because each is a dimension of evolutionary fitness. It is not a sign of strength or “rationality” to not care about them, it is a sign of either extreme abundance or being extremely neurodivergent.
(And if someone is trying to explicitly hold their neurodivergent traits up as a blanket flex on neurotypical norms, I have a different essay in mind for that sort of compartmentalized blindspot).
It isn’t impossible to sometimes “reason” your way into feelings of security and abundance, but that’s because our feelings of insecurity and scarcity are, again, lossy heuristics. They are not always accurate, and there are some ways to explicitly think your way through and around the catastrophization and fear that makes certain words hurtful.
But they nearly all require slow, careful exploration. I have never seen them dissolve from brute force, and if someone wields “truth” like a mallet to try and force another person’s behavior or feelings into the shape better suited to their own preferences or ideals, this is fundamentally a hostile act, whether they intend it as such or not. It’s the same as shouting in someone’s face after they tell you to stop because you think what you’re saying is just too important for them to ignore.
To be clear, if someone invites truthful claims or evaluations or feedback, there is nothing wrong with being totally honest in response, even if it might be painful to hear. If the listener gets upset after inviting honesty, that is not the speaker’s fault.
But I have seen too many people perpetuating emotionally abusive relationships justify their behavior with “I’m just being honest” to let this pass without clear and unpressured signs that the person hearing constant harsh truths actually wants to hear them that way.
Even worse, if someone takes another person’s pain as evidence that they must have said something true or valuable for them to hear, they are again fundamentally misunderstanding how emotions work.
If someone accuses me of thinking or feeling something hostile that I didn’t think or feel, it might bother me. It might even hurt, if it’s someone I consider a friend. But that hurt would come from feeling unseen or uncared for, completely independent of how accurate their perception was (which by default they certainly do not have better evidence of than I do).
And of course, I should emphasize that no one is perfect. People make mistakes. Grace and forgiveness are important for people to improve, and build robust and healthy relationships. I don’t mean to bash anyone who finds learning inhibition difficult, and I think “filtering” is genuinely not equally easy for everyone.
But if you express to a friend or partner or parent or child or even a stranger a clear preference against this sort of communication, and make it clear that you find it hurtful and not helpful, I think effort on their part to try and learn better ways to communicate is an integral signal of caring about you and the relationship.
If they decide they’d rather not, that it’s too stifling… that is their right. Maybe the relationship just isn’t meant to be, and by all means, people should self-select into the friend groups and romantic partnerships that work for them.
But if they try to justify it as a matter of virtue, or “rationality?” If you ask them to stop and they start quoting the Litany of Gendlin at you and insisting that if you were smarter you’d be thankful that they’re willing to honestly tell you how unattractive or stupid you are?
That’s not rationality.
I’m gatekeeping the term away from that sort of high-school negging bullshit, and if they continue to insist it is, I will suspect they are gaining negative-sum value from this form of communication which they do not want to lose.
3) Illegible preferences are not invalid.
“Well at least explain why you want the ice cream,” an increasingly frustrated Bryce may say. “You have to have a reason for it, right?”
“You just want me to give a reason?”
“Yeah, it doesn’t make sense to me.”
“The reason is it tastes good and will make me happy.”
“Those don’t seem like actual reasons to have ice cream specifically. If I find you something tasty but healthier, you’d have that instead, right?”
“Maybe? But I actually just want the ice cream right now.”
“Okay, but let’s look at this logically...”
There’s a great discussion to be had around the ways knowledge can be legible or illegible, explicit or implicit, as well as situations where putting too much weight on legibility can lead to ignoring or discounting implicit knowledge. By default, most things we know or think we know are hard to make legible to others. For the speaker, language is a lossy medium through which to compress experiences and concepts, and for the listener, attention and memory have many imperfections that can cause further corruption of transferred knowledge/meaning.
I’ve got the skeleton of a book written called When to Think and When to Feel, and my hope is that it will help people understand what feelings are “for,” what logical thought is “for,” and why it’s a bad idea to ignore either or confuse one for the other. The header above is how I would sum up about 1⁄5 of it.
Your preferences do not need to be legible to be valid, and you should never feel like you need to justify or defend them to others.
Models and predictions, certainly! Actions even more so, if they affect other people!
But again, things you want are just extensions of positive and negative valence qualia. “I like vanilla ice cream” and “I dislike chocolate ice cream” are not a matter of rationality or irrationality, and neither are any other preferences you have.
Accept them for what they are, and ignore people who want you not to.
“But what about—”
Yes, yes. Of course it is often helpful, for you or others, if your preferences are legible! Of course your preferences may be contradictory at times! Of course your preferences may not, in fact, all be things you endorse acting on at all times!
But they are not individually invalid just because you have other preferences, or reasons against acting on them, and they are certainly not invalid because someone else doesn’t understand them. Legibility is for planning and coordination. It is not for justification.
If someone says “let’s be logical about this” or tries to otherwise use “reason” to dissuade, or worse shame, a preference you have, they are not likely to be actually using reason, logic, or “rationality” to point to true things.
And if they do it specifically in a context where they are trying to change your preference to match their own, they may just be using authoritative words to manipulate you.
Please don’t get mad at “logic” or “legibility,” get mad at people attempting to gaslight you out of your preferences! It’s not a different monster than guilt-tripping, it’s the same monster wearing a “reason” skinsuit.
“Because I don’t want to” is always, always, always a sufficient reason to not do something or not accept something others do that affects you.
This is not a get-out-of-jail-free card! Others may respond accordingly! Your wants do not by default override theirs! Maybe people will judge you for not being able to make your reasoning more legible to them, or feel like they can’t then cooperate with you in some circumstances. Social or legal consequences exist independent of you being true to yourself.
And, of course, you may have other preferences that could make you feel compelled to do things you predict would be unpleasant. But in every case, the first step that makes the most difference is accepting (and being willing to defend) what you want and don’t want.
From there you become much more resistant to manipulations of all sorts, and can actually figure out what actions balance all your preferences, both short-term and long-term.
4) Your preferences do not need to fully match your community’s.
Every culture and subculture has norms and expectations, things that get socially reinforced or punished, often through status. This is mostly not a conscious action, but the simple flow of how much people like each other; communities having norms is important, as it’s the fundamental thing that makes their subculture more enjoyable for the sort of people who formed and gravitated to it in the first place.
But every norm can also act as a mechanism to pressure others into conformity, and if you let your sense of what’s right or wrong be determined by a group, not only are you inviting blindspots, you’re at risk of being manipulated, even by people with good intentions.
Sometimes people will come into a subculture or community and say “Hey, this place is great and all, but it would be much better if it did X instead of Y.” Then maybe things get discussed and changed, or maybe most people in the community say “No thanks, we made this subculture in part to not have that sort of thing in it. We’d rather keep it as it is.” If there’s enough fundamental preference mismatches there, then it might make sense to part ways with the community as a whole.[4]
These sorts of dialogue and outcomes are perfectly fine. But one of the surest signs of a High Control Group is discouraging communication with outsiders, whether by punishment or more subtly undermining the judgment of those outside the subculture. A lot of cultures have a norm of “don’t air dirty laundry from the family to outsiders,” and this helps hide a lot of abuse around the world.
“Don’t ask your family/friends about this, they’re not ‘one of us,’ they wouldn’t understand… here, let me call our friend/priest/village elder, they can act as a neutral party!”
Is it always a red flag? Not necessarily; it is often true that subcultures have norms that work well for some people in them but not for everyone, and unfortunately larger cultures will often pathologize those differences by default. So an “inside” third-person perspective can be useful!
But both people should have a say in who gets asked, and if either side starts to insist that only their picks are smart or rational or wise enough to be trusted to have a good take, that’s pretty sus. Additionally, it doesn’t make any sense to do it for disagreeing preferences, as opposed to specific strategies for fulfilling preferences! Any time someone tries, substitute with “He’ll tell us if I’m right for preferring chocolate ice cream over vanilla” and see if that helps.
If instead they try to insist that you just don’t understand all the information needed to agree with them, guess what? Convincing you is their job as the supposedly more rational or better informed person. And that happens not by making you feel worse about yourself, nor by dressing up a browbeating with rationalist shibboleths, but by understanding your model of the world, identifying the different beliefs or values, finding cruxes, and so on.
If they can’t do that, why on earth should you give up on your preferences? In what bizarro world would that sort of acquiescence to someone else’s self-claimed authority be “rational?”
By all means, work to find solutions or compromises where you can, but do not let anyone else tell you that your wants are any more irrational than theirs. Not an entire community of people, nor an entire world of them. If your preferences are inherently self-destructive or self-defeating, the territory should demonstrate that, and there should be compelling examples to point to.
Sometimes preferences are too far out of reach from each other for a solution or compromise to be reached in reasonable timeframes. That sucks, but it happens. If so, the appropriate response may be to grieve, alone or together, and move on. Maybe just changing the nature of the relationship will be enough, or adjusting how much of each person’s life is spent with one another.
And sometimes communities do in fact have explicit “preferences” that will cost people status just by having different ones. It might even be costly to find out what those diffuse preferences are, and especially daunting for people new to a community. But if you ultimately do discover that your preferences don’t fully line up with those, hopefully you can find others in the community who differ in the same way.
Meanwhile, if someone insists you’re just “not being rational enough” to concede to some preference or community norm? Separate entirely from all of the above, please be sure to check how representative they are in the community… and don’t just trust people they selected to tell you.
Final Thoughts
As I said, this topic contains a lot of nuance that I don’t have time to get into here. The frame above is one that treats values and preferences and desires as entirely separate from what’s “rational” in the spirit of the orthogonality thesis, distinguishing goals from capability-to-achieve-goals. There are other frames we could use, like one that highlights the ways our preferences, being the result of natural selection and behavioral conditioning, are thus are fully rational expressions of our genetic predispositions given the complexity and superstimuli of the modern world.
My main intent here is to push back against narratives that try to shove “rationality” into particular value sets, rather than keeping it a pure expression of epistemic rigor and effective action evaluation. Believing that preferences indicate intellectual inadequacy is a subtle, and common, extension of that.
Different brains find different things enjoyable, and I wouldn’t be surprised if there is some correlation between intelligence and what sorts of activities a person finds enjoyable for how long before they get bored or frustrated… but there is no value-gain in judging yourself or anyone else by generalities, or putting up with others who do.
Even if your literal job is to record preference correlations to form some Bayesian prior by which to evaluate applicants for a job or something, thinking that someone is less smart or less rational because of a preference is like judging someone’s cooking skills by whether they enjoy a fast food burger.
- ^
If “I’m allowed to be irrational sometimes” is what helps you live a better, less stressful life, feel free to ignore all this and go about your day. My point is that, by my definition of rational, there are many, many circumstances where it is completely rational to decide to eat ice cream (or whatever) rather than donate to malaria nets (or whatever).
- ^
I don’t mean to oversimplify this, and could talk for hours on the topic. Wants are often influenced by beliefs! Feelings are often hard to put into legible words! Some people have trouble feeling anything they want as physical sensations, and some things we say we want are only things we think we should want. Beliefs and predictions that influence wants may be false or miscalibrated, but the feeling itself, the want itself, just is what it is, the same way sensations of hunger or heat just are what they are.
Also, “you either do or do not feel a want” is not the same as “you either do now or you never will.” Preferences can, and inevitably often do, change over time from new experiences, or repetitions of similar experiences.
- ^
Reminded from a comment to highlight that this is an ongoing process. By no means are your emotional responses or preferences locked-in from some vague “past” or developmental experiences; people can and do change the way they feel about things and acquire new preferences over the course of their life. But in the moment, it is important to note that you cannot arbitrarily change your preferences.
- ^
Though the implications of what it means to “leave” a community should itself be unpacked. It implies that being part of a culture is all-or-nothing, which is more how cults operate than communities. Part of why it’s “healthier” to be part of multiple communities is that it creates a robustness against single-culture pressures. While some people might find only one community that deeply matches their ideals and preferences, it’s still extremely unlikely for one community to match every preference someone has. Instead, a healthy community should include multiple slightly different sub-cultures, so that people can shift from one to another if they feel their preferences aren’t sufficiently matched.
This post would be a LOT better with about half-a-dozen representative real examples you’ve run into.
I’m a little confused. Do the examples in the post all seem purely hypothetical to you? They’re all real things I have encountered or heard from others:
Whether or not it’s rational to have ice cream (or other unhealthy indulgences).
Whether or not wanting your high-earning partner to do housework is reasonable.
Whether or not you want to receive unfiltered criticism or judgements.
Whether being mono vs open vs poly is a sign of rationality.
Whether your career preference is a sign of how smart or rational you are.
Obviously not all are as equally detailed, and I could always add more, but if it doesn’t seem concrete enough yet, I’m not sure what else I should add or in how much detail to hit the target better.
The above statement could be applied to a LOT of other posts too, not just this one.
To add some nuance....
While I think this is a very useful frame, particularly for people who have oppressive legibility-valuing parts, and it is likely something many people would benefit from hearing, I doubt this is great as descriptive model.
Model in my view closer to reality is, there isn’t that sharp difference between “wants” and “beliefs”, and both “wants” and “beliefs” do update.
Wants are often represented by not very legible taste boxes, but these boxes do update upon being fed data. To continue an example from the post, let’s talk about literal taste and ice cream. While whether you want or don’t want or like or don’t like an icecream sounds like pure want, it can change, develop or even completely flip, based on what you do. There is the well known concept of acquired taste: maybe the first time you see a puerh ice cream on offer, it does not seem attractive. Maybe you taste it and still dislike it. But maybe, after doing it a few times, you actually start to like it. The output of the taste box changed. The box will likely also update if some flavour of icecream is very high-status in your social environment; when you will get horrible diarrhea from the meal you ate just before the ice cream; and in many other cases.
Realizing that your preferences can and do develop obviously opens the Pandora’s box of actions which do change preferences.[1] The ability to do that breaks orthogonality. Feed your taste boxes slop and you may start enjoying slop. Surround yourself with people who do a lot of [x] and … it you may find you like and do [x] as well, not because someone told you “it’s the rational thing to do”, but because learning, dynamics between your different parts, etc.
Actually, all actions do!
Completely agree, and for what it’s worth, I don’t think anything in the frame of my post contradicts these points.
“You either do or do not feel a want” is not the same as “you either do now or you never will,” and I note that conditioning is also a cause of preferences, though I will edit to highlight that this is an ongoing process in case it sounds like I was saying it’s all locked-in from some vague “past” or developmental experiences (which was not my intent).
The orthogonality thesis is also for human beings.
I don’t think so; even if it applies to the subset of hypothetical superintelligences that factor neatly into beliefs and values, humans don’t seem to factorize this way (see Obliqueness Thesis, esp. argument from brain messiness).
I think this may be part of the disconnect between me and the article. I often view the short jolt preferences (that you get from seeing an ice-cream shop) as heuristics, as effectively predictions paired with some simpler preference for “sweet things that make me feel all homey and nice”. These heuristics can be trained to know how to weigh the costs, though I agree just having a “that’s irrational” / “that’s dumb” is a poor approach to it. Other preferences, like “I prefer these people to be happy” are not short-jolts but rather thought about and endorsed values that would take quite a bit more to shift—but are also significantly influenced by beliefs too.
Other values like “I enjoy this aesthetic” seem more central to your argument than short-jolts or considered values.
This is why you could view a smoker’s preference for another cigarette as irrational: the ‘core want’ is just a simple preference for the general feel of smoking a cigarette, but the short-jolt preference has the added prediction of “and this will be good to do”. But that added prediction is false and inconsistent with everything they know. The usual statement of “you would regret this in the future”. Unfortunately, the short-jolt preference often has enough strength to get past the other preferences, which is why you want to downweight it.
So, I agree that there’s various preferences that having them is disentangled from whether you’re rational or not, but that I also think most preferences are quite entangled with predictions about reality.
I agree that humans can’t manage this, but it does still make sense for a non-monolithic entity—You’d take there being an inconsistency as a sign that there’s a problem, which is what people tend to do, even if ti can’t be fixed.
Commenting on a relatively isolated point in what you wrote; none of this affects your core point about preferences being entangled with predictions (actually it relies on it).
I think that the short-jolt preference’s prediction is actually often correct; it’s just over a shorter time horizon. The short-term preference predicts that “if I take this smoke, then I will feel better” and it is correct. The long-term preference predicts that “I will later regret taking this smoke, ” and it is also correct. Neither preference is irrational, they’re just optimizing over different goals and timescales.
Now it would certainly be tempting to define rationality as something like “only taking actions that you endorse in the long term”, but I’d be cautious of that. Some long-term preferences are genuinely that, but many of them are also optimizing for something looking good socially, while failing to model any of the genuine benefits of the socially-unpopular short-term actions.
For example, smoking a cigarette often gives smokers a temporary feeling of being in control, and if they are going out to smoke together with others, a break and some social connection. It is certainly valid to look at those benefits and judge that they are still not worth the long-term costs… but frequently the “long-term” preference may be based on something like “smoking is bad and uncool and I shouldn’t do it and I should never say that there could be a valid reason to do for otherwise everyone will scold me”.
Then by maintaining both the short-term preference (which continues the smoking habit) and the long-term preference (which might make socially-visible attempts to stop smoking), the person may be getting the benefit from smoking while also avoiding some of the social costs of continuing.
This is obviously not to say that the costs of smoking would only be social. Of course there are genuine health reasons as well. But I think that quite a few people who care about “health” actually care about not appearing low status by doing things that everyone knows are unhealthy.
Though even if that wasn’t the case—how do you weigh the pleasure of a cigarette now, versus increased probability of various health issues some time in the future? It’s certainly very valid to say that better health in the future outweighs the pleasure in the now, but there’s also no objective criteria for why that should be; you could equally consistently put things other way around.
So I don’t think that smoking a cigarette is necessarily irrational in the sense of making an incorrect prediction. It’s more like a correct but only locally optimal prediction. (Though it’s also valid to define rationality as something like “globally optimal behavior”, or as the thing that you’d do if you got both the long-term and the short-term preference to see each other’s points and then make a decision that took all the benefits and harms into consideration.)
Reminds me of “Self-Integrity and the Drowning Child” which talks about another kind of way that people in EA/rat communities are liable to hammer down parts of themselves.
I’d enjoy some acknowledgement that there IS an interplay between cognitive beliefs (based on intelligent modeling of the universe and other people) and intuitive experienced emotions. “not a monocausal result of how smart or stupid they are” does not imply total lack of correlation or impact. Nor does it imply that cognitive ability to choose a framing or model is not effective in changing one’s aliefs and preferences.
I’m fully onboard with countering the bullying and soldier-mindset debate techniques that smart people use against less-smart (or equally-smart but differently-educated) people. I don’t buy that everyone is entitled to express and follow any preferences, including anti-social or harmful-to-others beliefs. Some things are just wrong in modern social contexts.
To your first point, I do believe the post covers this; specifically, the idea that e.g. frames and predictions can be mistaken, and correcting those mistakes can change emotional reactivity. Is that not what you mean?
For the second point… if they follow their preferences, they are acting, and if they are wrong it’s because it causes harm, no? I do not believe preferences themselves, or expressing them, should ever be considered wrong; that seems an artifact of Puritanical norms and fears.
Are you really certain that one’s desires are just givens that one has no rational influence over? I’m skeptical.
https://www.lesswrong.com/posts/aQQ69PijQR2Z64m2z/notes-on-temperance#Can_we_shape_our_desires_
I don’t see how your question contradicts my statement, nor that link. People absolutely develop in their desires over time, and can change them, but that is not the same as being able to decide, in the moment, that you do not like the taste of pizza if your tongue is having the sensory experience of enjoying it.
Do you think wants that arise from conscious thought processes are equally valid to wants that arise from feelings? How do you think about that?
Good question. Curious to hear what the OP thinks, too.
Personally I’m not convinced that the results of a conscious process are actually “wants” in the sense the described here, until they become more deeply internalized. Like, obviously if I want ice cream it’s partly because at some point I consciously chose to try it and (plausibly) wanted to try it. But I don’t know that I can choose to want something, as opposed to putting myself in the position of choosing something with the hope or expectation that will come to want it.
The way I think about it, I can choose to try things, or do things. I can choose to want to want things, or want to like things. As I try and do the things I want to want and like, I may come to want them. I can use various techniques to make those subconscious changes faster, easier, or more likely. But I don’t think I can choose to want things.
I do think this matters, because in the long run, choosing to not do or get the things you want, in favor of the things you consciously think you should want, or want to want, but don’t, is not good for mental health.
I think wants that arise from conscious thought are, fundamentally, wants that arise from feelings attached to those conscious thoughts. The conscious thought processes may be mistaken in many ways, but they still evoke memories or predictions that trigger emotions associated with imagined world-states, which translate to wants or not-wants.
Well if they consistently make recommendations that in retrospect end up looking good then maybe you’re bad at understanding. Or maybe they’re bad at explaining. But trusting them when you don’t understand their recommendation is exploitable so maybe they’re running a strategy where they deliberately make good recommendations with poor explanations so when you start trusting them they can start mixing in exploitative recommendations (which you can’t tell apart because all recommendations have poor explanations).
So I’d really rather not do that in community context. There are ways to work with that. Eg. boss can skip some details of employees recommendations and if results are bad enough fire the employee. On the other hand I think it’s pretty common for employee to act in their own interest. But yeah, we’re talking principal-agent problem at that point and tradeoffs what’s more efficient...
Taking what you write as excuse to nerd a bit about Hyperbolic Discounting
One way to paraphrase esp. some of your ice cream example:
Hyperbolic discounting—the habit of valuing this moment a lot while abruptly (not smoothly exponentially) discounting everything coming even just a short while after—may in a technical sense be ‘time inconsistent’, but it’s misguided to call it ‘irrational’ in the common usage of the term: My current self may simply care about itself distinctly more than about the future selves, even if some of these future selves are forthcoming relatively soon. It’s my current self’s preference structure, and preferences are not rational or irrational, basta.
I agree and had been thinking this, and I find it an interesting counterpoint to the usual description of hyperbolic discounting as ‘irrational’.
It is a bit funny also as we have plenty of discussions trying to explain when/why some hyperbolic discounting may actually be “rational” (ex. here, here, here), but I’ve not yet seen any so fundamental (and simple) rejection of the notion of irrationality (though maybe I’ve just missed it so far).
(Then, with their dubious habits of using common terms in subtly misleading ways, fellow economists may rebut that we have simply defined irrationality in this debate as meaning to have non-exponential alias time-inconsistent preferences, justifying the term ‘irrationality’ here quasi by definition)