While I think this is a very useful frame, particularly for people who have oppressive legibility-valuing parts, and it is likely something many people would benefit from hearing, I doubt this is great as descriptive model.
Model in my view closer to reality is, there isn’t that sharp difference between “wants” and “beliefs”, and both “wants” and “beliefs” do update.
Wants are often represented by not very legible taste boxes, but these boxes do update upon being fed data. To continue an example from the post, let’s talk about literal taste and ice cream. While whether you want or don’t want or like or don’t like an icecream sounds like pure want, it can change, develop or even completely flip, based on what you do. There is the well known concept of acquired taste: maybe the first time you see a puerh ice cream on offer, it does not seem attractive. Maybe you taste it and still dislike it. But maybe, after doing it a few times, you actually start to like it. The output of the taste box changed. The box will likely also update if some flavour of icecream is very high-status in your social environment; when you will get horrible diarrhea from the meal you ate just before the ice cream; and in many other cases.
Realizing that your preferences can and do develop obviously opens the Pandora’s box of actions which do change preferences.[1] The ability to do that breaks orthogonality. Feed your taste boxes slop and you may start enjoying slop. Surround yourself with people who do a lot of [x] and … it you may find you like and do [x] as well, not because someone told you “it’s the rational thing to do”, but because learning, dynamics between your different parts, etc.
To add some nuance....
While I think this is a very useful frame, particularly for people who have oppressive legibility-valuing parts, and it is likely something many people would benefit from hearing, I doubt this is great as descriptive model.
Model in my view closer to reality is, there isn’t that sharp difference between “wants” and “beliefs”, and both “wants” and “beliefs” do update.
Wants are often represented by not very legible taste boxes, but these boxes do update upon being fed data. To continue an example from the post, let’s talk about literal taste and ice cream. While whether you want or don’t want or like or don’t like an icecream sounds like pure want, it can change, develop or even completely flip, based on what you do. There is the well known concept of acquired taste: maybe the first time you see a puerh ice cream on offer, it does not seem attractive. Maybe you taste it and still dislike it. But maybe, after doing it a few times, you actually start to like it. The output of the taste box changed. The box will likely also update if some flavour of icecream is very high-status in your social environment; when you will get horrible diarrhea from the meal you ate just before the ice cream; and in many other cases.
Realizing that your preferences can and do develop obviously opens the Pandora’s box of actions which do change preferences.[1] The ability to do that breaks orthogonality. Feed your taste boxes slop and you may start enjoying slop. Surround yourself with people who do a lot of [x] and … it you may find you like and do [x] as well, not because someone told you “it’s the rational thing to do”, but because learning, dynamics between your different parts, etc.
Actually, all actions do!