Well, a couple things. You can in part interpret that as being an underlying preference to do so, but you seem to have akrasia stopping you from actually choosing what you know you actually want.
Or perhaps you actually would prefer not to go on coasters, and consider the “after the fact” to be the same as “after the fact of taking some addictive drug, you might like it, so you wouldn’t want to in the first place”
As far as changing of preferences, you may think of your true preferences as encoded by the underlying algorithm your brain is effectively implementing, the thing that controls how your more visible to yourself preferences change in response to new information, arguments, etc etc etc...
Those underlying underlying preferences are the things that you wouldn’t want to change. You wouldn’t want to take a pill that makes you into the type of person that enjoys committing genocide or whatever, right? But you can predict in advance that if such a pill existed and you took it, then after it rewrote your preferences, you would retroactively prefer genociding. But since you (I assume) don’t want genocides to happen, you wouldn’t want to become the type of person that would want them to happen and would try to make them happen.
(skipping one or two minor caveats in this comment, but you get the idea, right?)
But also, humans tend to be slightly (minor understatement here) irrational. I mean, isn’t the whole project of LW and OB and so on based on the notion of “they way we are is not the way we wish to be. Let us become more rational”? So if something isn’t matching the way people normally behave, well… the problem may be “the way people normally behave”… I believe the usual phrasing is “this is a normative, rather than descriptive theory”
For the most part I think that starts to address it. At the same time, on your last point, there is an important difference between “this is how fully idealized rational agents of a certain sort behave” and “this is how you, a non-fully idealized, partially rational agent should behave, to improve your rationality”.
Someone in perfect physical condition (not just for humans, but for idealized physical beings) has a different optimal workout plan from me, and we should plan differently for various physical activities, even if this person is the ideal towards which I am aiming.
So if we idealize our bayesian models too much, we open up the question: “How does this idealized agent’s behavior relate to how I should behave?” It might be that, were we to design rational agents, it makes sense to use these idealized reasoners as models, but if the goal is personal improvement, we need some way to explain what one might call the Kantian inference from “I am an imperfectly rational being” to “I ought to behave the way such-and-such a perfectly rational being would”.
Well, a couple things. You can in part interpret that as being an underlying preference to do so, but you seem to have akrasia stopping you from actually choosing what you know you actually want.
Or perhaps you actually would prefer not to go on coasters, and consider the “after the fact” to be the same as “after the fact of taking some addictive drug, you might like it, so you wouldn’t want to in the first place”
As far as changing of preferences, you may think of your true preferences as encoded by the underlying algorithm your brain is effectively implementing, the thing that controls how your more visible to yourself preferences change in response to new information, arguments, etc etc etc...
Those underlying underlying preferences are the things that you wouldn’t want to change. You wouldn’t want to take a pill that makes you into the type of person that enjoys committing genocide or whatever, right? But you can predict in advance that if such a pill existed and you took it, then after it rewrote your preferences, you would retroactively prefer genociding. But since you (I assume) don’t want genocides to happen, you wouldn’t want to become the type of person that would want them to happen and would try to make them happen.
(skipping one or two minor caveats in this comment, but you get the idea, right?)
But also, humans tend to be slightly (minor understatement here) irrational. I mean, isn’t the whole project of LW and OB and so on based on the notion of “they way we are is not the way we wish to be. Let us become more rational”? So if something isn’t matching the way people normally behave, well… the problem may be “the way people normally behave”… I believe the usual phrasing is “this is a normative, rather than descriptive theory”
Or did I misunderstand?
For the most part I think that starts to address it. At the same time, on your last point, there is an important difference between “this is how fully idealized rational agents of a certain sort behave” and “this is how you, a non-fully idealized, partially rational agent should behave, to improve your rationality”.
Someone in perfect physical condition (not just for humans, but for idealized physical beings) has a different optimal workout plan from me, and we should plan differently for various physical activities, even if this person is the ideal towards which I am aiming.
So if we idealize our bayesian models too much, we open up the question: “How does this idealized agent’s behavior relate to how I should behave?” It might be that, were we to design rational agents, it makes sense to use these idealized reasoners as models, but if the goal is personal improvement, we need some way to explain what one might call the Kantian inference from “I am an imperfectly rational being” to “I ought to behave the way such-and-such a perfectly rational being would”.