Replace “give human heroin” with “replace the human with another being whose utility function is easier to satisfy, like a rock”, and this conclusion seems sort of trivial. It has nothing to do with whether or not humans are rational. Heroin is an example of a thing that modifies our utility functions. Heroin might as well replace the human with a different entity, that has a slightly different utility function.
In fact I don’t see how the human in this situation is being irrational at all. Not doing heroin unless you are already addicted seems like a reasonable behavior.
Heroin might as well replace the human with a different entity, that has a slightly different utility function.
We feel that that is true, but “heroin replaces the human’s utility” and “humans have composite utility where heroin is concerned” both lead to identical predictions. So you can’t deduce the human’s utility merely from observation; you need priors over what is irrational and what isn’t.
Replace “give human heroin” with “replace the human with another being whose utility function is easier to satisfy, like a rock”, and this conclusion seems sort of trivial. It has nothing to do with whether or not humans are rational. Heroin is an example of a thing that modifies our utility functions. Heroin might as well replace the human with a different entity, that has a slightly different utility function.
In fact I don’t see how the human in this situation is being irrational at all. Not doing heroin unless you are already addicted seems like a reasonable behavior.
We feel that that is true, but “heroin replaces the human’s utility” and “humans have composite utility where heroin is concerned” both lead to identical predictions. So you can’t deduce the human’s utility merely from observation; you need priors over what is irrational and what isn’t.