These preferences (like refusing the heroin) are still fully valid human preferences, and you’re going to need to look at human behavior to figure out what they are (barring big godlike a priori reasoning), which entails basically similar philosophical problems as getting all values from behavior without this framework.
I’m hopeful that this won’t be true in a certain, limited way. That is, in a certain sense, scanning brains and observing how neurons operate to determine the behavior of a human is a very different sort of operation from observing their behavior “from the outside” the way we observe people’s behavior today. Much of the difficulty is that because observing behavior we can see only with our unaided senses and without a deep model of the brain forces us to make very large normative assumptions to get the necessary power to infer things about how a human values things, but if we have a model like this and it appears to be correct then we can, practically speaking, make “smaller”, less powerful normative assumptions because we understand and can work out the details of more of the gears of the mind.
The result is that in a certain sense we are still concerned with behavior, but because the level of detail is so much higher and the model so much richer we are less likely to find ourselves making mistakes from having taken large inferential leaps as we would if we observed behavior in the normal sense.
I’m hopeful that this won’t be true in a certain, limited way. That is, in a certain sense, scanning brains and observing how neurons operate to determine the behavior of a human is a very different sort of operation from observing their behavior “from the outside” the way we observe people’s behavior today. Much of the difficulty is that because observing behavior we can see only with our unaided senses and without a deep model of the brain forces us to make very large normative assumptions to get the necessary power to infer things about how a human values things, but if we have a model like this and it appears to be correct then we can, practically speaking, make “smaller”, less powerful normative assumptions because we understand and can work out the details of more of the gears of the mind.
The result is that in a certain sense we are still concerned with behavior, but because the level of detail is so much higher and the model so much richer we are less likely to find ourselves making mistakes from having taken large inferential leaps as we would if we observed behavior in the normal sense.