You can at least simplify the problem somewhat by applying VNM utility using each of the candidate utility functions, and throwing out all solutions that do not appear in any of them. If you think you like either carrots or apples, you’re not going to go to the store and buy asparagus.
The other thorny issue is that uncertainty in the utility function makes learning about your utility function valuable. If you think you like either carrots or apples, then taking two grocery trips is the best answer—on the first trip you buy a carrot and an apple and figure out which one you like, and on the second trip you stock up.
The other thing is that I don’t think it’s possible to model uncertainty inside your utility function—you can only have uncertainty about how you evaluate certain events. If you don’t know whether or not you like carrots, that’s a fact about eating carrots and not one about how to decide whether or not to eat a carrot. I think that every uncertainty about a utility function is just a hidden uncertainty about how the being the experiences utility works.
Let me be specific about the math. Suppose you have a lottery L with a 1/3rd chance of result A and a 2/3rd chance of result B. Suppose furthermore that you are uncertain about whether you enjoy things as in U1 or U2, with equal probability of each. L is equivalent to a lottery with 1⁄6 chance (A, U1), 1⁄3 chance (B, U1), etc. Now you can make the first utility function of this exercise that takes into account all your uncertainty about preferences.
Note that U1 and U2 aren’t numbers—it’s how much you enjoy something if your preferences are as in U1.
What this lets us do is convert “there’s a chance I get turned into a whale and I’m not sure if I will like it” into “there’s a chance that I get turned into a whale and like it, and another chance that I get turned into a whale and don’t like it”.
Ooops. Radiation poisoning. Utility is about planning, not experiencing or enjoying.
What this lets us do is convert “there’s a chance I get turned into a whale and I’m not sure if I will like it” into “there’s a chance that I get turned into a whale and like it, and another chance that I get turned into a whale and don’t like it”.
I went through the math a couple days ago with another smart philosopher-type. We are pretty sure that this (adding preference uncertainty as an additional dimension of your ontology) is a fully general solution to preference uncertainty. Unfortunately, it requires a bit of moral philosophy to pin down the relative weights of the utility functions. That is, the utility functions and their respective probabilities is not enough to uniquely identify the combined utility function. Which is actually totally ok, because you can get that information from the same source where you got the partial utility functions.
I’ll go through the proof and implications/discussion in an upcoming post. Hopefully. I don’t exactly have a track record of following through on things...
Unfortunately, it requires a bit of moral philosophy to pin down the relative weights of the utility functions. That is, the utility functions and their respective probabilities is not enough to uniquely identify the combined utility function.
Right, to get that answer you need to look inside your utility function… which you’re uncertain about. Stated differently, your utility function tells you how to deal with uncertainty about your utility function, but that’s another thing you’re uncertain about. But luckily your utility function tells you how do deal with uncertainty about uncertainty about your utility function… I think you can see where this is going.
Naively, my intuition is that simply adding uncertainty about preferences as part of your ontology isn’t enough because of this regress—you still don’t even know in principle how to choose between actions without more precise knowledge of your utility function. However, this regress sounds suspiciously like the sort of thing that once formalized precisely isn’t really a problem at all—just “take the limit” as it were.
Your (partial) utility functions do not contain enough information to resolve uncertainty between them. As far as I can tell, utility functions can’t contain meta-preferences.
You can’t just pull a correct utility function out of thin air, though. You got the utility function from somewhere; it is the output of a moral-philosophy process. You resolve the uncertainty with the same information-source from which you constructed the partial utility functions from in the first place.
No need to take the limit or do any extrapolation (except that stuff like that does seem to show up inside the moral-philosophy process.)
I think we’re using “utility function” differently here. I take it to mean the function containing all information about your preferences, preferences about preferences, and higher level meta-preferences. I think you’re using the term to refer to the function containing just object-level preference information. Is that correct?
Now that I make this distinction, I’m not sure VNM utility applies to meta-preferences.
Now that I make this distinction, I’m not sure VNM utility applies to meta-preferences.
It doesn’t, AFAIK, which is why I said your utility function does not contain meta-preference and the whole moral dynamic. “utility function” is only a thing in VNM. Using it as a shorthand for “my whole reflective decision system” is incorrect use of the term, IMO.
I am not entirely sure that your utility function can’t contain meta-preference, though. I could be convinced by some well-placed mathematics.
My current understanding is that you put the preference uncertainty into your ontology, extend your utility function to deal with those extra dimensions, and lift the actual moral updating to epistemological work over those extra ontology-variables. This still requires some level of preliminary moral philosophy to shoehorn your current incoherent godshatter-soup into that formal framework.
I’ll hopefully formalize this some day soon to something coherent enough to be criticized.
Nice catch on the radiation poisoning. Revised sentence:
I think that every uncertainty about a utility function is just a hidden uncertainty about how to weigh the different experiences that generate a utility function
Also
That is, the utility functions and their respective probabilities is not enough to uniquely identify the combined utility function.
This is 100% expected, since utility functions that vary merely by a scaling factor and changing the zero point are equivalent.
I think we’re talking about the same thing when you say “adding preference uncertainty as an additional dimension of your ontology”. It’s kind of hard to tell at this level of abstraction.
You can at least simplify the problem somewhat by applying VNM utility using each of the candidate utility functions, and throwing out all solutions that do not appear in any of them. If you think you like either carrots or apples, you’re not going to go to the store and buy asparagus.
The other thorny issue is that uncertainty in the utility function makes learning about your utility function valuable. If you think you like either carrots or apples, then taking two grocery trips is the best answer—on the first trip you buy a carrot and an apple and figure out which one you like, and on the second trip you stock up.
The other thing is that I don’t think it’s possible to model uncertainty inside your utility function—you can only have uncertainty about how you evaluate certain events. If you don’t know whether or not you like carrots, that’s a fact about eating carrots and not one about how to decide whether or not to eat a carrot. I think that every uncertainty about a utility function is just a hidden uncertainty about how the being the experiences utility works.
Let me be specific about the math. Suppose you have a lottery L with a 1/3rd chance of result A and a 2/3rd chance of result B. Suppose furthermore that you are uncertain about whether you enjoy things as in U1 or U2, with equal probability of each. L is equivalent to a lottery with 1⁄6 chance (A, U1), 1⁄3 chance (B, U1), etc. Now you can make the first utility function of this exercise that takes into account all your uncertainty about preferences.
Note that U1 and U2 aren’t numbers—it’s how much you enjoy something if your preferences are as in U1.
What this lets us do is convert “there’s a chance I get turned into a whale and I’m not sure if I will like it” into “there’s a chance that I get turned into a whale and like it, and another chance that I get turned into a whale and don’t like it”.
Ooops. Radiation poisoning. Utility is about planning, not experiencing or enjoying.
I went through the math a couple days ago with another smart philosopher-type. We are pretty sure that this (adding preference uncertainty as an additional dimension of your ontology) is a fully general solution to preference uncertainty. Unfortunately, it requires a bit of moral philosophy to pin down the relative weights of the utility functions. That is, the utility functions and their respective probabilities is not enough to uniquely identify the combined utility function. Which is actually totally ok, because you can get that information from the same source where you got the partial utility functions.
I’ll go through the proof and implications/discussion in an upcoming post. Hopefully. I don’t exactly have a track record of following through on things...
Right, to get that answer you need to look inside your utility function… which you’re uncertain about. Stated differently, your utility function tells you how to deal with uncertainty about your utility function, but that’s another thing you’re uncertain about. But luckily your utility function tells you how do deal with uncertainty about uncertainty about your utility function… I think you can see where this is going.
Naively, my intuition is that simply adding uncertainty about preferences as part of your ontology isn’t enough because of this regress—you still don’t even know in principle how to choose between actions without more precise knowledge of your utility function. However, this regress sounds suspiciously like the sort of thing that once formalized precisely isn’t really a problem at all—just “take the limit” as it were.
That’s not the issue we ran into.
Your (partial) utility functions do not contain enough information to resolve uncertainty between them. As far as I can tell, utility functions can’t contain meta-preferences.
You can’t just pull a correct utility function out of thin air, though. You got the utility function from somewhere; it is the output of a moral-philosophy process. You resolve the uncertainty with the same information-source from which you constructed the partial utility functions from in the first place.
No need to take the limit or do any extrapolation (except that stuff like that does seem to show up inside the moral-philosophy process.)
I think we’re using “utility function” differently here. I take it to mean the function containing all information about your preferences, preferences about preferences, and higher level meta-preferences. I think you’re using the term to refer to the function containing just object-level preference information. Is that correct?
Now that I make this distinction, I’m not sure VNM utility applies to meta-preferences.
It doesn’t, AFAIK, which is why I said your utility function does not contain meta-preference and the whole moral dynamic. “utility function” is only a thing in VNM. Using it as a shorthand for “my whole reflective decision system” is incorrect use of the term, IMO.
I am not entirely sure that your utility function can’t contain meta-preference, though. I could be convinced by some well-placed mathematics.
My current understanding is that you put the preference uncertainty into your ontology, extend your utility function to deal with those extra dimensions, and lift the actual moral updating to epistemological work over those extra ontology-variables. This still requires some level of preliminary moral philosophy to shoehorn your current incoherent godshatter-soup into that formal framework.
I’ll hopefully formalize this some day soon to something coherent enough to be criticized.
I look forward to it!
Nice catch on the radiation poisoning. Revised sentence:
Also
This is 100% expected, since utility functions that vary merely by a scaling factor and changing the zero point are equivalent.
I think we’re talking about the same thing when you say “adding preference uncertainty as an additional dimension of your ontology”. It’s kind of hard to tell at this level of abstraction.