The bad thing about circular preferences is that they make the number you assign to how good an option is (“utility”-ish thing) not be a function). If you have some quantity of X, normally we’d expect that to determine a unique utility. But that only works because utility is a function! If your decision-maker is based off of some “utility”-ish number that isn’t single-valued, you could have different utilities correspond to the very same quantity of X.
Your post basically says “getting money pumped would require us to have higher utility even if we just had a lower-probability version of something nice we already had—and since any thinking being knows that utility is a function, they will avoid this silly case once they notice it.”
Well, no. If they have circular preferences, they won’t be held back by how utility is actually supposed to be a function. They will make the trades, and really truly have a higher number stored under “utility” than they did before they made the trades. You build a robot that has a high utility if it drives off a cliff, and the robot doesn’t go “wait, this sounds like a bad idea. Maybe I should take up needlepoint instead.” It drives off the cliff. You build a decision-making system with circular preferences, and it drives of a cliff too.
I’m not saying “any thinking being knows that utility is a function,” I’m saying that this creature with a broken brain prefers more X to less X. Instead of having a utility function they have a system of comparing quantities of X, Y, and Z.
I was thinking they would make comparison between what they have at the beginning and what they would have at the end, and it looks like you are making a chain of favorable comparisons to find your way back to X with less of it.
I’m not really sure what algorithm I would write into a robot to decide which path of comparisons to make. Maybe the shortest one (in number of comparisons) that compares the present state to one as far in the future as the robot can predict? But this seems kind of like deducing from contradictory premises.
prefers more X to less X. Instead of having a utility function they have a system of comparing quantities of X, Y, and Z.
Looks like an example might help you to connect this to what I was talking about.
Imagine sqrt(X). Normally people just pick the positive square root or the negative square root—but imagine the whole thing, the parabola-turned-sideways, the thing that isn’t a function.
Now. Is it a valid question to ask whether sqrt(5) is greater or less than sqrt(6)?
-
What a decision-maker with circular preferences can have is local preferences—they can make a single decision at any one moment. But they can’t know how they’d feel about some hypothetical situation unless they also knew how they’d get there. Which sounds strange, I know, because it seems like they should feel sad about coming back to the same place over and over, with slowly diminishing amounts of X. But that’s anthropomorphism talking.
Why would they make that single decision to trade X for Y based on the comparison between X and Y instead of the comparison between X and the less X they know they’ll get? I’m saying that at the moment want Y more than X, but they also want more X more than less X. And they know what kind of decisions they will make in the future.
Now I actually think both positions are answers to an invalid question.
Did you know that the concept of utility was originally used to describe preferences over choices, not states of the world? Might be worth reading up on. One important idea is that you can write preferences entirely in a language of what decisions you make, not how much you value different amounts of X. This is the language that circular preferences can be written in. Pop quiz: if your utility was the angle of a wheel (so, if it turns twice, your utility is 4 pi), could you prove that writing utility in terms of the physical state of the wheel breaks down, but that you can still always have preferences over choices?
This is true as far as it goes, but the OP seems to be talking about a mostly rational agent with some buggy preferences. And in particular it knows its preferences are buggy and in exactly what way. As I mentioned elsewhere, I would expect such an agent to self-modify to apply an internal utility tax on trades among the affected assets, or otherwise compensate for the error. Exactly how it would do this, or even whether it’s possible in a coherent way, is an interesting problem.
The bad thing about circular preferences is that they make the number you assign to how good an option is (“utility”-ish thing) not be a function). If you have some quantity of X, normally we’d expect that to determine a unique utility. But that only works because utility is a function! If your decision-maker is based off of some “utility”-ish number that isn’t single-valued, you could have different utilities correspond to the very same quantity of X.
Your post basically says “getting money pumped would require us to have higher utility even if we just had a lower-probability version of something nice we already had—and since any thinking being knows that utility is a function, they will avoid this silly case once they notice it.”
Well, no. If they have circular preferences, they won’t be held back by how utility is actually supposed to be a function. They will make the trades, and really truly have a higher number stored under “utility” than they did before they made the trades. You build a robot that has a high utility if it drives off a cliff, and the robot doesn’t go “wait, this sounds like a bad idea. Maybe I should take up needlepoint instead.” It drives off the cliff. You build a decision-making system with circular preferences, and it drives of a cliff too.
I thought one usually took the VNM axioms as desiderata and derived that one must have a utility function, rather than the other way round.
This is true, but I think it’s easier to get the picture if you use the full axioms as your point of departure. Iunno.
I’m not saying “any thinking being knows that utility is a function,” I’m saying that this creature with a broken brain prefers more X to less X. Instead of having a utility function they have a system of comparing quantities of X, Y, and Z.
I was thinking they would make comparison between what they have at the beginning and what they would have at the end, and it looks like you are making a chain of favorable comparisons to find your way back to X with less of it.
I’m not really sure what algorithm I would write into a robot to decide which path of comparisons to make. Maybe the shortest one (in number of comparisons) that compares the present state to one as far in the future as the robot can predict? But this seems kind of like deducing from contradictory premises.
Looks like an example might help you to connect this to what I was talking about.
Imagine sqrt(X). Normally people just pick the positive square root or the negative square root—but imagine the whole thing, the parabola-turned-sideways, the thing that isn’t a function.
Now. Is it a valid question to ask whether sqrt(5) is greater or less than sqrt(6)?
-
What a decision-maker with circular preferences can have is local preferences—they can make a single decision at any one moment. But they can’t know how they’d feel about some hypothetical situation unless they also knew how they’d get there. Which sounds strange, I know, because it seems like they should feel sad about coming back to the same place over and over, with slowly diminishing amounts of X. But that’s anthropomorphism talking.
Why would they make that single decision to trade X for Y based on the comparison between X and Y instead of the comparison between X and the less X they know they’ll get? I’m saying that at the moment want Y more than X, but they also want more X more than less X. And they know what kind of decisions they will make in the future.
Now I actually think both positions are answers to an invalid question.
Did you know that the concept of utility was originally used to describe preferences over choices, not states of the world? Might be worth reading up on. One important idea is that you can write preferences entirely in a language of what decisions you make, not how much you value different amounts of X. This is the language that circular preferences can be written in. Pop quiz: if your utility was the angle of a wheel (so, if it turns twice, your utility is 4 pi), could you prove that writing utility in terms of the physical state of the wheel breaks down, but that you can still always have preferences over choices?
This is true as far as it goes, but the OP seems to be talking about a mostly rational agent with some buggy preferences. And in particular it knows its preferences are buggy and in exactly what way. As I mentioned elsewhere, I would expect such an agent to self-modify to apply an internal utility tax on trades among the affected assets, or otherwise compensate for the error. Exactly how it would do this, or even whether it’s possible in a coherent way, is an interesting problem.