Redefining “utility” like this doesn’t help us with the actual problem at hand:
I guess what I’m suggesting, in part, is that the actual problem at hand isn’t well-defined, unless you specify what you mean by utility in advance.
what do we do if Omega offers to double the f(x) which we’re actually maximizing?
You take cards every time, obviously. But then the result is tautologically true and pretty uninteresting, AFAICT. (The thread beginning here has more on this.) It’s also worth noting that there are vNM-rational preferences for which Omega could not possibly make this offer (f(x) bounded above and q greater than half the bound.)
In your restatement of the problem, the only thing we assume about Omega’s offer is that it would change the universe in a desirable way.
That’s only true given a particular assumption about what the output of V(x) means. If I say that V(x) is, say, a cardinally measurable and interpersonally comparable measure of my well-being, then Omega’s offer to double means rather more than that.
But all this means is that Omega is offering us the wrong thing,
“Wrong” relative to what? Omega offers whatever Omega offers. We can specify the thought experiment any way we like if it helps us answer questions we are interested in. My point is that you can’t learn anything interesting from the thought experiment if Omega is offering to double f(x), so we shouldn’t set it up that way.
which we don’t really value.
Eh? “Valutilons” are specifically defined to be a measure of what we value.
I guess what I’m suggesting, in part, is that the actual problem at hand isn’t well-defined, unless you specify what you mean by utility in advance.
Utility means “the function f, whose expectation I am in fact maximizing”. The discussion then indeed becomes whether f exists and whether it can be doubled.
My point is that you can’t learn anything interesting from the thought experiment if Omega is offering to double f(x), so we shouldn’t set it up that way.
That was the original point of the thread where the thought experiment was first discussed, though.
The interesting result is that if you’re maximizing something you may be vulnerable to a failure mode of taking risks that can be considered excessive. This is in view of the original goals you want to achieve, to which maximizing f is a proxy—whether a designed one (in AI) or an evolved strategy (in humans).
“Valutilons” are specifically defined to be a measure of what we value.
If “we” refers to humans, then “what we value” isn’t well defined.
Utility means “the function f, whose expectation I am in fact maximizing”.
There are many definitions of utility, of which that is one. Usage in general is pretty inconsistent. (Wasn’t that the point of this post?) Either way, definitional arguments aren’t very interesting. ;)
The interesting result is that if you’re maximizing something you may be vulnerable to a failure mode of taking risks that can be considered excessive.
Your maximand already embodies a particular view as to what sorts of risk are excessive. I tend to the view that if you consider the risks demanded by your maximand excessive, then you should either change your maximand, or change your view of what constitutes excessive risk.
There are many definitions of utility, of which that is one. Usage in general is pretty inconsistent. (Wasn’t that the point of this post?) Either way, definitional arguments aren’t very interesting. ;)
Yes, that was the point :-) On my reading of OP, this is the meaning of utility that was intended.
Your maximand already embodies a particular view as to what sorts of risk are excessive. I tend to the view that if you consider the risks demanded by your maximand excessive, then you should either change your maximand, or change your view of what constitutes excessive risk.
Yes. Here’s my current take:
The OP argument demonstrates the danger of using a function-maximizer as a proxy for some other goal. If there can always exist a chance to increase f by an amount proportional to its previous value (e.g. double it), then the maximizer will fall into the trap of taking ever-increasing risks for ever-increasing payoffs in the value of f, and will lose with probability approaching 1 in a finite (and short) timespan.
This qualifies as losing if the original goal (the goal of the AI’s designer, perhaps) does not itself have this quality. This can be the case when the designer sloppily specifies its goal (chooses f poorly), but perhaps more interesting/vivid examples can be found.
To expand on this slightly, it seems like it should be possible to separate goal achievement from risk preference (at least under certain conditions).
You first specify a goal function g(x) designating the degree to which your goals are met in a particular world history, x. You then specify another (monotonic) function, f(g) that embodies your risk-preference with respect to goal attainment (with concavity indicating risk-aversion, convexity risk-tolerance, and linearity risk-neutrality, in the usual way). Then you maximise E[f(g(x))].
If g(x) is only ordinal, this won’t be especially helpful, but if you had a reasonable way of establishing an origin and scale it would seem potentially useful. Note also that f could be unbounded even if g were bounded, and vice-versa. In theory, that seems to suggest that taking ever increasing risks to achieve a bounded goal could be rational, if one were sufficiently risk-loving (though it does seem unlikely that anyone would really be that “crazy”). Also, one could avoid ever taking such risks, even in the pursuit of an unbounded goal, if one were sufficiently risk-averse that one’s f function were bounded.
P.S.
On my reading of OP, this is the meaning of utility that was intended.
I guess what I’m suggesting, in part, is that the actual problem at hand isn’t well-defined, unless you specify what you mean by utility in advance.
You take cards every time, obviously. But then the result is tautologically true and pretty uninteresting, AFAICT. (The thread beginning here has more on this.) It’s also worth noting that there are vNM-rational preferences for which Omega could not possibly make this offer (f(x) bounded above and q greater than half the bound.)
That’s only true given a particular assumption about what the output of V(x) means. If I say that V(x) is, say, a cardinally measurable and interpersonally comparable measure of my well-being, then Omega’s offer to double means rather more than that.
“Wrong” relative to what? Omega offers whatever Omega offers. We can specify the thought experiment any way we like if it helps us answer questions we are interested in. My point is that you can’t learn anything interesting from the thought experiment if Omega is offering to double f(x), so we shouldn’t set it up that way.
Eh? “Valutilons” are specifically defined to be a measure of what we value.
Utility means “the function f, whose expectation I am in fact maximizing”. The discussion then indeed becomes whether f exists and whether it can be doubled.
That was the original point of the thread where the thought experiment was first discussed, though.
The interesting result is that if you’re maximizing something you may be vulnerable to a failure mode of taking risks that can be considered excessive. This is in view of the original goals you want to achieve, to which maximizing f is a proxy—whether a designed one (in AI) or an evolved strategy (in humans).
If “we” refers to humans, then “what we value” isn’t well defined.
There are many definitions of utility, of which that is one. Usage in general is pretty inconsistent. (Wasn’t that the point of this post?) Either way, definitional arguments aren’t very interesting. ;)
Your maximand already embodies a particular view as to what sorts of risk are excessive. I tend to the view that if you consider the risks demanded by your maximand excessive, then you should either change your maximand, or change your view of what constitutes excessive risk.
Yes, that was the point :-) On my reading of OP, this is the meaning of utility that was intended.
Yes. Here’s my current take:
The OP argument demonstrates the danger of using a function-maximizer as a proxy for some other goal. If there can always exist a chance to increase f by an amount proportional to its previous value (e.g. double it), then the maximizer will fall into the trap of taking ever-increasing risks for ever-increasing payoffs in the value of f, and will lose with probability approaching 1 in a finite (and short) timespan.
This qualifies as losing if the original goal (the goal of the AI’s designer, perhaps) does not itself have this quality. This can be the case when the designer sloppily specifies its goal (chooses f poorly), but perhaps more interesting/vivid examples can be found.
To expand on this slightly, it seems like it should be possible to separate goal achievement from risk preference (at least under certain conditions).
You first specify a goal function g(x) designating the degree to which your goals are met in a particular world history, x. You then specify another (monotonic) function, f(g) that embodies your risk-preference with respect to goal attainment (with concavity indicating risk-aversion, convexity risk-tolerance, and linearity risk-neutrality, in the usual way). Then you maximise E[f(g(x))].
If g(x) is only ordinal, this won’t be especially helpful, but if you had a reasonable way of establishing an origin and scale it would seem potentially useful. Note also that f could be unbounded even if g were bounded, and vice-versa. In theory, that seems to suggest that taking ever increasing risks to achieve a bounded goal could be rational, if one were sufficiently risk-loving (though it does seem unlikely that anyone would really be that “crazy”). Also, one could avoid ever taking such risks, even in the pursuit of an unbounded goal, if one were sufficiently risk-averse that one’s f function were bounded.
P.S.
You’re probably right.
Crap. Sorry about the delete. :(