Epistemic status: I may misunderstand parts of the post.
Utility functions are inherently coherent:
More utility is better
Twice as much is twice as good.
One problem is that we usually do not know the utility function of a system. Knowing this could be useful, for example, if we want to talk about what an advanced AI is optimizing for.
So how do we figure out the utility of a system? One solution is to have a measuring stick of it, i.e., a resource such that:
The system finds the resource only instrumentally valuable.
The system can trade the resource in equal proportions to utility
Possible examples of such a resource:
Money
Energy
If we have found such a measuring stick for a certain utility maximizer, then we can use it to determine the utility function by observing which trades the system would make. Additionally, we may measure the extent to which the system is a utility maximizer at all by measuring how coherent its trades with the resource are.
The measuring stick problem asks: How do we find a measuring stick in the wild, when being handed a universe?
Opinion/Confusions
There are some things that confuse me about this and that make me wonder if this is the right frame. There seem some hidden assumptions on optimization in the post that are not transparent to me personally. Some of the confusions may be answered in the post itself, but they do not… “stick”.
John seems to assume that there is one resource that is equally valuable for maximizing utility for all systems. Why is that? Resources like energy may seem useful for transforming the whole universe, but maybe my utility function is not about making such large-scale transformations.
Take humans as examples: most humans experience strong diminishing returns in almost all (or maybe even all) resources.
Possibly John’s answer would be “well, if you don’t want to make large-scale transformations, then you don’t have a utility function, because utility functions are about transforming a large system into a small set of states”.
John’s earlier post on utility maximization being description length minimization seems to support that view.
There is, however, an intuition in myself that finds that post unnatural, especially since the mapping between utility function and probability distribution had a free parameter
What if a system has a utility function, but it’s simply not about states in the same space that John cares about?
E.g., some entity may care about optimizing trajectories, not states.
In that example, we would end up with a distribution over trajectories instead of states. I would imagine that a different type of resource would help in optimizing that than in optimizing states themselves.
I’m not sure that “twice as much is twice as good” is a necessary property of utility functions. Though I guess it may be true by definition?
IIRC, utility functions are only unique up to positive affine transformations. And also I think only the gaps in utility between lairs of options are meaningful.
100⁄200 utility is meaningless without a “baseline” with which to evaluate it. And even then, it’s the gap between the baseline and a given utility value that would be meaningful not the utility value itself.
Suspect your last claim about the coherence of utility functions is just wrong.
The system can trade the resource in equal proportions to utility
Human utility functions over money are very much non linear (closer to logarithmic in fact) and utility functions over energy may also be sublinear.
John seems to assume that there is one resource that is equally valuable for maximizing utility for all systems. Why is that?
Resources like energy may seem useful for transforming the whole universe, but maybe my utility function is not about making such large-scale transformations.
Take humans as examples: most humans experience strong diminishing returns in almost all (or maybe even all) resources.
I think this is only an issue because you assume that for a resource to qualify as a “measuring stick”, the quantity of the resource possessed must be a linear function of utility.
I think that’s an unnecessary assumption and not very sensible because as you said, diminishing marginal returns on resources is nigh universal for humans.
Also I don’t think making large scale changes is relevant/load bearing for a resource to be a measuring stick of utility.
I think the only requirements are that:
The resource is fungible/can be traded for other resources or things the agent cares about
Agent preferences over the resource are monotonically nondecreasing as a function of the quantity of the resource possessed (this is IMO the property that Wentworth was gesturing at with “additive”)
Summary
Epistemic status: I may misunderstand parts of the post.
Utility functions are inherently coherent:
More utility is better
Twice as much is twice as good.
One problem is that we usually do not know the utility function of a system. Knowing this could be useful, for example, if we want to talk about what an advanced AI is optimizing for.
So how do we figure out the utility of a system? One solution is to have a measuring stick of it, i.e., a resource such that:
The system finds the resource only instrumentally valuable.
The system can trade the resource in equal proportions to utility
Possible examples of such a resource:
Money
Energy
If we have found such a measuring stick for a certain utility maximizer, then we can use it to determine the utility function by observing which trades the system would make. Additionally, we may measure the extent to which the system is a utility maximizer at all by measuring how coherent its trades with the resource are.
The measuring stick problem asks: How do we find a measuring stick in the wild, when being handed a universe?
Opinion/Confusions
There are some things that confuse me about this and that make me wonder if this is the right frame. There seem some hidden assumptions on optimization in the post that are not transparent to me personally. Some of the confusions may be answered in the post itself, but they do not… “stick”.
John seems to assume that there is one resource that is equally valuable for maximizing utility for all systems. Why is that? Resources like energy may seem useful for transforming the whole universe, but maybe my utility function is not about making such large-scale transformations.
Take humans as examples: most humans experience strong diminishing returns in almost all (or maybe even all) resources.
Possibly John’s answer would be “well, if you don’t want to make large-scale transformations, then you don’t have a utility function, because utility functions are about transforming a large system into a small set of states”.
John’s earlier post on utility maximization being description length minimization seems to support that view.
There is, however, an intuition in myself that finds that post unnatural, especially since the mapping between utility function and probability distribution had a free parameter
What if a system has a utility function, but it’s simply not about states in the same space that John cares about?
E.g., some entity may care about optimizing trajectories, not states.
In that example, we would end up with a distribution over trajectories instead of states. I would imagine that a different type of resource would help in optimizing that than in optimizing states themselves.
I’m not sure that “twice as much is twice as good” is a necessary property of utility functions. Though I guess it may be true by definition?
IIRC, utility functions are only unique up to positive affine transformations. And also I think only the gaps in utility between lairs of options are meaningful.
100⁄200 utility is meaningless without a “baseline” with which to evaluate it. And even then, it’s the gap between the baseline and a given utility value that would be meaningful not the utility value itself.
Suspect your last claim about the coherence of utility functions is just wrong.
Human utility functions over money are very much non linear (closer to logarithmic in fact) and utility functions over energy may also be sublinear.
The law of diminishing marginal utility suggests that sublinear utility over any resource is the norm for humans.
I think this is only an issue because you assume that for a resource to qualify as a “measuring stick”, the quantity of the resource possessed must be a linear function of utility.
I think that’s an unnecessary assumption and not very sensible because as you said, diminishing marginal returns on resources is nigh universal for humans.
Also I don’t think making large scale changes is relevant/load bearing for a resource to be a measuring stick of utility.
I think the only requirements are that:
The resource is fungible/can be traded for other resources or things the agent cares about
Agent preferences over the resource are monotonically nondecreasing as a function of the quantity of the resource possessed (this is IMO the property that Wentworth was gesturing at with “additive”)
See also my top level comment.