Clarification: A utility function maps each state of the world to the real number denoting its utility.
How does this scenario operate under the assumption that humans do not have real-valued utility functions but rather utility orderings? IOW, we can’t arrange all world-states on a number line, but we can always say if one world-state is as good as (or better than) another.
This allows us to deal with infinities, such as “I wouldn’t kill my baby for anything.” That is: There doesn’t exist an N such that U(1) · N > U(B). That simply can’t be true on the (positive) reals; for any A and B real, there’s always a C such that A · C > B.
On any denumerable set with a total ordering on it, we can construct a map into the real numbers that preserves the ordering: Map the first element to 0, the second to 1 if it’s better and −1 if it’s worse, and put each additional one at the end or beginning of the line if it’s better or worse than all, or else into the exact middle of the interval that it falls into.
If you don’t like the denumerability requirement (who knows, the universe accessible to us might eventually come to be infinite, and then there would be more than denumerably many states of the universe), you can also take a utility function you already have, and then add a state that’s better than all others, while preserving the rest of the ordering: Assign to each state from our previous utility function the value that is the arctan of its previous value (the arctan 1-to-1-maps the real numbers onto the numbers between -pi/2 and pi/2 and preserves ordering), then give the new state utility 10.
This allows us to deal with infinities, such as “I wouldn’t kill my baby for anything.”
I don’t know how you will deal with infinities and real humans. It’s quite trivial to construct scenarios under which the person making this statement would change her mind.
Real-valued utility functions can only deal with agents among whom “everybody has their price” — utilities are fungible and all are of the same order. That may actually be the case in the real world, or it may not. But if we assume real-valued utilities, we can’t ask the question of whether it is the case or not, because with real-valued utilities it must be the case.
To pick another example, there could exist a suicidally depressed agent to whom no amount of utility will cause them to evaluate their life as worth living: there doesn’t exist an N such that N + L > 0. Can’t happen with reals. The only way to make this agent become nonsuicidal is to modify the agent, not to drop a bunch of utils on their doorstep.
Well, I’m no mathematician, but I was thinking of something like ordinal arithmetic.
If I understand it correctly, this would let us express value-systems such as —
Both snuggles and chocolate bars have positive utility, but I’d always rather have another snuggle than any number of chocolate bars. So we could say U(snuggle) = ω and U(chocolate bar) = 1. For any amount of snuggling, I’d prefer to have that amount and a chocolate bar (ω·n+1 > ω·n), but given the choice between more snuggling and more chocolate bars I’ll always pick the former, no matter how much the quantities are (ω·(n+1) > ω·n+c, for any c). A minute of snuggling is better than all the chocolate bars in the world.
This also lets us say that paperclips do have nonzero value, but there is no amount of paperclips that is as valuable as the survival of humanity. If we program this into an AI, it will know that it can’t maximize value by maximizing paperclips, even if it’s much easier to produce a lot of paperclips than to save humanity.
Edited to add: This might even let us shoehorn deontological rules into a utility-based system. To give an obviously simplified example, consider Asimov’s Three Laws of Robotics, which come with explicit rank ordering: the First Law is supposed to always trump the Second, which is supposed to always trump the third. There’s not supposed to be any amount of Second Law value (obedience to humans) that can be greater than First Law value (protecting humans).
The problem with using hyperreals for utility is that unless you also use them for probabilities only the most infinite utilities actually affect your decision.
To use your example if U(snuggle) = ω and U(chocolate bar) = 1. Then you might as well say that U(snuggle) = 1 and U(chocolate bar) = 0 since tiny probabilities of getting a snuggle will always override any considerations related to chocolate bars.
How does this scenario operate under the assumption that humans do not have real-valued utility functions but rather utility orderings? IOW, we can’t arrange all world-states on a number line, but we can always say if one world-state is as good as (or better than) another.
This allows us to deal with infinities, such as “I wouldn’t kill my baby for anything.” That is: There doesn’t exist an N such that U(1) · N > U(B). That simply can’t be true on the (positive) reals; for any A and B real, there’s always a C such that A · C > B.
On any denumerable set with a total ordering on it, we can construct a map into the real numbers that preserves the ordering: Map the first element to 0, the second to 1 if it’s better and −1 if it’s worse, and put each additional one at the end or beginning of the line if it’s better or worse than all, or else into the exact middle of the interval that it falls into.
If you don’t like the denumerability requirement (who knows, the universe accessible to us might eventually come to be infinite, and then there would be more than denumerably many states of the universe), you can also take a utility function you already have, and then add a state that’s better than all others, while preserving the rest of the ordering: Assign to each state from our previous utility function the value that is the arctan of its previous value (the arctan 1-to-1-maps the real numbers onto the numbers between -pi/2 and pi/2 and preserves ordering), then give the new state utility 10.
I don’t know how you will deal with infinities and real humans. It’s quite trivial to construct scenarios under which the person making this statement would change her mind.
Real-valued utility functions can only deal with agents among whom “everybody has their price” — utilities are fungible and all are of the same order. That may actually be the case in the real world, or it may not. But if we assume real-valued utilities, we can’t ask the question of whether it is the case or not, because with real-valued utilities it must be the case.
To pick another example, there could exist a suicidally depressed agent to whom no amount of utility will cause them to evaluate their life as worth living: there doesn’t exist an N such that N + L > 0. Can’t happen with reals. The only way to make this agent become nonsuicidal is to modify the agent, not to drop a bunch of utils on their doorstep.
I am not arguing for real-valued utility functions. I am just pointing out that the “deal with infinities” claim looks suspect to me.
Well, I’m no mathematician, but I was thinking of something like ordinal arithmetic.
If I understand it correctly, this would let us express value-systems such as —
Both snuggles and chocolate bars have positive utility, but I’d always rather have another snuggle than any number of chocolate bars. So we could say U(snuggle) = ω and U(chocolate bar) = 1. For any amount of snuggling, I’d prefer to have that amount and a chocolate bar (ω·n+1 > ω·n), but given the choice between more snuggling and more chocolate bars I’ll always pick the former, no matter how much the quantities are (ω·(n+1) > ω·n+c, for any c). A minute of snuggling is better than all the chocolate bars in the world.
This also lets us say that paperclips do have nonzero value, but there is no amount of paperclips that is as valuable as the survival of humanity. If we program this into an AI, it will know that it can’t maximize value by maximizing paperclips, even if it’s much easier to produce a lot of paperclips than to save humanity.
Edited to add: This might even let us shoehorn deontological rules into a utility-based system. To give an obviously simplified example, consider Asimov’s Three Laws of Robotics, which come with explicit rank ordering: the First Law is supposed to always trump the Second, which is supposed to always trump the third. There’s not supposed to be any amount of Second Law value (obedience to humans) that can be greater than First Law value (protecting humans).
The problem with using hyperreals for utility is that unless you also use them for probabilities only the most infinite utilities actually affect your decision.
To use your example if U(snuggle) = ω and U(chocolate bar) = 1. Then you might as well say that U(snuggle) = 1 and U(chocolate bar) = 0 since tiny probabilities of getting a snuggle will always override any considerations related to chocolate bars.