Ok. This is mathematically correct, except that bounded utility means that if U(Y) is too high, U(X) cannot have a double utility, which means that the behavior is not so weird anymore. So in this case my question is why Kaj suggests his proposal instead of using bounded utility. Bounded utility will preserve the thing he seems to be mainly interested in, namely not accepting bets with extremely low probabilities, at least under normal circumstances, and it can preserve the order of our preferences (because even if utility is bounded, there are an infinite number of possible values for a utility.)
But Kaj’s method will also lead to the Allais paradox and the like, which won’t happen with bounded utility. This seems like undesirable behavior, so unless there is some additional reason why this is better than bounded utility, I don’t see why it would be a good proposal.
So in this case my question is why Kaj suggests his proposal instead of using bounded utility.
Two reasons.
First, like was mentioned elsewhere in the thread, bounded utility seems to produce unwanted effects, like we want utility to be linear in human lives and bounded utility seems to fail that.
Second, the way I arrived at this proposal was that RyanCarey asked me what’s my approach for dealing with Pascal’s Mugging. I replied that I just ignore probabilities that are small enough, which seems to be thing that most people do in practice. He objected that that seemed rather ad-hoc and wanted to have a more principled approach, so I started thinking about why exactly it would make sense to ignore sufficiently small probabilities, and came up with this as a somewhat principled answer.
Admittedly, as a principled answer to which probabilities are actually small enough to ignore, this isn’t all that satisfying of an answer, since it still depends on a rather arbitrary parameter. But it still seemed to point to some hidden assumptions behind utility maximization as well as raising some very interesting questions about what it is that we actually care about.
First, like was mentioned elsewhere in the thread, bounded utility seems to produce unwanted effects, like we want utility to be linear in human lives and bounded utility seems to fail that.
This is not quite what happens. When you do UDT properly, the result is that the Tegmark level IV multiverse has finite capacity for human lives (when human lives are counted with 2^-{Kolomogorov complexity} weights, as they should). Therefore the “bare” utility function has some kind of diminishing returns but the “effective” utility function is roughly linear in human lives once you take their “measure of existence” into account.
I consider it highly likely that bounded utility is the correct solution.
I agree that bounded utility implies that utility is not linear in human lives or in other similar matters.
But I have two problems with saying that we should try to get this property. First of all, no one in real life actually acts like it is linear. That’s why we talk about scope insensitivity, because people don’t treat it as linear. That suggests that people’s real utility functions, insofar as there are such things, are bounded.
Second, I think it won’t be possible to have a logically coherent set of preferences if you do that (at least combined with your proposal), namely because you will lose the independence property.
I agree that, insofar as people have something like utility functions, those are probably bounded. But I don’t think that an AI’s utility function should have the same properties as my utility function, or for that matter the same properties as the utility function of any human. I wouldn’t want the AI to discount the well-being of me or my close ones simply because a billion other people are already doing pretty well.
Though ironically given my answer to your first point, I’m somewhat unconcerned by your second point, because humans probably don’t have coherent preferences either, and still seem to do fine. My hunch is that rather than trying to make your preferences perfectly coherent, one is better off making a system for detecting sets of circular trades and similar exploits as they happen, and then making local adjustments to fix that particular inconsistency.
I edited this comment to include the statement that “bounded utility means that if U(Y) is too high, U(X) cannot have a double utility etc.” But then it occurred to me that I should say something else, which I’m adding here because I don’t want to keep changing the comment.
Evand’s statement that “these behaviors are both fairly weird (except in the universe where there’s no possible construction for an outcome with double the utility of Y, or the universe where you can’t construct a sufficiently low probability for some reason)” implies a particular understanding of bounded utility.
For example, someone could say, “My utility is in lives saved, and goes up to 10,000,000,000.” In this way he would say that saving 7 billion lives has a utility of 7 billion, saving 9 billion lives has a utility of 9 billion, and so on. But since he is bounding his utility, he would say that saving 20 billion lives has a utility of 10 billion, and so with saving any other number of lives over 10 billion.
This is definitely weird behavior. But this is not what I am suggesting by bounded utility. Basically I am saying that someone might bound his utility at 10 billion, but keep his order of preferences so that e.g. he would always prefer saving more lives to saving less.
This of course leads to something that could be considered scope insensitivity: a person will prefer a chance of saving 10 billion lives to a chance ten times as small of saving 100 billion lives, rather than being indifferent. But basically according to Kaj’s post this is the behavior we were trying to get to in the first place, namely ignoring the smaller probability bet in certain circumstances. It does correspond to people’s behavior in real life, and it doesn’t have the “switching preferences” effect that Kaj’s method will have when you change probabilities.
Ok. This is mathematically correct, except that bounded utility means that if U(Y) is too high, U(X) cannot have a double utility, which means that the behavior is not so weird anymore. So in this case my question is why Kaj suggests his proposal instead of using bounded utility. Bounded utility will preserve the thing he seems to be mainly interested in, namely not accepting bets with extremely low probabilities, at least under normal circumstances, and it can preserve the order of our preferences (because even if utility is bounded, there are an infinite number of possible values for a utility.)
But Kaj’s method will also lead to the Allais paradox and the like, which won’t happen with bounded utility. This seems like undesirable behavior, so unless there is some additional reason why this is better than bounded utility, I don’t see why it would be a good proposal.
Two reasons.
First, like was mentioned elsewhere in the thread, bounded utility seems to produce unwanted effects, like we want utility to be linear in human lives and bounded utility seems to fail that.
Second, the way I arrived at this proposal was that RyanCarey asked me what’s my approach for dealing with Pascal’s Mugging. I replied that I just ignore probabilities that are small enough, which seems to be thing that most people do in practice. He objected that that seemed rather ad-hoc and wanted to have a more principled approach, so I started thinking about why exactly it would make sense to ignore sufficiently small probabilities, and came up with this as a somewhat principled answer.
Admittedly, as a principled answer to which probabilities are actually small enough to ignore, this isn’t all that satisfying of an answer, since it still depends on a rather arbitrary parameter. But it still seemed to point to some hidden assumptions behind utility maximization as well as raising some very interesting questions about what it is that we actually care about.
This is not quite what happens. When you do UDT properly, the result is that the Tegmark level IV multiverse has finite capacity for human lives (when human lives are counted with 2^-{Kolomogorov complexity} weights, as they should). Therefore the “bare” utility function has some kind of diminishing returns but the “effective” utility function is roughly linear in human lives once you take their “measure of existence” into account.
I consider it highly likely that bounded utility is the correct solution.
I agree that bounded utility implies that utility is not linear in human lives or in other similar matters.
But I have two problems with saying that we should try to get this property. First of all, no one in real life actually acts like it is linear. That’s why we talk about scope insensitivity, because people don’t treat it as linear. That suggests that people’s real utility functions, insofar as there are such things, are bounded.
Second, I think it won’t be possible to have a logically coherent set of preferences if you do that (at least combined with your proposal), namely because you will lose the independence property.
I agree that, insofar as people have something like utility functions, those are probably bounded. But I don’t think that an AI’s utility function should have the same properties as my utility function, or for that matter the same properties as the utility function of any human. I wouldn’t want the AI to discount the well-being of me or my close ones simply because a billion other people are already doing pretty well.
Though ironically given my answer to your first point, I’m somewhat unconcerned by your second point, because humans probably don’t have coherent preferences either, and still seem to do fine. My hunch is that rather than trying to make your preferences perfectly coherent, one is better off making a system for detecting sets of circular trades and similar exploits as they happen, and then making local adjustments to fix that particular inconsistency.
I edited this comment to include the statement that “bounded utility means that if U(Y) is too high, U(X) cannot have a double utility etc.” But then it occurred to me that I should say something else, which I’m adding here because I don’t want to keep changing the comment.
Evand’s statement that “these behaviors are both fairly weird (except in the universe where there’s no possible construction for an outcome with double the utility of Y, or the universe where you can’t construct a sufficiently low probability for some reason)” implies a particular understanding of bounded utility.
For example, someone could say, “My utility is in lives saved, and goes up to 10,000,000,000.” In this way he would say that saving 7 billion lives has a utility of 7 billion, saving 9 billion lives has a utility of 9 billion, and so on. But since he is bounding his utility, he would say that saving 20 billion lives has a utility of 10 billion, and so with saving any other number of lives over 10 billion.
This is definitely weird behavior. But this is not what I am suggesting by bounded utility. Basically I am saying that someone might bound his utility at 10 billion, but keep his order of preferences so that e.g. he would always prefer saving more lives to saving less.
This of course leads to something that could be considered scope insensitivity: a person will prefer a chance of saving 10 billion lives to a chance ten times as small of saving 100 billion lives, rather than being indifferent. But basically according to Kaj’s post this is the behavior we were trying to get to in the first place, namely ignoring the smaller probability bet in certain circumstances. It does correspond to people’s behavior in real life, and it doesn’t have the “switching preferences” effect that Kaj’s method will have when you change probabilities.