This is just a complicated way of saying, “Let’s use bounded utility.” In other words, the fact that people don’t want to take deals where they will overall expect to get nothing out of it (in fact), means that they don’t value bets of that kind enough to take them. Which means they have bounded utility. Bounded utility is the correct reponse to PM.
If you don’t want to violate the independence axiom (which perhaps you did), then you will need bounded utility also when considering deals with non-PEST probabilities.
In any case, if you effectively give probability a lower bound, unbounded utility doesn’t have any specific meaning. The whole point of a double utility is that you will be willing to accept the double utility with half the probability. Once you won’t accept it with half the probability (as will happen in your situation) there is no point in saying that something has twice the utility.
It’s weird, but it’s not quite the same as bounded utility (though it looks pretty similar). In particular, there’s still a point in saying it has double the utility even though you sometimes won’t accept it at half the utility. Note the caveat “sometimes”: at other times, you will accept it.
Suppose event X has utility U(X) = 2 U(Y). Normally, you’ll accept it instead of Y at anything over half the probability. But if you reduce the probabilities of both* events enough, that changes. If you simply had a bound on utility, you would get a different behavior: you’d always accept X and over half the probability of Y for any P(Y), unless the utility of Y was too high. These behaviors are both fairly weird (except in the universe where there’s no possible construction of an outcome with double the utility of Y, or the universe where you can’t construct a sufficiently low probability for some reason), but they’re not the same.
Ok. This is mathematically correct, except that bounded utility means that if U(Y) is too high, U(X) cannot have a double utility, which means that the behavior is not so weird anymore. So in this case my question is why Kaj suggests his proposal instead of using bounded utility. Bounded utility will preserve the thing he seems to be mainly interested in, namely not accepting bets with extremely low probabilities, at least under normal circumstances, and it can preserve the order of our preferences (because even if utility is bounded, there are an infinite number of possible values for a utility.)
But Kaj’s method will also lead to the Allais paradox and the like, which won’t happen with bounded utility. This seems like undesirable behavior, so unless there is some additional reason why this is better than bounded utility, I don’t see why it would be a good proposal.
So in this case my question is why Kaj suggests his proposal instead of using bounded utility.
Two reasons.
First, like was mentioned elsewhere in the thread, bounded utility seems to produce unwanted effects, like we want utility to be linear in human lives and bounded utility seems to fail that.
Second, the way I arrived at this proposal was that RyanCarey asked me what’s my approach for dealing with Pascal’s Mugging. I replied that I just ignore probabilities that are small enough, which seems to be thing that most people do in practice. He objected that that seemed rather ad-hoc and wanted to have a more principled approach, so I started thinking about why exactly it would make sense to ignore sufficiently small probabilities, and came up with this as a somewhat principled answer.
Admittedly, as a principled answer to which probabilities are actually small enough to ignore, this isn’t all that satisfying of an answer, since it still depends on a rather arbitrary parameter. But it still seemed to point to some hidden assumptions behind utility maximization as well as raising some very interesting questions about what it is that we actually care about.
First, like was mentioned elsewhere in the thread, bounded utility seems to produce unwanted effects, like we want utility to be linear in human lives and bounded utility seems to fail that.
This is not quite what happens. When you do UDT properly, the result is that the Tegmark level IV multiverse has finite capacity for human lives (when human lives are counted with 2^-{Kolomogorov complexity} weights, as they should). Therefore the “bare” utility function has some kind of diminishing returns but the “effective” utility function is roughly linear in human lives once you take their “measure of existence” into account.
I consider it highly likely that bounded utility is the correct solution.
I agree that bounded utility implies that utility is not linear in human lives or in other similar matters.
But I have two problems with saying that we should try to get this property. First of all, no one in real life actually acts like it is linear. That’s why we talk about scope insensitivity, because people don’t treat it as linear. That suggests that people’s real utility functions, insofar as there are such things, are bounded.
Second, I think it won’t be possible to have a logically coherent set of preferences if you do that (at least combined with your proposal), namely because you will lose the independence property.
I agree that, insofar as people have something like utility functions, those are probably bounded. But I don’t think that an AI’s utility function should have the same properties as my utility function, or for that matter the same properties as the utility function of any human. I wouldn’t want the AI to discount the well-being of me or my close ones simply because a billion other people are already doing pretty well.
Though ironically given my answer to your first point, I’m somewhat unconcerned by your second point, because humans probably don’t have coherent preferences either, and still seem to do fine. My hunch is that rather than trying to make your preferences perfectly coherent, one is better off making a system for detecting sets of circular trades and similar exploits as they happen, and then making local adjustments to fix that particular inconsistency.
I edited this comment to include the statement that “bounded utility means that if U(Y) is too high, U(X) cannot have a double utility etc.” But then it occurred to me that I should say something else, which I’m adding here because I don’t want to keep changing the comment.
Evand’s statement that “these behaviors are both fairly weird (except in the universe where there’s no possible construction for an outcome with double the utility of Y, or the universe where you can’t construct a sufficiently low probability for some reason)” implies a particular understanding of bounded utility.
For example, someone could say, “My utility is in lives saved, and goes up to 10,000,000,000.” In this way he would say that saving 7 billion lives has a utility of 7 billion, saving 9 billion lives has a utility of 9 billion, and so on. But since he is bounding his utility, he would say that saving 20 billion lives has a utility of 10 billion, and so with saving any other number of lives over 10 billion.
This is definitely weird behavior. But this is not what I am suggesting by bounded utility. Basically I am saying that someone might bound his utility at 10 billion, but keep his order of preferences so that e.g. he would always prefer saving more lives to saving less.
This of course leads to something that could be considered scope insensitivity: a person will prefer a chance of saving 10 billion lives to a chance ten times as small of saving 100 billion lives, rather than being indifferent. But basically according to Kaj’s post this is the behavior we were trying to get to in the first place, namely ignoring the smaller probability bet in certain circumstances. It does correspond to people’s behavior in real life, and it doesn’t have the “switching preferences” effect that Kaj’s method will have when you change probabilities.
I think I agree that the OP does not follow independence, but everything else here seems wrong.
Actions A and B are identical except that A gives me 2 utils with .5 probability, while B gives me Graham’s number with .5 probability. I do B. (Likewise if there are ~Graham’s number of alternatives with intermediate payoffs.)
I’m not sure how you thought this was relevant to what I said.
What I was saying was this:
Suppose I say that A has utility 5, and B has utility 10. Basically the statement that B has twice the utility A has, has no particular meaning except that if I would like to have A at a probability of 10%, I would equally like to have B at a probability of 5%. If I would take the 10% chance and not the 5% chance, then there is no longer any meaning to saying that B has “double” the utility of A.
This does totally defy the intuitive understanding of expected utility. Intuitively, you can just set your utility function as whatever you want. If you want to maximize something like saving human lives, you can do that. As in one person dying is exactly half as bad as 2 people dying, which itself is exactly half as bad as 4 people dying, etc.
The justification for expected utility is that as the number of bets you take approach infinity, it becomes the optimal strategy. You save the most lives than you would with any other strategy. You lose some people on some bets, but gain many more on other bets.
But this justification and intuitive understanding totally breaks down in the real world. Where there are finite horizons. You don’t get to take an infinite number of bets. An agent following expected utility will just continuously bet away human lives on mugger-like bets, without ever gaining anything. It will always do worse than other strategies.
You can do some tricks to maybe fix this somewhat by modifying the utility function. But that seems wrong. Why are 2 lives not twice as valuable as 1 life. Why are 400 lives not twice as valuable as 200 lives? Will this change the decisions you make in everyday, non-muggle/excessively low probability bets? It seems like it would.
Or we could just keep the intuitive justification for expected utility, and generalize it to work on finite horizons. There are some proposals for methods that do this like mean of quantiles.
I don’t think that “the justification for expected utility is that as the number of bets you take approach infinity, it becomes the optimal strategy,” is quite true. Kaj did say something similar to this, but that seems to me a problem with his approach.
Basically, expected utility is supposed to give a mathematical formalization to people’s preferences. But consider this fact: in itself, it does not have any particular sense to say that “I like vanilla ice cream twice as much as chocolate.” It makes sense to say I like it more than chocolate. This means if I am given a choice between vanilla and chocolate, I will choose vanilla. But what on earth does it mean to say that I like it “twice” as much as chocolate? In itself, nothing. We have to define this in order to construct a mathematical analysis of our preferences.
In practice we make this definition by saying that I like vanilla so much that I am indifferent to having chocolate, or to having a 50% chance of having vanilla, and a 50% chance of having nothing.
Perhaps I justify this by saying that it will get me a certain amount of vanilla in my life. But perhaps I don’t—the definition does not justify the preference, it simply says what it means. This means that in order to say I like it twice as much, I have to say that I am indifferent to the 50% bet and to the certain chocolate, no matter what the justification for this might or might not be. If I change my preference when the number of cases goes down, then it will not be mathematically consistent to say that I like it twice as much as chocolate, unless we change the definition of “like it twice as much.”
Basically I think you are mixing up things like “lives”, which can be mathematically quantified in themselves, more or less, and people’s preferences, which only have a quantity if we define one.
It may be possible for Kaj to come up with a new definition for the amount of someone’s preference, but I suspect that it will result in a situation basically the same as keeping our definition, but admitting that people have only a limited amount of preference for things. In other words, they might prefer saving 100,000 lives to saving 10,000 lives, but they certainly do not prefer it 10 times as much, meaning they will not always accept the 100,000 lives saved at a 10% chance, compared to a 100% chance of saving 10,000.
But what on earth does it mean to say that I like it “twice” as much as chocolate?
Obviously it means you would be willing to trade 2 units of chocolate ice cream for 1 unit of vanilla. And over the course of your life, you would prefer to have more vanilla ice cream than chocolate ice cream. Perhaps before you die, you will add up all the ice creams you’ve ever eaten. And you would prefer for that number to be higher rather than lower.
Nowhere in the above description did I talk about probability. And the utility function is already completely defined. I just need to decide on a decision procedure to maximize it.
Expected utility seems like a good choice, because, over the course of my life, different bets I make on ice cream should average themselves out, and I should do better than otherwise. But that might not be true if there are ice cream muggers. Which promise lots of ice cream in exchange for a down payment, but usually lie.
So trying to convince the ice cream maximizer to follow expected utility is a lost cause. They will just end up losing all their ice cream to muggers. They need a system which ignores muggers.
This is definitely not what I mean if I say I like vanilla twice as much as chocolate. I might like it twice as much even though there is no chance that I can ever eat more than one serving of ice cream. If I have the choice of a small serving of vanilla or a triple serving of chocolate, I might still choose the vanilla. That does not mean I like it three times as much.
It is not about “How much ice cream.” It is about “how much wanting”.
I’m saying that the experience of eating chocolate is objectively twice as valuable. Maybe there is a limit on how much ice cream you can eat at a single sitting. But you can still choose to give up eating vanilla today and tomorrow, in exchange for eating chocolate once.
Again, you are assuming there is a quantitative measure over eating chocolate and eating vanilla, and that this determines the measure of my utility. This is not necessarily true, since these are arbitrary examples. I can still value one twice as much as the other, even if they both are experiences that can happen only once in a lifetime, or even only once in the lifetime of the universe.
Sure. But there is still some internal, objective measure of value of experiences. The constraints you add make it harder to determine what they are. But in simple cases, like trading ice cream, it’s easy to determine how much value a person has for a thing.
This has nothing to do with bounded utility. Bounded utility means you don’t care about any utilities above a certain large amount. Like if you care about saving lives, and you save 1,000 lives, after that you just stop caring. No amount of lives after that matters at all.
This solution allows for unbounded utility. Because you can always care about saving more lives. You just won’t take bets that could save huge numbers of lives, but have very very small probabilities.
This isn’t what I meant by bounded utility. I explained that in another comment. It refers to utility as a real number and simply sets a limit on that number. It does not mean that at any point “you just stop caring.”
If your utility has a limit, then you can’t care about anything past that limit. Even a continuous limit doesn’t work, because you care less and less about obtaining more utility, as you get closer to it. You would take a 50% chance at saving 2 people the same as a guaranteed chance at saving 1 person. But not a 50% chance at saving 2,000 people, over a chance at saving 1,000.
Yes, that would be the effect in general, that you would be less willing to take chances when the numbers involved are higher. That’s why you wouldn’t get mugged.
But that still doesn’t mean that “you don’t care.” You still prefer saving 2,000 lives to saving 1,000, whenever the chances are equal; your preference for the two cases does not suddenly become equal, as you originally said.
If utility is strictly bounded, then you do literally not care about saving 1,000 lives or 2,000.
You can fix that with asymptote. Then you do have a preference for 2,000. But the preference is only very slight. You wouldn’t take a 1% risk of losing 1,000 people, to save 2,000 people otherwise. Even though the risk is very small and the gain is very huge.
So it does fix Pascal’s mugging, but causes a whole new class of issues.
Your understanding of “strictly bounded” is artificial, and not what I was talking about. I was talking about assigning a strict, numerical bound to utility. That does not prevent having an infinite number of values underneath that bound.
It would be silly to assign a bound and a function low enough that “You wouldn’t take a 1% risk of losing 1,000 people, to save 2,000 people otherwise,” if you meant this literally, with these values.
But it is easy enough to assign a bound and a function that result in the choices we actually make in terms of real world values. It is true that if you increase the values enough, something like that will happen. And that is exactly the way real people would behave, as well.
Your understanding of “strictly bounded” is artificial, and not what I was talking about. I was talking about assigning a strict, numerical bound to utility. That does not prevent having an infinite number of values underneath that bound.
Isn’t that the same as an asymptote, which I talked about?
It would be silly to assign a bound and a function low enough that “You wouldn’t take a 1% risk of losing 1,000 people, to save 2,000 people otherwise,” if you meant this literally, with these values.
You can set the bound wherever you want. It’s arbitrary. My point is that if you ever reach it, you start behaving weird. It is not a very natural fix. It creates other issues.
It is true that if you increase the values enough, something like that will happen. And that is exactly the way real people would behave, as well.
Maybe human utility functions are bounded. Maybe they aren’t. We don’t know for sure. Assuming they are is a big risk. And even if they are bounded, it doesn’t mean we should put that into an AI. If, somehow, it ever runs into a situation where it can help 3^^^3 people, it really should.
You were speaking about bounded utility functions. Not bounded probability functions.
The whole point of the Pascal’s mugger scenario is that these scenarios aren’t impossible. Solomonoff induction halves the probability of each hypothesis based on how many additional bits it takes to describe. This means the probability of different models decreases fairly rapidly. But not as rapidly as functions like 3^^^3 grow. So there are hypotheses that describe things that are 3^^^3 units large in much fewer than log(3^^^3) bits.
So the utility of hypotheses can grow much faster than their probability shrinks.
If you think this is something with a reasonable probability, you should accept the mugging.
Well the probability isn’t reasonable. It’s just not as unreasonably small as 3^^^3 is big.
But yes you could bite the bullet and say that the expected utility is so big, it doesn’t matter what the probability is, and pay the mugger.
The problem is, expected utility doesn’t even converge. There is a hypothesis that paying the mugger saves 3^^^3 lives. And there’s an even more unlikely hypothesis that not paying him will save 3^^^^3 lives. And an even more complicated hypothesis that he will really save 3^^^^^3 lives. Etc. The expected utility of every action grows to infinity, and never converges on any finite value. More and more unlikely hypotheses totally dominate the calculation.
Solomonoff induction halves the probability of each hypothesis based on how many additional bits it takes to describe.
See, I told everyone that people here say this.
Fake muggings with large numbers are more profitable to the mugger than fake muggings with small numbers because the fake mugging with the larger number is more likely to convince a naive rationalist. And the profitability depends on the size of the number, not the number of bits in the number. Which makes the likelihood of a large number being fake grow faster than the number of bits in the number.
You are solving the specific problem of the mugger, and not the general problem of tiny bets with huge rewards.
Regardless, there’s no way the probability decreases faster than the reward the mugger promises. I don’t think you can assign 1/3^^^3 probability to anything. That’s an unfathomably small probability. You are literally saying there is no amount of evidence the mugger could give you to convince you otherwise. Even if he showed you his matrix powers, and the computer simulation of 3^^^3 people, you still wouldn’t believe him.
You probably couldn’t verify it. There’s always the possibility that any evidence you see is made up. For all you know you are just in a computer simulation and the entire thing is virtual.
I’m just saying he can show you evidence which increases the probability. Show you the racks of servers, show you the computer system, explain the physics that allows it, lets you do the experiments that shows those physics are correct. You could solve any NP complete problem on the computer. And you could run programs that take known numbers of steps to compute. Like actually calculating 3^^^3, etc.
Sure. But I think there are generally going to be more parsimonious explanations than any that involve him having the power to torture 3^^^3 people, let alone having that power and caring about whether I give him some money.
Parsimonious, sure. The possibility is very unlikely. But it doesn’t just need to be “very unlikely”, it needs to have smaller than 1/3^^^3 probability.
Sure. But if you have an argument that some guy who shows me apparent magical powers has the power to torture 3^^^3 people with probability substantially over 1/3^^^3, then I bet I can turn it into an argument that anyone, with or without a demonstration of magical powers, with or without any sort of claim that they have such powers, has the power to torture 3^^^3 people with probability nearly as substantially over 1/3^^^3. Because surely for anyone under any circumstances, Pr(I experience what seems to be a convincing demonstration that they have such powers) is much larger than 1/3^^^3, whether they actually have such powers or not.
Sure. But if you have an argument that some guy who shows me apparent magical powers has the power to torture 3^^^3 people with probability substantially over 1/3^^^3, then I bet I can turn it into an argument that anyone, with or without a demonstration of magical powers, with or without any sort of claim that they have such powers, has the power to torture 3^^^3 people with probability nearly as substantially over 1/3^^^3.
Correct. That still doesn’t solve the decision theory problem, it makes it worse. Since you have to take into account the possibility that anyone you meet might have the power to torture (or reward with utopia) 3^^^3 people.
It makes it worse or better, depending on whether you decide (1) that everyone has the power to do that with probability >~ 1/3^^^3 or (2) that no one has. I think #2 rather than #1 is correct.
I don’t see what your point is. Yes that’s a small number. It’s not a feeling, that’s just math. If you are assigning things 1/3^^^3 probability, you are basically saying they are impossible and no amount of evidence could convince you otherwise.
You can do that and be perfectly consistent. If that’s your point I don’t disagree. You can’t argue about priors. We can only agree to disagree, if those are your true priors.
Just remember that reality could always say “WRONG!” and punish you for assigning 0 probability to something. If you don’t want to be wrong, don’t assign 1/3^^^3 probability to things you aren’t 99.9999...% sure absolutely can’t happen.
Basically, human beings do not have an actual prior probability distribution. This should be obvious, since it means assigning a numerical probability to every possible state of affairs. No human being has ever done this, or ever will.
But you have something like a prior, but you build the prior itself based on your experience. At the moment we don’t have a specific number for the probability of the mugging situation coming up, but just think it’s very improbable, so that we don’t expect any evidence to ever come up that would convince us. But if the mugger shows matrix powers, we would change our prior so that the probability of the mugging situation was high enough to be convinced by being shown matrix powers.
You might say that means it was already that high, but it does not mean this, given the objective fact that people do not have real priors.
Maybe humans don’t really have probability distributions. But that doesn’t help us actually build an AI which reproduces the same result. If we had infinite computing power and could do ideal Solomonoff induction, it would pay the mugger.
Though I would argue that humans do have approximate probability functions and approximate priors. We wouldn’t be able to function in a probabilistic world if we didn’t. But it’s not relevant.
But if the mugger shows matrix powers, we would change our prior so that the probability of the mugging situation was high enough to be convinced by being shown matrix powers.
That’s just a regular bayesian probability update! You don’t need to change terminology and call it something different.
At the moment we don’t have a specific number for the probability of the mugging situation coming up, but just think it’s very improbable, so that we don’t expect any evidence to ever come up that would convince us.
That’s fine. I too think the situation is extraordinarily implausible. Even Solomonoff induction would agree with us. The probability that the mugger is real would be something like 1/10^100. Or perhaps the exponent should be orders of magnitude larger than that. That’s small enough that it shouldn’t even remotely register as a plausible hypothesis in your mind. But big enough some amount of evidence could convince you.
You don’t need to posit new models of how probability theory should work. Regular probability works fine at assigning really implausible hypotheses really low probability.
So there are hypotheses that describe things that are 3^^^3 units large in much fewer than log(3^^^3) bits.
So the utility of hypotheses can grow much faster than their probability shrinks.
If utility is straightforwardly additive, yes. But perhaps it isn’t. Imagine two possible worlds. In one, there are a billion copies of our planet and its population, all somehow leading exactly the same lives. In another, there are a billion planets like ours, with different people on them. Now someone proposes to blow up one of the planets. I find that I feel less awful about this in the first case than the second (though of course either is awful) because what’s being lost from the universe is something of which we have a billion copies anyway. If we stipulate that the destruction of the planet is instantaneous and painless, and that the people really are living exactly identical lives on each planet, then actually I’m not sure I care very much that one planet is gone. (But my feelings about this fluctuate.)
A world with 3^^^3 inhabitants that’s described by (say) no more than a billion bits seems a little like the first of those hypothetical worlds.
I’m not very sure about this. For instance, perhaps the description would take the form: “Seed a good random number generator as follows. [...] Now use it to generate 3^^^3 person-like agents in a deterministic universe with such-and-such laws. Now run it for 20 years.” and maybe you can get 3^^^3 genuinely non-redundant lives that way. But 3^^^3 is a very large number, and I’m not even quite sure there’s such a thing as 3^^^3 genuinely non-redundant lives even in principle.
I dunno. If I imagine a world with a billion identical copies of me living identical lives, having all of them tortured doesn’t seem a billion times worse than having one tortured. Would an AI’s experiences matter more if, to reduce the impact of hardware error, all its computations were performed on ten identical computers?
What if any of the Big World hypotheses are true? E.g. many worlds interpretation, multiverse theories, Tegmark’s hypothesis, or just a regular infinite universe. In that case anything that can exist does exist. There already are a billion versions of you being tortured. An infinite number actually. All you can ever really do is reduce the probability that you will find yourself in a good world or a bad one.
Bounded utility functions effectively give “bounded probability functions,” in the sense that you (more or less) stop caring about things with very low probability.
For example, if my maximum utility is 1,000, then my maximum utility for something with a probability of one in a billion is .0000001, an extremely small utiliity, so something that I will care about very little. The probability of of the 3^^^3 scenarios may be more than one in 3^^^3. But it will still be small enough that a bounded utility function won’t care about situations like that, at least not to any significant extent.
That is precisely the reason that it will do the things you object to, if that situation comes up.
That is no different from pointing out that the post’s proposal will reject a “mugging” even when it will actually cost 3^^^3 lives.
Both proposals have that particular downside. That is not something peculiar to mine.
Bounded utility functions mean you stop caring about things with very high utility. That you care less about certain low probability events is just a side effect. But those events can also have very high probability and you still don’t care.
If you want to just stop caring about really low probability events, why not just do that?
I just explained. There is no situation involving 3^^^3 people which will ever have a high probability. Telling me I need to adopt a utility function which will handle such situations well is trying to mug me, because such situations will never come up.
Also, I don’t care about the difference between 3^^^^^3 people and 3^^^^^^3 people even if the probability is 100%, and neither does anyone else. So it isn’t true that I just want to stop caring about low probability events. My utility is actually bounded. That’s why I suggest using a bounded utility function, like everyone else does.
There is no situation involving 3^^^3 people which will ever have a high probability.
Really? No situation? Not even if we discover new laws of physics that allow us to have infinite computing power?
Telling me I need to adopt a utility function which will handle such situations well is trying to mug me, because such situations will never come up.
We are talking about utility functions. Probability is irrelevant. All that matters for the utility function is that if the situation came up, you would care about it.
Also, I don’t care about the difference between 3^^^^^3 people and 3^^^^^^3 people even if the probability is 100%, and neither does anyone else.
I totally disagree with you. These numbers are so incomprehensibly huge you can’t picture them in your head, sure. There is massive scope insensitivity. But if you had to make moral choices that affect those two numbers of people, you should always value the bigger number proportionally more.
E.g. if you had to torture 3^^^^^3 to save 3^^^^^^3 from getting dust specks in their eyes. Or make bets involving probabilities between various things happening to the different groups. Etc. I don’t think you can make these decisions correctly if you have a bounded utility function.
If you don’t make them correctly, well that 3^^^3 people probably contains a basically infinite number of copies of you. By making the correct tradeoffs, you maximize the probability that the other versions of yoruself find themselves in a universe with higher utility.
This is just a complicated way of saying, “Let’s use bounded utility.” In other words, the fact that people don’t want to take deals where they will overall expect to get nothing out of it (in fact), means that they don’t value bets of that kind enough to take them. Which means they have bounded utility. Bounded utility is the correct reponse to PM.
But nothing about the approach implies that our utility functions would need to be bounded when considering deals involving non-PEST probabilities?
If you don’t want to violate the independence axiom (which perhaps you did), then you will need bounded utility also when considering deals with non-PEST probabilities.
In any case, if you effectively give probability a lower bound, unbounded utility doesn’t have any specific meaning. The whole point of a double utility is that you will be willing to accept the double utility with half the probability. Once you won’t accept it with half the probability (as will happen in your situation) there is no point in saying that something has twice the utility.
It’s weird, but it’s not quite the same as bounded utility (though it looks pretty similar). In particular, there’s still a point in saying it has double the utility even though you sometimes won’t accept it at half the utility. Note the caveat “sometimes”: at other times, you will accept it.
Suppose event X has utility U(X) = 2 U(Y). Normally, you’ll accept it instead of Y at anything over half the probability. But if you reduce the probabilities of both* events enough, that changes. If you simply had a bound on utility, you would get a different behavior: you’d always accept X and over half the probability of Y for any P(Y), unless the utility of Y was too high. These behaviors are both fairly weird (except in the universe where there’s no possible construction of an outcome with double the utility of Y, or the universe where you can’t construct a sufficiently low probability for some reason), but they’re not the same.
Ok. This is mathematically correct, except that bounded utility means that if U(Y) is too high, U(X) cannot have a double utility, which means that the behavior is not so weird anymore. So in this case my question is why Kaj suggests his proposal instead of using bounded utility. Bounded utility will preserve the thing he seems to be mainly interested in, namely not accepting bets with extremely low probabilities, at least under normal circumstances, and it can preserve the order of our preferences (because even if utility is bounded, there are an infinite number of possible values for a utility.)
But Kaj’s method will also lead to the Allais paradox and the like, which won’t happen with bounded utility. This seems like undesirable behavior, so unless there is some additional reason why this is better than bounded utility, I don’t see why it would be a good proposal.
Two reasons.
First, like was mentioned elsewhere in the thread, bounded utility seems to produce unwanted effects, like we want utility to be linear in human lives and bounded utility seems to fail that.
Second, the way I arrived at this proposal was that RyanCarey asked me what’s my approach for dealing with Pascal’s Mugging. I replied that I just ignore probabilities that are small enough, which seems to be thing that most people do in practice. He objected that that seemed rather ad-hoc and wanted to have a more principled approach, so I started thinking about why exactly it would make sense to ignore sufficiently small probabilities, and came up with this as a somewhat principled answer.
Admittedly, as a principled answer to which probabilities are actually small enough to ignore, this isn’t all that satisfying of an answer, since it still depends on a rather arbitrary parameter. But it still seemed to point to some hidden assumptions behind utility maximization as well as raising some very interesting questions about what it is that we actually care about.
This is not quite what happens. When you do UDT properly, the result is that the Tegmark level IV multiverse has finite capacity for human lives (when human lives are counted with 2^-{Kolomogorov complexity} weights, as they should). Therefore the “bare” utility function has some kind of diminishing returns but the “effective” utility function is roughly linear in human lives once you take their “measure of existence” into account.
I consider it highly likely that bounded utility is the correct solution.
I agree that bounded utility implies that utility is not linear in human lives or in other similar matters.
But I have two problems with saying that we should try to get this property. First of all, no one in real life actually acts like it is linear. That’s why we talk about scope insensitivity, because people don’t treat it as linear. That suggests that people’s real utility functions, insofar as there are such things, are bounded.
Second, I think it won’t be possible to have a logically coherent set of preferences if you do that (at least combined with your proposal), namely because you will lose the independence property.
I agree that, insofar as people have something like utility functions, those are probably bounded. But I don’t think that an AI’s utility function should have the same properties as my utility function, or for that matter the same properties as the utility function of any human. I wouldn’t want the AI to discount the well-being of me or my close ones simply because a billion other people are already doing pretty well.
Though ironically given my answer to your first point, I’m somewhat unconcerned by your second point, because humans probably don’t have coherent preferences either, and still seem to do fine. My hunch is that rather than trying to make your preferences perfectly coherent, one is better off making a system for detecting sets of circular trades and similar exploits as they happen, and then making local adjustments to fix that particular inconsistency.
I edited this comment to include the statement that “bounded utility means that if U(Y) is too high, U(X) cannot have a double utility etc.” But then it occurred to me that I should say something else, which I’m adding here because I don’t want to keep changing the comment.
Evand’s statement that “these behaviors are both fairly weird (except in the universe where there’s no possible construction for an outcome with double the utility of Y, or the universe where you can’t construct a sufficiently low probability for some reason)” implies a particular understanding of bounded utility.
For example, someone could say, “My utility is in lives saved, and goes up to 10,000,000,000.” In this way he would say that saving 7 billion lives has a utility of 7 billion, saving 9 billion lives has a utility of 9 billion, and so on. But since he is bounding his utility, he would say that saving 20 billion lives has a utility of 10 billion, and so with saving any other number of lives over 10 billion.
This is definitely weird behavior. But this is not what I am suggesting by bounded utility. Basically I am saying that someone might bound his utility at 10 billion, but keep his order of preferences so that e.g. he would always prefer saving more lives to saving less.
This of course leads to something that could be considered scope insensitivity: a person will prefer a chance of saving 10 billion lives to a chance ten times as small of saving 100 billion lives, rather than being indifferent. But basically according to Kaj’s post this is the behavior we were trying to get to in the first place, namely ignoring the smaller probability bet in certain circumstances. It does correspond to people’s behavior in real life, and it doesn’t have the “switching preferences” effect that Kaj’s method will have when you change probabilities.
I think I agree that the OP does not follow independence, but everything else here seems wrong.
Actions A and B are identical except that A gives me 2 utils with .5 probability, while B gives me Graham’s number with .5 probability. I do B. (Likewise if there are ~Graham’s number of alternatives with intermediate payoffs.)
I’m not sure how you thought this was relevant to what I said.
What I was saying was this:
Suppose I say that A has utility 5, and B has utility 10. Basically the statement that B has twice the utility A has, has no particular meaning except that if I would like to have A at a probability of 10%, I would equally like to have B at a probability of 5%. If I would take the 10% chance and not the 5% chance, then there is no longer any meaning to saying that B has “double” the utility of A.
This does totally defy the intuitive understanding of expected utility. Intuitively, you can just set your utility function as whatever you want. If you want to maximize something like saving human lives, you can do that. As in one person dying is exactly half as bad as 2 people dying, which itself is exactly half as bad as 4 people dying, etc.
The justification for expected utility is that as the number of bets you take approach infinity, it becomes the optimal strategy. You save the most lives than you would with any other strategy. You lose some people on some bets, but gain many more on other bets.
But this justification and intuitive understanding totally breaks down in the real world. Where there are finite horizons. You don’t get to take an infinite number of bets. An agent following expected utility will just continuously bet away human lives on mugger-like bets, without ever gaining anything. It will always do worse than other strategies.
You can do some tricks to maybe fix this somewhat by modifying the utility function. But that seems wrong. Why are 2 lives not twice as valuable as 1 life. Why are 400 lives not twice as valuable as 200 lives? Will this change the decisions you make in everyday, non-muggle/excessively low probability bets? It seems like it would.
Or we could just keep the intuitive justification for expected utility, and generalize it to work on finite horizons. There are some proposals for methods that do this like mean of quantiles.
I don’t think that “the justification for expected utility is that as the number of bets you take approach infinity, it becomes the optimal strategy,” is quite true. Kaj did say something similar to this, but that seems to me a problem with his approach.
Basically, expected utility is supposed to give a mathematical formalization to people’s preferences. But consider this fact: in itself, it does not have any particular sense to say that “I like vanilla ice cream twice as much as chocolate.” It makes sense to say I like it more than chocolate. This means if I am given a choice between vanilla and chocolate, I will choose vanilla. But what on earth does it mean to say that I like it “twice” as much as chocolate? In itself, nothing. We have to define this in order to construct a mathematical analysis of our preferences.
In practice we make this definition by saying that I like vanilla so much that I am indifferent to having chocolate, or to having a 50% chance of having vanilla, and a 50% chance of having nothing.
Perhaps I justify this by saying that it will get me a certain amount of vanilla in my life. But perhaps I don’t—the definition does not justify the preference, it simply says what it means. This means that in order to say I like it twice as much, I have to say that I am indifferent to the 50% bet and to the certain chocolate, no matter what the justification for this might or might not be. If I change my preference when the number of cases goes down, then it will not be mathematically consistent to say that I like it twice as much as chocolate, unless we change the definition of “like it twice as much.”
Basically I think you are mixing up things like “lives”, which can be mathematically quantified in themselves, more or less, and people’s preferences, which only have a quantity if we define one.
It may be possible for Kaj to come up with a new definition for the amount of someone’s preference, but I suspect that it will result in a situation basically the same as keeping our definition, but admitting that people have only a limited amount of preference for things. In other words, they might prefer saving 100,000 lives to saving 10,000 lives, but they certainly do not prefer it 10 times as much, meaning they will not always accept the 100,000 lives saved at a 10% chance, compared to a 100% chance of saving 10,000.
Obviously it means you would be willing to trade 2 units of chocolate ice cream for 1 unit of vanilla. And over the course of your life, you would prefer to have more vanilla ice cream than chocolate ice cream. Perhaps before you die, you will add up all the ice creams you’ve ever eaten. And you would prefer for that number to be higher rather than lower.
Nowhere in the above description did I talk about probability. And the utility function is already completely defined. I just need to decide on a decision procedure to maximize it.
Expected utility seems like a good choice, because, over the course of my life, different bets I make on ice cream should average themselves out, and I should do better than otherwise. But that might not be true if there are ice cream muggers. Which promise lots of ice cream in exchange for a down payment, but usually lie.
So trying to convince the ice cream maximizer to follow expected utility is a lost cause. They will just end up losing all their ice cream to muggers. They need a system which ignores muggers.
This is definitely not what I mean if I say I like vanilla twice as much as chocolate. I might like it twice as much even though there is no chance that I can ever eat more than one serving of ice cream. If I have the choice of a small serving of vanilla or a triple serving of chocolate, I might still choose the vanilla. That does not mean I like it three times as much.
It is not about “How much ice cream.” It is about “how much wanting”.
I’m saying that the experience of eating chocolate is objectively twice as valuable. Maybe there is a limit on how much ice cream you can eat at a single sitting. But you can still choose to give up eating vanilla today and tomorrow, in exchange for eating chocolate once.
Again, you are assuming there is a quantitative measure over eating chocolate and eating vanilla, and that this determines the measure of my utility. This is not necessarily true, since these are arbitrary examples. I can still value one twice as much as the other, even if they both are experiences that can happen only once in a lifetime, or even only once in the lifetime of the universe.
Sure. But there is still some internal, objective measure of value of experiences. The constraints you add make it harder to determine what they are. But in simple cases, like trading ice cream, it’s easy to determine how much value a person has for a thing.
This has nothing to do with bounded utility. Bounded utility means you don’t care about any utilities above a certain large amount. Like if you care about saving lives, and you save 1,000 lives, after that you just stop caring. No amount of lives after that matters at all.
This solution allows for unbounded utility. Because you can always care about saving more lives. You just won’t take bets that could save huge numbers of lives, but have very very small probabilities.
This isn’t what I meant by bounded utility. I explained that in another comment. It refers to utility as a real number and simply sets a limit on that number. It does not mean that at any point “you just stop caring.”
If your utility has a limit, then you can’t care about anything past that limit. Even a continuous limit doesn’t work, because you care less and less about obtaining more utility, as you get closer to it. You would take a 50% chance at saving 2 people the same as a guaranteed chance at saving 1 person. But not a 50% chance at saving 2,000 people, over a chance at saving 1,000.
Yes, that would be the effect in general, that you would be less willing to take chances when the numbers involved are higher. That’s why you wouldn’t get mugged.
But that still doesn’t mean that “you don’t care.” You still prefer saving 2,000 lives to saving 1,000, whenever the chances are equal; your preference for the two cases does not suddenly become equal, as you originally said.
If utility is strictly bounded, then you do literally not care about saving 1,000 lives or 2,000.
You can fix that with asymptote. Then you do have a preference for 2,000. But the preference is only very slight. You wouldn’t take a 1% risk of losing 1,000 people, to save 2,000 people otherwise. Even though the risk is very small and the gain is very huge.
So it does fix Pascal’s mugging, but causes a whole new class of issues.
Your understanding of “strictly bounded” is artificial, and not what I was talking about. I was talking about assigning a strict, numerical bound to utility. That does not prevent having an infinite number of values underneath that bound.
It would be silly to assign a bound and a function low enough that “You wouldn’t take a 1% risk of losing 1,000 people, to save 2,000 people otherwise,” if you meant this literally, with these values.
But it is easy enough to assign a bound and a function that result in the choices we actually make in terms of real world values. It is true that if you increase the values enough, something like that will happen. And that is exactly the way real people would behave, as well.
Isn’t that the same as an asymptote, which I talked about?
You can set the bound wherever you want. It’s arbitrary. My point is that if you ever reach it, you start behaving weird. It is not a very natural fix. It creates other issues.
Maybe human utility functions are bounded. Maybe they aren’t. We don’t know for sure. Assuming they are is a big risk. And even if they are bounded, it doesn’t mean we should put that into an AI. If, somehow, it ever runs into a situation where it can help 3^^^3 people, it really should.
“If, somehow, it ever runs into a situation where it can help 3^^^3 people, it really should.”
I thought the whole idea behind this proposal was that the probability of this happening is essentially zero.
If you think this is something with a reasonable probability, you should accept the mugging.
You were speaking about bounded utility functions. Not bounded probability functions.
The whole point of the Pascal’s mugger scenario is that these scenarios aren’t impossible. Solomonoff induction halves the probability of each hypothesis based on how many additional bits it takes to describe. This means the probability of different models decreases fairly rapidly. But not as rapidly as functions like 3^^^3 grow. So there are hypotheses that describe things that are 3^^^3 units large in much fewer than log(3^^^3) bits.
So the utility of hypotheses can grow much faster than their probability shrinks.
Well the probability isn’t reasonable. It’s just not as unreasonably small as 3^^^3 is big.
But yes you could bite the bullet and say that the expected utility is so big, it doesn’t matter what the probability is, and pay the mugger.
The problem is, expected utility doesn’t even converge. There is a hypothesis that paying the mugger saves 3^^^3 lives. And there’s an even more unlikely hypothesis that not paying him will save 3^^^^3 lives. And an even more complicated hypothesis that he will really save 3^^^^^3 lives. Etc. The expected utility of every action grows to infinity, and never converges on any finite value. More and more unlikely hypotheses totally dominate the calculation.
See, I told everyone that people here say this.
Fake muggings with large numbers are more profitable to the mugger than fake muggings with small numbers because the fake mugging with the larger number is more likely to convince a naive rationalist. And the profitability depends on the size of the number, not the number of bits in the number. Which makes the likelihood of a large number being fake grow faster than the number of bits in the number.
You are solving the specific problem of the mugger, and not the general problem of tiny bets with huge rewards.
Regardless, there’s no way the probability decreases faster than the reward the mugger promises. I don’t think you can assign 1/3^^^3 probability to anything. That’s an unfathomably small probability. You are literally saying there is no amount of evidence the mugger could give you to convince you otherwise. Even if he showed you his matrix powers, and the computer simulation of 3^^^3 people, you still wouldn’t believe him.
How could he show you “the computer simulation of 3^^^3 people”? What could you do to verify that 3^^^3 people were really being simulated?
You probably couldn’t verify it. There’s always the possibility that any evidence you see is made up. For all you know you are just in a computer simulation and the entire thing is virtual.
I’m just saying he can show you evidence which increases the probability. Show you the racks of servers, show you the computer system, explain the physics that allows it, lets you do the experiments that shows those physics are correct. You could solve any NP complete problem on the computer. And you could run programs that take known numbers of steps to compute. Like actually calculating 3^^^3, etc.
Sure. But I think there are generally going to be more parsimonious explanations than any that involve him having the power to torture 3^^^3 people, let alone having that power and caring about whether I give him some money.
Parsimonious, sure. The possibility is very unlikely. But it doesn’t just need to be “very unlikely”, it needs to have smaller than 1/3^^^3 probability.
Sure. But if you have an argument that some guy who shows me apparent magical powers has the power to torture 3^^^3 people with probability substantially over 1/3^^^3, then I bet I can turn it into an argument that anyone, with or without a demonstration of magical powers, with or without any sort of claim that they have such powers, has the power to torture 3^^^3 people with probability nearly as substantially over 1/3^^^3. Because surely for anyone under any circumstances, Pr(I experience what seems to be a convincing demonstration that they have such powers) is much larger than 1/3^^^3, whether they actually have such powers or not.
Correct. That still doesn’t solve the decision theory problem, it makes it worse. Since you have to take into account the possibility that anyone you meet might have the power to torture (or reward with utopia) 3^^^3 people.
It makes it worse or better, depending on whether you decide (1) that everyone has the power to do that with probability >~ 1/3^^^3 or (2) that no one has. I think #2 rather than #1 is correct.
Well, doing basic Bayes with a Kolmogorov priot gives you (1).
About as unfathomably small as the number of 3^^^3 people is unfathomably large?
I think you’re relying on “but I feel this can’t be right!” a bit too much.
I don’t see what your point is. Yes that’s a small number. It’s not a feeling, that’s just math. If you are assigning things 1/3^^^3 probability, you are basically saying they are impossible and no amount of evidence could convince you otherwise.
You can do that and be perfectly consistent. If that’s your point I don’t disagree. You can’t argue about priors. We can only agree to disagree, if those are your true priors.
Just remember that reality could always say “WRONG!” and punish you for assigning 0 probability to something. If you don’t want to be wrong, don’t assign 1/3^^^3 probability to things you aren’t 99.9999...% sure absolutely can’t happen.
Eliezer showed a problem that that reasoning in his post on Pascal’s Muggle.
Basically, human beings do not have an actual prior probability distribution. This should be obvious, since it means assigning a numerical probability to every possible state of affairs. No human being has ever done this, or ever will.
But you have something like a prior, but you build the prior itself based on your experience. At the moment we don’t have a specific number for the probability of the mugging situation coming up, but just think it’s very improbable, so that we don’t expect any evidence to ever come up that would convince us. But if the mugger shows matrix powers, we would change our prior so that the probability of the mugging situation was high enough to be convinced by being shown matrix powers.
You might say that means it was already that high, but it does not mean this, given the objective fact that people do not have real priors.
Maybe humans don’t really have probability distributions. But that doesn’t help us actually build an AI which reproduces the same result. If we had infinite computing power and could do ideal Solomonoff induction, it would pay the mugger.
Though I would argue that humans do have approximate probability functions and approximate priors. We wouldn’t be able to function in a probabilistic world if we didn’t. But it’s not relevant.
That’s just a regular bayesian probability update! You don’t need to change terminology and call it something different.
That’s fine. I too think the situation is extraordinarily implausible. Even Solomonoff induction would agree with us. The probability that the mugger is real would be something like 1/10^100. Or perhaps the exponent should be orders of magnitude larger than that. That’s small enough that it shouldn’t even remotely register as a plausible hypothesis in your mind. But big enough some amount of evidence could convince you.
You don’t need to posit new models of how probability theory should work. Regular probability works fine at assigning really implausible hypotheses really low probability.
But that is still way, way bigger than 1/3^^^3.
If utility is straightforwardly additive, yes. But perhaps it isn’t. Imagine two possible worlds. In one, there are a billion copies of our planet and its population, all somehow leading exactly the same lives. In another, there are a billion planets like ours, with different people on them. Now someone proposes to blow up one of the planets. I find that I feel less awful about this in the first case than the second (though of course either is awful) because what’s being lost from the universe is something of which we have a billion copies anyway. If we stipulate that the destruction of the planet is instantaneous and painless, and that the people really are living exactly identical lives on each planet, then actually I’m not sure I care very much that one planet is gone. (But my feelings about this fluctuate.)
A world with 3^^^3 inhabitants that’s described by (say) no more than a billion bits seems a little like the first of those hypothetical worlds.
I’m not very sure about this. For instance, perhaps the description would take the form: “Seed a good random number generator as follows. [...] Now use it to generate 3^^^3 person-like agents in a deterministic universe with such-and-such laws. Now run it for 20 years.” and maybe you can get 3^^^3 genuinely non-redundant lives that way. But 3^^^3 is a very large number, and I’m not even quite sure there’s such a thing as 3^^^3 genuinely non-redundant lives even in principle.
Well what if instead of killing them, he tortured them for an hour? Death might not matter in a Big World, but total suffering still does.
I dunno. If I imagine a world with a billion identical copies of me living identical lives, having all of them tortured doesn’t seem a billion times worse than having one tortured. Would an AI’s experiences matter more if, to reduce the impact of hardware error, all its computations were performed on ten identical computers?
What if any of the Big World hypotheses are true? E.g. many worlds interpretation, multiverse theories, Tegmark’s hypothesis, or just a regular infinite universe. In that case anything that can exist does exist. There already are a billion versions of you being tortured. An infinite number actually. All you can ever really do is reduce the probability that you will find yourself in a good world or a bad one.
Bounded utility functions effectively give “bounded probability functions,” in the sense that you (more or less) stop caring about things with very low probability.
For example, if my maximum utility is 1,000, then my maximum utility for something with a probability of one in a billion is .0000001, an extremely small utiliity, so something that I will care about very little. The probability of of the 3^^^3 scenarios may be more than one in 3^^^3. But it will still be small enough that a bounded utility function won’t care about situations like that, at least not to any significant extent.
That is precisely the reason that it will do the things you object to, if that situation comes up.
That is no different from pointing out that the post’s proposal will reject a “mugging” even when it will actually cost 3^^^3 lives.
Both proposals have that particular downside. That is not something peculiar to mine.
Bounded utility functions mean you stop caring about things with very high utility. That you care less about certain low probability events is just a side effect. But those events can also have very high probability and you still don’t care.
If you want to just stop caring about really low probability events, why not just do that?
I just explained. There is no situation involving 3^^^3 people which will ever have a high probability. Telling me I need to adopt a utility function which will handle such situations well is trying to mug me, because such situations will never come up.
Also, I don’t care about the difference between 3^^^^^3 people and 3^^^^^^3 people even if the probability is 100%, and neither does anyone else. So it isn’t true that I just want to stop caring about low probability events. My utility is actually bounded. That’s why I suggest using a bounded utility function, like everyone else does.
Really? No situation? Not even if we discover new laws of physics that allow us to have infinite computing power?
We are talking about utility functions. Probability is irrelevant. All that matters for the utility function is that if the situation came up, you would care about it.
I totally disagree with you. These numbers are so incomprehensibly huge you can’t picture them in your head, sure. There is massive scope insensitivity. But if you had to make moral choices that affect those two numbers of people, you should always value the bigger number proportionally more.
E.g. if you had to torture 3^^^^^3 to save 3^^^^^^3 from getting dust specks in their eyes. Or make bets involving probabilities between various things happening to the different groups. Etc. I don’t think you can make these decisions correctly if you have a bounded utility function.
If you don’t make them correctly, well that 3^^^3 people probably contains a basically infinite number of copies of you. By making the correct tradeoffs, you maximize the probability that the other versions of yoruself find themselves in a universe with higher utility.