Your understanding of “strictly bounded” is artificial, and not what I was talking about. I was talking about assigning a strict, numerical bound to utility. That does not prevent having an infinite number of values underneath that bound.
Isn’t that the same as an asymptote, which I talked about?
It would be silly to assign a bound and a function low enough that “You wouldn’t take a 1% risk of losing 1,000 people, to save 2,000 people otherwise,” if you meant this literally, with these values.
You can set the bound wherever you want. It’s arbitrary. My point is that if you ever reach it, you start behaving weird. It is not a very natural fix. It creates other issues.
It is true that if you increase the values enough, something like that will happen. And that is exactly the way real people would behave, as well.
Maybe human utility functions are bounded. Maybe they aren’t. We don’t know for sure. Assuming they are is a big risk. And even if they are bounded, it doesn’t mean we should put that into an AI. If, somehow, it ever runs into a situation where it can help 3^^^3 people, it really should.
You were speaking about bounded utility functions. Not bounded probability functions.
The whole point of the Pascal’s mugger scenario is that these scenarios aren’t impossible. Solomonoff induction halves the probability of each hypothesis based on how many additional bits it takes to describe. This means the probability of different models decreases fairly rapidly. But not as rapidly as functions like 3^^^3 grow. So there are hypotheses that describe things that are 3^^^3 units large in much fewer than log(3^^^3) bits.
So the utility of hypotheses can grow much faster than their probability shrinks.
If you think this is something with a reasonable probability, you should accept the mugging.
Well the probability isn’t reasonable. It’s just not as unreasonably small as 3^^^3 is big.
But yes you could bite the bullet and say that the expected utility is so big, it doesn’t matter what the probability is, and pay the mugger.
The problem is, expected utility doesn’t even converge. There is a hypothesis that paying the mugger saves 3^^^3 lives. And there’s an even more unlikely hypothesis that not paying him will save 3^^^^3 lives. And an even more complicated hypothesis that he will really save 3^^^^^3 lives. Etc. The expected utility of every action grows to infinity, and never converges on any finite value. More and more unlikely hypotheses totally dominate the calculation.
Solomonoff induction halves the probability of each hypothesis based on how many additional bits it takes to describe.
See, I told everyone that people here say this.
Fake muggings with large numbers are more profitable to the mugger than fake muggings with small numbers because the fake mugging with the larger number is more likely to convince a naive rationalist. And the profitability depends on the size of the number, not the number of bits in the number. Which makes the likelihood of a large number being fake grow faster than the number of bits in the number.
You are solving the specific problem of the mugger, and not the general problem of tiny bets with huge rewards.
Regardless, there’s no way the probability decreases faster than the reward the mugger promises. I don’t think you can assign 1/3^^^3 probability to anything. That’s an unfathomably small probability. You are literally saying there is no amount of evidence the mugger could give you to convince you otherwise. Even if he showed you his matrix powers, and the computer simulation of 3^^^3 people, you still wouldn’t believe him.
You probably couldn’t verify it. There’s always the possibility that any evidence you see is made up. For all you know you are just in a computer simulation and the entire thing is virtual.
I’m just saying he can show you evidence which increases the probability. Show you the racks of servers, show you the computer system, explain the physics that allows it, lets you do the experiments that shows those physics are correct. You could solve any NP complete problem on the computer. And you could run programs that take known numbers of steps to compute. Like actually calculating 3^^^3, etc.
Sure. But I think there are generally going to be more parsimonious explanations than any that involve him having the power to torture 3^^^3 people, let alone having that power and caring about whether I give him some money.
Parsimonious, sure. The possibility is very unlikely. But it doesn’t just need to be “very unlikely”, it needs to have smaller than 1/3^^^3 probability.
Sure. But if you have an argument that some guy who shows me apparent magical powers has the power to torture 3^^^3 people with probability substantially over 1/3^^^3, then I bet I can turn it into an argument that anyone, with or without a demonstration of magical powers, with or without any sort of claim that they have such powers, has the power to torture 3^^^3 people with probability nearly as substantially over 1/3^^^3. Because surely for anyone under any circumstances, Pr(I experience what seems to be a convincing demonstration that they have such powers) is much larger than 1/3^^^3, whether they actually have such powers or not.
Sure. But if you have an argument that some guy who shows me apparent magical powers has the power to torture 3^^^3 people with probability substantially over 1/3^^^3, then I bet I can turn it into an argument that anyone, with or without a demonstration of magical powers, with or without any sort of claim that they have such powers, has the power to torture 3^^^3 people with probability nearly as substantially over 1/3^^^3.
Correct. That still doesn’t solve the decision theory problem, it makes it worse. Since you have to take into account the possibility that anyone you meet might have the power to torture (or reward with utopia) 3^^^3 people.
It makes it worse or better, depending on whether you decide (1) that everyone has the power to do that with probability >~ 1/3^^^3 or (2) that no one has. I think #2 rather than #1 is correct.
I don’t see what your point is. Yes that’s a small number. It’s not a feeling, that’s just math. If you are assigning things 1/3^^^3 probability, you are basically saying they are impossible and no amount of evidence could convince you otherwise.
You can do that and be perfectly consistent. If that’s your point I don’t disagree. You can’t argue about priors. We can only agree to disagree, if those are your true priors.
Just remember that reality could always say “WRONG!” and punish you for assigning 0 probability to something. If you don’t want to be wrong, don’t assign 1/3^^^3 probability to things you aren’t 99.9999...% sure absolutely can’t happen.
Basically, human beings do not have an actual prior probability distribution. This should be obvious, since it means assigning a numerical probability to every possible state of affairs. No human being has ever done this, or ever will.
But you have something like a prior, but you build the prior itself based on your experience. At the moment we don’t have a specific number for the probability of the mugging situation coming up, but just think it’s very improbable, so that we don’t expect any evidence to ever come up that would convince us. But if the mugger shows matrix powers, we would change our prior so that the probability of the mugging situation was high enough to be convinced by being shown matrix powers.
You might say that means it was already that high, but it does not mean this, given the objective fact that people do not have real priors.
Maybe humans don’t really have probability distributions. But that doesn’t help us actually build an AI which reproduces the same result. If we had infinite computing power and could do ideal Solomonoff induction, it would pay the mugger.
Though I would argue that humans do have approximate probability functions and approximate priors. We wouldn’t be able to function in a probabilistic world if we didn’t. But it’s not relevant.
But if the mugger shows matrix powers, we would change our prior so that the probability of the mugging situation was high enough to be convinced by being shown matrix powers.
That’s just a regular bayesian probability update! You don’t need to change terminology and call it something different.
At the moment we don’t have a specific number for the probability of the mugging situation coming up, but just think it’s very improbable, so that we don’t expect any evidence to ever come up that would convince us.
That’s fine. I too think the situation is extraordinarily implausible. Even Solomonoff induction would agree with us. The probability that the mugger is real would be something like 1/10^100. Or perhaps the exponent should be orders of magnitude larger than that. That’s small enough that it shouldn’t even remotely register as a plausible hypothesis in your mind. But big enough some amount of evidence could convince you.
You don’t need to posit new models of how probability theory should work. Regular probability works fine at assigning really implausible hypotheses really low probability.
So there are hypotheses that describe things that are 3^^^3 units large in much fewer than log(3^^^3) bits.
So the utility of hypotheses can grow much faster than their probability shrinks.
If utility is straightforwardly additive, yes. But perhaps it isn’t. Imagine two possible worlds. In one, there are a billion copies of our planet and its population, all somehow leading exactly the same lives. In another, there are a billion planets like ours, with different people on them. Now someone proposes to blow up one of the planets. I find that I feel less awful about this in the first case than the second (though of course either is awful) because what’s being lost from the universe is something of which we have a billion copies anyway. If we stipulate that the destruction of the planet is instantaneous and painless, and that the people really are living exactly identical lives on each planet, then actually I’m not sure I care very much that one planet is gone. (But my feelings about this fluctuate.)
A world with 3^^^3 inhabitants that’s described by (say) no more than a billion bits seems a little like the first of those hypothetical worlds.
I’m not very sure about this. For instance, perhaps the description would take the form: “Seed a good random number generator as follows. [...] Now use it to generate 3^^^3 person-like agents in a deterministic universe with such-and-such laws. Now run it for 20 years.” and maybe you can get 3^^^3 genuinely non-redundant lives that way. But 3^^^3 is a very large number, and I’m not even quite sure there’s such a thing as 3^^^3 genuinely non-redundant lives even in principle.
I dunno. If I imagine a world with a billion identical copies of me living identical lives, having all of them tortured doesn’t seem a billion times worse than having one tortured. Would an AI’s experiences matter more if, to reduce the impact of hardware error, all its computations were performed on ten identical computers?
What if any of the Big World hypotheses are true? E.g. many worlds interpretation, multiverse theories, Tegmark’s hypothesis, or just a regular infinite universe. In that case anything that can exist does exist. There already are a billion versions of you being tortured. An infinite number actually. All you can ever really do is reduce the probability that you will find yourself in a good world or a bad one.
Bounded utility functions effectively give “bounded probability functions,” in the sense that you (more or less) stop caring about things with very low probability.
For example, if my maximum utility is 1,000, then my maximum utility for something with a probability of one in a billion is .0000001, an extremely small utiliity, so something that I will care about very little. The probability of of the 3^^^3 scenarios may be more than one in 3^^^3. But it will still be small enough that a bounded utility function won’t care about situations like that, at least not to any significant extent.
That is precisely the reason that it will do the things you object to, if that situation comes up.
That is no different from pointing out that the post’s proposal will reject a “mugging” even when it will actually cost 3^^^3 lives.
Both proposals have that particular downside. That is not something peculiar to mine.
Bounded utility functions mean you stop caring about things with very high utility. That you care less about certain low probability events is just a side effect. But those events can also have very high probability and you still don’t care.
If you want to just stop caring about really low probability events, why not just do that?
I just explained. There is no situation involving 3^^^3 people which will ever have a high probability. Telling me I need to adopt a utility function which will handle such situations well is trying to mug me, because such situations will never come up.
Also, I don’t care about the difference between 3^^^^^3 people and 3^^^^^^3 people even if the probability is 100%, and neither does anyone else. So it isn’t true that I just want to stop caring about low probability events. My utility is actually bounded. That’s why I suggest using a bounded utility function, like everyone else does.
There is no situation involving 3^^^3 people which will ever have a high probability.
Really? No situation? Not even if we discover new laws of physics that allow us to have infinite computing power?
Telling me I need to adopt a utility function which will handle such situations well is trying to mug me, because such situations will never come up.
We are talking about utility functions. Probability is irrelevant. All that matters for the utility function is that if the situation came up, you would care about it.
Also, I don’t care about the difference between 3^^^^^3 people and 3^^^^^^3 people even if the probability is 100%, and neither does anyone else.
I totally disagree with you. These numbers are so incomprehensibly huge you can’t picture them in your head, sure. There is massive scope insensitivity. But if you had to make moral choices that affect those two numbers of people, you should always value the bigger number proportionally more.
E.g. if you had to torture 3^^^^^3 to save 3^^^^^^3 from getting dust specks in their eyes. Or make bets involving probabilities between various things happening to the different groups. Etc. I don’t think you can make these decisions correctly if you have a bounded utility function.
If you don’t make them correctly, well that 3^^^3 people probably contains a basically infinite number of copies of you. By making the correct tradeoffs, you maximize the probability that the other versions of yoruself find themselves in a universe with higher utility.
Isn’t that the same as an asymptote, which I talked about?
You can set the bound wherever you want. It’s arbitrary. My point is that if you ever reach it, you start behaving weird. It is not a very natural fix. It creates other issues.
Maybe human utility functions are bounded. Maybe they aren’t. We don’t know for sure. Assuming they are is a big risk. And even if they are bounded, it doesn’t mean we should put that into an AI. If, somehow, it ever runs into a situation where it can help 3^^^3 people, it really should.
“If, somehow, it ever runs into a situation where it can help 3^^^3 people, it really should.”
I thought the whole idea behind this proposal was that the probability of this happening is essentially zero.
If you think this is something with a reasonable probability, you should accept the mugging.
You were speaking about bounded utility functions. Not bounded probability functions.
The whole point of the Pascal’s mugger scenario is that these scenarios aren’t impossible. Solomonoff induction halves the probability of each hypothesis based on how many additional bits it takes to describe. This means the probability of different models decreases fairly rapidly. But not as rapidly as functions like 3^^^3 grow. So there are hypotheses that describe things that are 3^^^3 units large in much fewer than log(3^^^3) bits.
So the utility of hypotheses can grow much faster than their probability shrinks.
Well the probability isn’t reasonable. It’s just not as unreasonably small as 3^^^3 is big.
But yes you could bite the bullet and say that the expected utility is so big, it doesn’t matter what the probability is, and pay the mugger.
The problem is, expected utility doesn’t even converge. There is a hypothesis that paying the mugger saves 3^^^3 lives. And there’s an even more unlikely hypothesis that not paying him will save 3^^^^3 lives. And an even more complicated hypothesis that he will really save 3^^^^^3 lives. Etc. The expected utility of every action grows to infinity, and never converges on any finite value. More and more unlikely hypotheses totally dominate the calculation.
See, I told everyone that people here say this.
Fake muggings with large numbers are more profitable to the mugger than fake muggings with small numbers because the fake mugging with the larger number is more likely to convince a naive rationalist. And the profitability depends on the size of the number, not the number of bits in the number. Which makes the likelihood of a large number being fake grow faster than the number of bits in the number.
You are solving the specific problem of the mugger, and not the general problem of tiny bets with huge rewards.
Regardless, there’s no way the probability decreases faster than the reward the mugger promises. I don’t think you can assign 1/3^^^3 probability to anything. That’s an unfathomably small probability. You are literally saying there is no amount of evidence the mugger could give you to convince you otherwise. Even if he showed you his matrix powers, and the computer simulation of 3^^^3 people, you still wouldn’t believe him.
How could he show you “the computer simulation of 3^^^3 people”? What could you do to verify that 3^^^3 people were really being simulated?
You probably couldn’t verify it. There’s always the possibility that any evidence you see is made up. For all you know you are just in a computer simulation and the entire thing is virtual.
I’m just saying he can show you evidence which increases the probability. Show you the racks of servers, show you the computer system, explain the physics that allows it, lets you do the experiments that shows those physics are correct. You could solve any NP complete problem on the computer. And you could run programs that take known numbers of steps to compute. Like actually calculating 3^^^3, etc.
Sure. But I think there are generally going to be more parsimonious explanations than any that involve him having the power to torture 3^^^3 people, let alone having that power and caring about whether I give him some money.
Parsimonious, sure. The possibility is very unlikely. But it doesn’t just need to be “very unlikely”, it needs to have smaller than 1/3^^^3 probability.
Sure. But if you have an argument that some guy who shows me apparent magical powers has the power to torture 3^^^3 people with probability substantially over 1/3^^^3, then I bet I can turn it into an argument that anyone, with or without a demonstration of magical powers, with or without any sort of claim that they have such powers, has the power to torture 3^^^3 people with probability nearly as substantially over 1/3^^^3. Because surely for anyone under any circumstances, Pr(I experience what seems to be a convincing demonstration that they have such powers) is much larger than 1/3^^^3, whether they actually have such powers or not.
Correct. That still doesn’t solve the decision theory problem, it makes it worse. Since you have to take into account the possibility that anyone you meet might have the power to torture (or reward with utopia) 3^^^3 people.
It makes it worse or better, depending on whether you decide (1) that everyone has the power to do that with probability >~ 1/3^^^3 or (2) that no one has. I think #2 rather than #1 is correct.
Well, doing basic Bayes with a Kolmogorov priot gives you (1).
About as unfathomably small as the number of 3^^^3 people is unfathomably large?
I think you’re relying on “but I feel this can’t be right!” a bit too much.
I don’t see what your point is. Yes that’s a small number. It’s not a feeling, that’s just math. If you are assigning things 1/3^^^3 probability, you are basically saying they are impossible and no amount of evidence could convince you otherwise.
You can do that and be perfectly consistent. If that’s your point I don’t disagree. You can’t argue about priors. We can only agree to disagree, if those are your true priors.
Just remember that reality could always say “WRONG!” and punish you for assigning 0 probability to something. If you don’t want to be wrong, don’t assign 1/3^^^3 probability to things you aren’t 99.9999...% sure absolutely can’t happen.
Eliezer showed a problem that that reasoning in his post on Pascal’s Muggle.
Basically, human beings do not have an actual prior probability distribution. This should be obvious, since it means assigning a numerical probability to every possible state of affairs. No human being has ever done this, or ever will.
But you have something like a prior, but you build the prior itself based on your experience. At the moment we don’t have a specific number for the probability of the mugging situation coming up, but just think it’s very improbable, so that we don’t expect any evidence to ever come up that would convince us. But if the mugger shows matrix powers, we would change our prior so that the probability of the mugging situation was high enough to be convinced by being shown matrix powers.
You might say that means it was already that high, but it does not mean this, given the objective fact that people do not have real priors.
Maybe humans don’t really have probability distributions. But that doesn’t help us actually build an AI which reproduces the same result. If we had infinite computing power and could do ideal Solomonoff induction, it would pay the mugger.
Though I would argue that humans do have approximate probability functions and approximate priors. We wouldn’t be able to function in a probabilistic world if we didn’t. But it’s not relevant.
That’s just a regular bayesian probability update! You don’t need to change terminology and call it something different.
That’s fine. I too think the situation is extraordinarily implausible. Even Solomonoff induction would agree with us. The probability that the mugger is real would be something like 1/10^100. Or perhaps the exponent should be orders of magnitude larger than that. That’s small enough that it shouldn’t even remotely register as a plausible hypothesis in your mind. But big enough some amount of evidence could convince you.
You don’t need to posit new models of how probability theory should work. Regular probability works fine at assigning really implausible hypotheses really low probability.
But that is still way, way bigger than 1/3^^^3.
If utility is straightforwardly additive, yes. But perhaps it isn’t. Imagine two possible worlds. In one, there are a billion copies of our planet and its population, all somehow leading exactly the same lives. In another, there are a billion planets like ours, with different people on them. Now someone proposes to blow up one of the planets. I find that I feel less awful about this in the first case than the second (though of course either is awful) because what’s being lost from the universe is something of which we have a billion copies anyway. If we stipulate that the destruction of the planet is instantaneous and painless, and that the people really are living exactly identical lives on each planet, then actually I’m not sure I care very much that one planet is gone. (But my feelings about this fluctuate.)
A world with 3^^^3 inhabitants that’s described by (say) no more than a billion bits seems a little like the first of those hypothetical worlds.
I’m not very sure about this. For instance, perhaps the description would take the form: “Seed a good random number generator as follows. [...] Now use it to generate 3^^^3 person-like agents in a deterministic universe with such-and-such laws. Now run it for 20 years.” and maybe you can get 3^^^3 genuinely non-redundant lives that way. But 3^^^3 is a very large number, and I’m not even quite sure there’s such a thing as 3^^^3 genuinely non-redundant lives even in principle.
Well what if instead of killing them, he tortured them for an hour? Death might not matter in a Big World, but total suffering still does.
I dunno. If I imagine a world with a billion identical copies of me living identical lives, having all of them tortured doesn’t seem a billion times worse than having one tortured. Would an AI’s experiences matter more if, to reduce the impact of hardware error, all its computations were performed on ten identical computers?
What if any of the Big World hypotheses are true? E.g. many worlds interpretation, multiverse theories, Tegmark’s hypothesis, or just a regular infinite universe. In that case anything that can exist does exist. There already are a billion versions of you being tortured. An infinite number actually. All you can ever really do is reduce the probability that you will find yourself in a good world or a bad one.
Bounded utility functions effectively give “bounded probability functions,” in the sense that you (more or less) stop caring about things with very low probability.
For example, if my maximum utility is 1,000, then my maximum utility for something with a probability of one in a billion is .0000001, an extremely small utiliity, so something that I will care about very little. The probability of of the 3^^^3 scenarios may be more than one in 3^^^3. But it will still be small enough that a bounded utility function won’t care about situations like that, at least not to any significant extent.
That is precisely the reason that it will do the things you object to, if that situation comes up.
That is no different from pointing out that the post’s proposal will reject a “mugging” even when it will actually cost 3^^^3 lives.
Both proposals have that particular downside. That is not something peculiar to mine.
Bounded utility functions mean you stop caring about things with very high utility. That you care less about certain low probability events is just a side effect. But those events can also have very high probability and you still don’t care.
If you want to just stop caring about really low probability events, why not just do that?
I just explained. There is no situation involving 3^^^3 people which will ever have a high probability. Telling me I need to adopt a utility function which will handle such situations well is trying to mug me, because such situations will never come up.
Also, I don’t care about the difference between 3^^^^^3 people and 3^^^^^^3 people even if the probability is 100%, and neither does anyone else. So it isn’t true that I just want to stop caring about low probability events. My utility is actually bounded. That’s why I suggest using a bounded utility function, like everyone else does.
Really? No situation? Not even if we discover new laws of physics that allow us to have infinite computing power?
We are talking about utility functions. Probability is irrelevant. All that matters for the utility function is that if the situation came up, you would care about it.
I totally disagree with you. These numbers are so incomprehensibly huge you can’t picture them in your head, sure. There is massive scope insensitivity. But if you had to make moral choices that affect those two numbers of people, you should always value the bigger number proportionally more.
E.g. if you had to torture 3^^^^^3 to save 3^^^^^^3 from getting dust specks in their eyes. Or make bets involving probabilities between various things happening to the different groups. Etc. I don’t think you can make these decisions correctly if you have a bounded utility function.
If you don’t make them correctly, well that 3^^^3 people probably contains a basically infinite number of copies of you. By making the correct tradeoffs, you maximize the probability that the other versions of yoruself find themselves in a universe with higher utility.