Given the mixed strategy, taking and not taking your bet both have infinite expected utility, even if there are no other infinite expected utility lotteries.
I agree that there are many bets with infinite expected utility for a person who has unbounded utility. If the subject takes those bets into account, it’s unlikely that I’ll win in the sense of getting the subject to send me money. However, if the subject takes them into account, it’s very likely that the subject will lose, in the sense that the subject’s estimated utility from all of these infinite expected utility bets is going to swamp utility from ordinary things. If someone is hungry and has an apple, they should pay attention to the apple to decide whether to eat it, and not pay attention to the relative risk of Tim being a god who disapproves of apple-eating or Carl being a god who promotes apple-eating.
2) To get a decision theory that actually would take infinite expected utility lotteries with high probability we would need to use something like the hyperreals, which would allow for differences in the utility of different probabilities of infinite payoff. But once we do that, the fact that your offer is so implausible penalizes it.
I don’t care much whether someone accepts my offer. I really care whether they can pay attention to an apple when they’re hungry to decide whether to eat the apple, as opposed to considering obscure consequences of how various possible unlikely gods might react to the eating of the apple. I am not convinced that hyperreals solve that problem—so far as I can tell, the outcome would be unchanged. Can you explain why you think hyperreals might help?
(ETA: hyperreals, AKA the non-standard reals, aren’t mysterious. Imagine the real numbers, imagine a new one we might call “infinity”, then add other new numbers as required so all of the usual first-order properties still hold. So we’d have infinity − 3 and 5 * infinity − 376/infinity and so forth. So far as I can tell, if you do the procedure described in the OP with hyperreal utilities, you still conclude that the utility of giving me money exceeds the utility of keeping the money and spending it on something ordinary.)
Conditional on there being any sources of infinite utility, it is far more likely that they will be better obtained by other routes than by succumbing to this trick.
Perhaps you meant unbounded instead of infinite there.
I’m concerned that the tricky routes will dominate the non-tricky routes. I don’t really expect anyone to fall for my specific trick.
Also see Nick Bostrom’s infinitarian ethics paper.
Last I read that was long ago. I glanced at it just now and it seems to be concerned with ethics, rather than an individual deciding what to do, so I’m having doubts about it being directly relevant. It’s probably worth looking at anyway, but if you can say specifically how it’s relevant and cite a specific page it would help.
The problem here is technical, in the construction of your example.
Does the problem still exist if we assume the purpose of my example is to show that people with unbounded utility lose, rather than to make people send me money?
However, if the subject takes them into account, it’s very likely that the subject will lose, in the sense that the subject’s estimated utility from all of these infinite expected utility bets is going to swamp utility from ordinary things.
The point of mixed strategies is that without distinctions between lotteries with infinite expected utility all actions have the same infinite (or undefined) expected utility, so on that framework there is no reason to prefer one action over another. Hyperreals or some other modification to the standard framework (see discussion of “infinity shades” in Bostrom) are necessary in order to say that a 50% chance of infinite utility is better than a 1/3^^^3 chance of infinite utility. Read the Hajek paper for the full details.
It’s probably worth looking at anyway, but if you can say specifically how it’s relevant and cite a specific page it would help.
Hyperreals or some other modification to the standard framework (see discussion of “infinity shades” in Bostrom) are necessary in order to say that a 50% chance of infinite utility is better than a 1/3^^^3 chance of infinite utility.
No it isn’t, unless like Hayek you think there’s something ‘not blindingly obvious’ about the ‘modification to the standard framework’ that consists of stipulating that probability p of infinite utility is better than probability q of infinite utility whenever p > q.
This sort of ‘move’ doesn’t need a name. (What does he call it? “Vector valued utilities” or something like that?) It doesn’t need to have a paper written about it. It certainly shouldn’t be pretended that we’re somehow ‘improving on’ or ‘fixing the flaws in’ Pascal’s original argument by explicitly writing this move down.
A system which selects actions so as to maximize the probability of receiving infinitely many units of some good, without differences in the valuation of different infinite payouts, approximates to a bounded utility function, e.g. assigning utility 1 to world-histories with an infinite payout of the good, and 0 to all other world-histories.
Hyperreals or some other modification to the standard framework (see discussion of “infinity shades” in Bostrom) are necessary in order to say that a 50% chance of infinite utility is better than a 1/3^^^3 chance of infinite utility.
Sigh, we seem to be talking past each other. You’re talking about choosing which unlikely god jerks you around, and I’m trying to say that it’s eventually time to eat lunch. If you have infinite utilities, how can you ever justify prioritizing something finite and likely, like eating lunch, over something unlikely but infinite? Keeping a few dollars is like eating lunch, so if you can’t rationally decide to eat lunch, the question is which unlikely god you’ll give your money to. I agree that it probably won’t be me.
If you have infinite utilities, how can you ever justify prioritizing something finite and likely, like eating lunch, over something unlikely but infinite?
Why is eating lunch “finite”, given that we have the possibility of becoming gods ourselves, and eating lunch makes that possibility more likely (compared to not eating lunch)?
ETA: Suppose you saw convincing evidence that skipping lunch would make you more productive at FAI-building (say there’s an experiment showing that lunch makes people mentally slow in the afternoon), you would start skipping lunch, right? Even if you wouldn’t, would it be irrational for someone else to do so?
There are two issues here: 1) What the most plausible cashing-out of an unbounded utility function recommends 2) Whether that cashing-out is a sensible summary of someone’s values. I agree with you on 2) but think that you are giving bogus examples for 1). As with previous posts, if you concoct examples that have many independent things wrong with them, they don’t clearly condemn any particular component.
My understanding is that you want to say, with respect to 2), that you don’t want to act in accord with any such cashing-out, i.e. that your utility function is bounded insofar as you have one. Fine with me, I would say my own utility function is bounded too (although some of the things I assign finite utility to involve infinite amounts of stuff, e.g. I would prefer living forever to living 10,000 years, although boundedly so). Is that right?
But you also keep using what seem to be mistaken cashing-outs in response to 1). For instance, you say that:
Keeping a few dollars is like eating lunch, so if you can’t rationally decide to eat lunch,
But any decision theory/prior/utility function combination that gives in to Pascal’s Mugging will also recommend eating lunch (if you don’t eat lunch you will be hungry and have reduced probability of gaining your aims, whether infinite or finite). Can we agree on that?
If we can, then you should use examples where a bounded utility function and an unbounded utility function actually give conflicting recommendations about which action to take. As far as I can see, you haven’t done so yet.
I think you mean Arguments for—and against—probabilism. If you meant something else, please correct me.
I meant the paper that I already linked to earlier in this thread.
There are two issues here: 1) What the most plausible cashing-out of an unbounded utility function recommends 2) Whether that cashing-out is a sensible summary of someone’s values. I agree with you on 2) but think that you are giving bogus examples for 1). As with previous posts, if you concoct examples that have many independent things wrong with them, they don’t clearly condemn any particular component.
I agree that we agree on 2).
The conflict here seems to be that you’re trying to persist and do math after getting unbounded utilities, and I’m inclined to look at ridiculous inputs and outputs from the decision making procedure and say “See? It’s broken. Don’t do that!”. In this case the ridiculous input is a guess about the odds of me being god, and the ridiculous output is to send me money, or divert resources to some other slightly less unlikely god if I don’t win the contest.
But any decision theory/prior/utility function combination that gives in to Pascal’s Mugging will also recommend eating lunch (if you don’t eat lunch you will be hungry and have reduced probability of gaining your aims, whether infinite or finite). Can we agree on that?
Maybe. I don’t know what it would conclude about eating lunch. Maybe the decision would be to eat lunch, or maybe some unknown interaction of the guesses about the unlikely gods would lead to performing bizarre actions to satisfy whichever of them seemed more likely than the others. Maybe there’s a reason people don’t trust fanatics.
If we can, then you should use examples where a bounded utility function and an unbounded utility function actually give conflicting recommendations about which action to take. As far as I can see, you haven’t done so yet.
Well, if we can exclude all but one of the competing unlikely gods, the OP is such an example. A bounded utility function would lead to a decision to keep the money rather than send it to me.
Otherwise I don’t have one. I don’t expect to have one because I think that working with unbounded utility functions is intractible even if we can get it to be mathematically well-defined, since there are too many unlikely gods to enumerate.
But at this point I think I should retreat and reconsider. I want to read that paper by Hajek, and I want to understand the argument for bounded utility from Savage’s axioms, and I want to understand where having utilities that are surreal or hyperreal numbers fails to match those axioms. I found are a few papers about how to avoid paradoxes with unbounded utilities, too.
This has turned up lots of stuff that I want to pay attention to. Thanks for the pointers.
I really care whether they can pay attention to an apple when they’re hungry to decide whether to eat the apple, as opposed to considering obscure consequences of how various possible unlikely gods might react to the eating of the apple.
Personal survival makes it a lot easier to please unlikely gods, so eating the apple is preferred. For more general situations, some paths to infinity are much more probable than others. For example, perhaps we can build a god.
I really care whether they can pay attention to an apple when they’re hungry to decide whether to eat the apple, as opposed to considering obscure consequences of how various possible unlikely gods might react to the eating of the apple.
Personal survival makes it a lot easier to please unlikely gods, so eating the apple is preferred.
Eating an apple was meant to be an example of a trivial thing. Inflating it to personal survival misses the point. Eating an apple should be connected to your own personal values concerning hunger and apples, and there should be a way to make a decision about eating an apple or not eating it and being slightly hungry based on your personal values about apples and hunger. If we have to think about unlikely gods to decide whether to eat an apple, something is broken.
For more general situations, some paths to infinity are much more probable than others. For example, perhaps we can build a god [SIAI pointer].
That’s a likely god, not an unlikely god, so it’s a little bit different. Even then, low-probability interactions between eating an apple and the nature of the likely god seem likely to lead to bizarre decision processes about apple-eating, unless you have bounded utilities.
Even then, low-probability interactions between eating an apple and the nature of the likely god seem likely to lead to bizarre decision processes about apple-eating, unless you have bounded utilities.
I don’t see why this is a problem. What causes you to find it so unlikely that our desires could work this way?
Even then, low-probability interactions between eating an apple and the nature of the likely god seem likely to lead to bizarre decision processes about apple-eating, unless you have bounded utilities.
What causes you to find it so unlikely that our desires could work this way?
Pay attention next time you eat something. Do you look at the food and eat what you like or what you think will improve your health, or do you try to prioritize eating the food against sending me money because I might be a god, and against giving all of the other unlikely gods what they might want?
We are human and cannot really do that. With unbounded utilities, there are an absurdly large number of possible ways that an ordinary action can have very low-probability influence on a wide variety of very high-utility things, and you have to take them all into account and balance them properly to do the right thing. If an AI is doing that, I have no confidence at all that it will weigh these things the way I would like, especially given that it’s not likely to search all of the space. Someone who thinks about a million unlikely gods to decide whether to eat an apple is broken. In practice, they won’t be able to do that, and their decision about whether to eat the apple will be driven by whatever unlikely gods have been brought to their attention in the last minute. (As I said before, an improbable minor change to a likely god is an unlikely god, for the purposes of this discussion.)
If utilities are bounded, then the number of alternatives you have to look at doesn’t grow so pathologically large, and you look at the apple to decide whether to eat the apple. The unlikely gods don’t enter into it because you don’t imagine that they can make enough of a difference to outweigh their unlikeliness.
Someone who thinks about a million unlikely gods to decide whether to eat an apple is broken. In practice, they won’t be able to do that, and their decision about whether to eat the apple will be driven by whatever unlikely gods have been brought to their attention in the last minute.
Why can’t they either estimate or prove that eating an apple has more expected utility (by please more gods overall than not eating an apple, say), without iterating over each god and considering them separately? And if for some reason you build an AI that does compute expected utility by brute force iteration of possibilities, then you obviously would not want it to consider only possibilities that “have been brought to their attention in the last minute”. That’s going to lead to trouble no matter what kind of utility function you give it.
(ETA: I think it’s likely that humans do have bounded utility functions (if we can be said to have utility functions at all) but your arguments here are not very good. BTW, have you seen The Lifespan Dilemma?)
Pay attention next time you eat something. Do you look at the food and eat what you like or what you think will improve your health, or do you try to prioritize eating the food against sending me money because I might be a god, and against giving all of the other unlikely gods what they might want?
I would like to do whichever of these two alternatives leads to more utility.
We are human and cannot really do that.
If an AI is doing that, I have no confidence at all that it will weigh these things the way I would like, especially given that it’s not likely to search all of the space.
Are you saying that we shouldn’t maximize utility because it’s too hard?
Someone who thinks about a million unlikely gods to decide whether to eat an apple is broken.
If your actual utility function is unbounded and thinking about a million “unlikely gods” is worth the computational resources that could be spent on likely gods (though you specified that small changes to likely gods are unlikely gods, there is a distinction in that there are not a metaphorical million of them), than that is your actual preference. The utility function is not up for grabs.
Your argument seems to be that maximizing an unbounded utility function is impractical, so we should maximize a bounded utility function instead. I find it improbable that you would make this argument, so, if I am missing anything, please clarify.
Yes, the utility function is not up for grabs, but introspection doesn’t tell you what it is either. In particular, the statement “endoself acts approximately consistently with utility function U” is an empirical statement for any given U (and any particular notion of “approximately”, but let’s skip that part for now). I believe I have provided fine arguments that you are not acting approximately consistently with an unbounded utility function, and that you will never be able to do so. If those arguments are valid, and you say you have an unbounded utility function, then you are wrong.
If those arguments are valid, and you say you want to have an unbounded utility function, then you’re wanting something impossible because you falsely believe it to be possible. The best I could do in that case if I were helping you would be to give you what you would want if you had true beliefs. I don’t know what that would be. What would you want from an unbounded utility function that you couldn’t get if the math turned out so that only bounded utility functions can be used in a decision procedure?
you specified that small changes to likely gods are unlikely gods, there is a distinction in that there are not a metaphorical million of them
There are many paths by which small actions taken today might in unlikely ways influence the details of how a likely god is built. If those paths have infinite utility, you have to analyze them to decide what to do.
I believe I have provided fine arguments that you are not acting approximately consistently with an unbounded utility function, and that you will never be able to do so.
I am currently researching logical uncertainty. I believe that the increased chance of FAI due to this research makes it the best way to act according to my utility function, taking into account the limits to my personal rationality (part of this is personal; I am particularly interested in logical uncertainty right now, so I am more likely to make progress in it than on other problems). This is because, among other things, an FAI will be far better at understanding the difficulties associated with unbounded utility functions than I am.
If those arguments are valid, and you say you want to have an unbounded utility function, then you’re wanting something impossible because you falsely believe it to be possible.
You have not demonstrated it to be impossible, you have just shown that the most obvious approach to it does not work. Given how questionable some of the axioms we use are, this is not particularly surprising.
What would you want from an unbounded utility function that you couldn’t get if the math turned out so that only bounded utility functions can be used in a decision procedure?
There are many paths by which small actions taken today might in unlikely ways influence the details of how a likely god is built. If those paths have infinite utility, you have to analyze them to decide what to do.
Some paths are far more likely than others. Actively researching FAI in a way that is unlikely to significantly increase the probability of UFAI provides far more expected utility than unlikely ways to help the development of FAI.
What would you want from an unbounded utility function that you couldn’t get if the math turned out so that only bounded utility functions can be used in a decision procedure?
An actual description of my preferences. I am unsure whether my utility function is actually unbounded but I find it probable that, for example, my utility function is linear in people. I don’t want to rule this out just because that current framework is insufficient for it.
Predicting your preferences requires specifying both the utility function and the framework, so offering a utility function without the framework as an explanation for your preferences does not actually explain them. I actually don’t know if my question was hypothetical or not. Do we have a decision procedure that gives reasonable results for an unbounded utility function?
The phrase “rule this out” seems interesting here. At any given time, you’ll have a set of explanations for your behavior. That doesn’t rule out coming up with better explanations later. Does the best explanation you have for your preferences that works with a known decision theory have bounded utility?
Perhaps I see what’s going on here—people who want unbounded utility are feeling loss when they imagine giving that up that unbounded goodness in order to avoid bugs like the one described in the OP. I, on the other hand, feel loss when people dither over difficult math problems when the actual issues confronting us have nothing to do with difficult math. Specifically, dealing effectively with the default future, in which one or more corporations make AI’s that optimize for something having no connection to the preferences of any individual human.
Do we have a decision procedure that gives reasonable results for an unbounded utility function?
Not one compatible with a Solomonoff prior. I agree that a utility function alone is not a full description of preferences.
Does the best explanation you have for your preferences that works with a known decision theory have bounded utility?
The best explanation that I have for my preferences does not, AFAICT, work with any known decision theory. However, I know enough of what such a decision theory would look like if it were possible to say that it would not have bounded utility.
I, on the other hand, feel loss when people dither over difficult math problems when the actual issues confronting us have nothing to do with difficult math.
I disagree that I am doing such. Whether or not the math is relevant to the issue is a question of values, not fact. Your estimates of your values do not find the math relevant; my estimates of my values do.
downvoted because you actually said “I would like to do whichever of these two alternatives leads to more utility.”
A) no one or almost no one thinks this way, and advice based on this sort of thinking is useless to almost everyone.
B) The entire point of the original post was that, if you try to do this, then you immediately get completely taken over by consideration of any gods you can imagine. When you say that thinking about unlikely gods is not “worth” the computational resources, you are sidestepping the very issue we are discussing. You have already decided it’s not worth thinking about tiny probabilities of huge returns.
I think he actually IS making the argument that you assign a low probability to, but instead of dismissing it I think it’s actually extremely important to decide whether to take certain courses based on how practical they are. The entire original purpose of this community is research into AI, and while you can’t choose your own utility function, you can choose an AI’s. If this problem is practically insoluble, then we should design AIs with only bounded utility functions.
downvoted because you actually said “I would like to do whichever of these two alternatives leads to more utility.”
Tim seemed to be implying that it would be absurd for unlikely gods to be the most important motive for determining how to act, but I did not see how anything that he said showed that doing so is actually a bad idea.
When you say that thinking about unlikely gods is not “worth” the computational resources, you are sidestepping the very issue we are discussing.
What? I did not say that; I said that thinking about unlikely gods might just be one’s actual preference. I also pointed out that Tim did not prove that unlikely gods are more important than likely gods, so one who accepts most of his argument might still not motivated by “a million unlikely gods”.
I agree that there are many bets with infinite expected utility for a person who has unbounded utility. If the subject takes those bets into account, it’s unlikely that I’ll win in the sense of getting the subject to send me money. However, if the subject takes them into account, it’s very likely that the subject will lose, in the sense that the subject’s estimated utility from all of these infinite expected utility bets is going to swamp utility from ordinary things. If someone is hungry and has an apple, they should pay attention to the apple to decide whether to eat it, and not pay attention to the relative risk of Tim being a god who disapproves of apple-eating or Carl being a god who promotes apple-eating.
I don’t care much whether someone accepts my offer. I really care whether they can pay attention to an apple when they’re hungry to decide whether to eat the apple, as opposed to considering obscure consequences of how various possible unlikely gods might react to the eating of the apple. I am not convinced that hyperreals solve that problem—so far as I can tell, the outcome would be unchanged. Can you explain why you think hyperreals might help?
(ETA: hyperreals, AKA the non-standard reals, aren’t mysterious. Imagine the real numbers, imagine a new one we might call “infinity”, then add other new numbers as required so all of the usual first-order properties still hold. So we’d have infinity − 3 and 5 * infinity − 376/infinity and so forth. So far as I can tell, if you do the procedure described in the OP with hyperreal utilities, you still conclude that the utility of giving me money exceeds the utility of keeping the money and spending it on something ordinary.)
Perhaps you meant unbounded instead of infinite there.
I’m concerned that the tricky routes will dominate the non-tricky routes. I don’t really expect anyone to fall for my specific trick.
Last I read that was long ago. I glanced at it just now and it seems to be concerned with ethics, rather than an individual deciding what to do, so I’m having doubts about it being directly relevant. It’s probably worth looking at anyway, but if you can say specifically how it’s relevant and cite a specific page it would help.
Does the problem still exist if we assume the purpose of my example is to show that people with unbounded utility lose, rather than to make people send me money?
The point of mixed strategies is that without distinctions between lotteries with infinite expected utility all actions have the same infinite (or undefined) expected utility, so on that framework there is no reason to prefer one action over another. Hyperreals or some other modification to the standard framework (see discussion of “infinity shades” in Bostrom) are necessary in order to say that a 50% chance of infinite utility is better than a 1/3^^^3 chance of infinite utility. Read the Hajek paper for the full details.
“Empirical stabilizing assumptions” (naturalistic), page 34.
No it isn’t, unless like Hayek you think there’s something ‘not blindingly obvious’ about the ‘modification to the standard framework’ that consists of stipulating that probability p of infinite utility is better than probability q of infinite utility whenever p > q.
This sort of ‘move’ doesn’t need a name. (What does he call it? “Vector valued utilities” or something like that?) It doesn’t need to have a paper written about it. It certainly shouldn’t be pretended that we’re somehow ‘improving on’ or ‘fixing the flaws in’ Pascal’s original argument by explicitly writing this move down.
A system which selects actions so as to maximize the probability of receiving infinitely many units of some good, without differences in the valuation of different infinite payouts, approximates to a bounded utility function, e.g. assigning utility 1 to world-histories with an infinite payout of the good, and 0 to all other world-histories.
We are making the argument more formal. Doing so is a good idea in a wide variety of situations.
Do you disagree with any of these claims?
Introducing hyperreals makes the argument more formal
Making an argument more formal is often good
Here, making the argument more formal is more likely good than bad.
Sigh, we seem to be talking past each other. You’re talking about choosing which unlikely god jerks you around, and I’m trying to say that it’s eventually time to eat lunch. If you have infinite utilities, how can you ever justify prioritizing something finite and likely, like eating lunch, over something unlikely but infinite? Keeping a few dollars is like eating lunch, so if you can’t rationally decide to eat lunch, the question is which unlikely god you’ll give your money to. I agree that it probably won’t be me.
I think you mean Arguments for—and against—probabilism. If you meant something else, please correct me.
Why is eating lunch “finite”, given that we have the possibility of becoming gods ourselves, and eating lunch makes that possibility more likely (compared to not eating lunch)?
ETA: Suppose you saw convincing evidence that skipping lunch would make you more productive at FAI-building (say there’s an experiment showing that lunch makes people mentally slow in the afternoon), you would start skipping lunch, right? Even if you wouldn’t, would it be irrational for someone else to do so?
There are two issues here: 1) What the most plausible cashing-out of an unbounded utility function recommends 2) Whether that cashing-out is a sensible summary of someone’s values. I agree with you on 2) but think that you are giving bogus examples for 1). As with previous posts, if you concoct examples that have many independent things wrong with them, they don’t clearly condemn any particular component.
My understanding is that you want to say, with respect to 2), that you don’t want to act in accord with any such cashing-out, i.e. that your utility function is bounded insofar as you have one. Fine with me, I would say my own utility function is bounded too (although some of the things I assign finite utility to involve infinite amounts of stuff, e.g. I would prefer living forever to living 10,000 years, although boundedly so). Is that right?
But you also keep using what seem to be mistaken cashing-outs in response to 1). For instance, you say that:
But any decision theory/prior/utility function combination that gives in to Pascal’s Mugging will also recommend eating lunch (if you don’t eat lunch you will be hungry and have reduced probability of gaining your aims, whether infinite or finite). Can we agree on that?
If we can, then you should use examples where a bounded utility function and an unbounded utility function actually give conflicting recommendations about which action to take. As far as I can see, you haven’t done so yet.
I meant the paper that I already linked to earlier in this thread.
I agree that we agree on 2).
The conflict here seems to be that you’re trying to persist and do math after getting unbounded utilities, and I’m inclined to look at ridiculous inputs and outputs from the decision making procedure and say “See? It’s broken. Don’t do that!”. In this case the ridiculous input is a guess about the odds of me being god, and the ridiculous output is to send me money, or divert resources to some other slightly less unlikely god if I don’t win the contest.
Maybe. I don’t know what it would conclude about eating lunch. Maybe the decision would be to eat lunch, or maybe some unknown interaction of the guesses about the unlikely gods would lead to performing bizarre actions to satisfy whichever of them seemed more likely than the others. Maybe there’s a reason people don’t trust fanatics.
Well, if we can exclude all but one of the competing unlikely gods, the OP is such an example. A bounded utility function would lead to a decision to keep the money rather than send it to me.
Otherwise I don’t have one. I don’t expect to have one because I think that working with unbounded utility functions is intractible even if we can get it to be mathematically well-defined, since there are too many unlikely gods to enumerate.
But at this point I think I should retreat and reconsider. I want to read that paper by Hajek, and I want to understand the argument for bounded utility from Savage’s axioms, and I want to understand where having utilities that are surreal or hyperreal numbers fails to match those axioms. I found are a few papers about how to avoid paradoxes with unbounded utilities, too.
This has turned up lots of stuff that I want to pay attention to. Thanks for the pointers.
ETA: Readers may want to check my earlier comment pointing to a free substitute for the paywalled Hajek article.
Personal survival makes it a lot easier to please unlikely gods, so eating the apple is preferred. For more general situations, some paths to infinity are much more probable than others. For example, perhaps we can build a god.
Eating an apple was meant to be an example of a trivial thing. Inflating it to personal survival misses the point. Eating an apple should be connected to your own personal values concerning hunger and apples, and there should be a way to make a decision about eating an apple or not eating it and being slightly hungry based on your personal values about apples and hunger. If we have to think about unlikely gods to decide whether to eat an apple, something is broken.
That’s a likely god, not an unlikely god, so it’s a little bit different. Even then, low-probability interactions between eating an apple and the nature of the likely god seem likely to lead to bizarre decision processes about apple-eating, unless you have bounded utilities.
I don’t see why this is a problem. What causes you to find it so unlikely that our desires could work this way?
Pay attention next time you eat something. Do you look at the food and eat what you like or what you think will improve your health, or do you try to prioritize eating the food against sending me money because I might be a god, and against giving all of the other unlikely gods what they might want?
We are human and cannot really do that. With unbounded utilities, there are an absurdly large number of possible ways that an ordinary action can have very low-probability influence on a wide variety of very high-utility things, and you have to take them all into account and balance them properly to do the right thing. If an AI is doing that, I have no confidence at all that it will weigh these things the way I would like, especially given that it’s not likely to search all of the space. Someone who thinks about a million unlikely gods to decide whether to eat an apple is broken. In practice, they won’t be able to do that, and their decision about whether to eat the apple will be driven by whatever unlikely gods have been brought to their attention in the last minute. (As I said before, an improbable minor change to a likely god is an unlikely god, for the purposes of this discussion.)
If utilities are bounded, then the number of alternatives you have to look at doesn’t grow so pathologically large, and you look at the apple to decide whether to eat the apple. The unlikely gods don’t enter into it because you don’t imagine that they can make enough of a difference to outweigh their unlikeliness.
Why can’t they either estimate or prove that eating an apple has more expected utility (by please more gods overall than not eating an apple, say), without iterating over each god and considering them separately? And if for some reason you build an AI that does compute expected utility by brute force iteration of possibilities, then you obviously would not want it to consider only possibilities that “have been brought to their attention in the last minute”. That’s going to lead to trouble no matter what kind of utility function you give it.
(ETA: I think it’s likely that humans do have bounded utility functions (if we can be said to have utility functions at all) but your arguments here are not very good. BTW, have you seen The Lifespan Dilemma?)
I would like to do whichever of these two alternatives leads to more utility.
Are you saying that we shouldn’t maximize utility because it’s too hard?
If your actual utility function is unbounded and thinking about a million “unlikely gods” is worth the computational resources that could be spent on likely gods (though you specified that small changes to likely gods are unlikely gods, there is a distinction in that there are not a metaphorical million of them), than that is your actual preference. The utility function is not up for grabs.
Your argument seems to be that maximizing an unbounded utility function is impractical, so we should maximize a bounded utility function instead. I find it improbable that you would make this argument, so, if I am missing anything, please clarify.
Yes, the utility function is not up for grabs, but introspection doesn’t tell you what it is either. In particular, the statement “endoself acts approximately consistently with utility function U” is an empirical statement for any given U (and any particular notion of “approximately”, but let’s skip that part for now). I believe I have provided fine arguments that you are not acting approximately consistently with an unbounded utility function, and that you will never be able to do so. If those arguments are valid, and you say you have an unbounded utility function, then you are wrong.
If those arguments are valid, and you say you want to have an unbounded utility function, then you’re wanting something impossible because you falsely believe it to be possible. The best I could do in that case if I were helping you would be to give you what you would want if you had true beliefs. I don’t know what that would be. What would you want from an unbounded utility function that you couldn’t get if the math turned out so that only bounded utility functions can be used in a decision procedure?
There are many paths by which small actions taken today might in unlikely ways influence the details of how a likely god is built. If those paths have infinite utility, you have to analyze them to decide what to do.
I am currently researching logical uncertainty. I believe that the increased chance of FAI due to this research makes it the best way to act according to my utility function, taking into account the limits to my personal rationality (part of this is personal; I am particularly interested in logical uncertainty right now, so I am more likely to make progress in it than on other problems). This is because, among other things, an FAI will be far better at understanding the difficulties associated with unbounded utility functions than I am.
You have not demonstrated it to be impossible, you have just shown that the most obvious approach to it does not work. Given how questionable some of the axioms we use are, this is not particularly surprising.
An actual description of my preferences. I am unsure whether my utility function is actually unbounded but I find it probable that, for example, my utility function is linear in people. I don’t want to rule this out just because that current framework is insufficient for it.
Some paths are far more likely than others. Actively researching FAI in a way that is unlikely to significantly increase the probability of UFAI provides far more expected utility than unlikely ways to help the development of FAI.
Predicting your preferences requires specifying both the utility function and the framework, so offering a utility function without the framework as an explanation for your preferences does not actually explain them. I actually don’t know if my question was hypothetical or not. Do we have a decision procedure that gives reasonable results for an unbounded utility function?
The phrase “rule this out” seems interesting here. At any given time, you’ll have a set of explanations for your behavior. That doesn’t rule out coming up with better explanations later. Does the best explanation you have for your preferences that works with a known decision theory have bounded utility?
Perhaps I see what’s going on here—people who want unbounded utility are feeling loss when they imagine giving that up that unbounded goodness in order to avoid bugs like the one described in the OP. I, on the other hand, feel loss when people dither over difficult math problems when the actual issues confronting us have nothing to do with difficult math. Specifically, dealing effectively with the default future, in which one or more corporations make AI’s that optimize for something having no connection to the preferences of any individual human.
Not one compatible with a Solomonoff prior. I agree that a utility function alone is not a full description of preferences.
The best explanation that I have for my preferences does not, AFAICT, work with any known decision theory. However, I know enough of what such a decision theory would look like if it were possible to say that it would not have bounded utility.
I disagree that I am doing such. Whether or not the math is relevant to the issue is a question of values, not fact. Your estimates of your values do not find the math relevant; my estimates of my values do.
downvoted because you actually said “I would like to do whichever of these two alternatives leads to more utility.”
A) no one or almost no one thinks this way, and advice based on this sort of thinking is useless to almost everyone.
B) The entire point of the original post was that, if you try to do this, then you immediately get completely taken over by consideration of any gods you can imagine. When you say that thinking about unlikely gods is not “worth” the computational resources, you are sidestepping the very issue we are discussing. You have already decided it’s not worth thinking about tiny probabilities of huge returns.
I think he actually IS making the argument that you assign a low probability to, but instead of dismissing it I think it’s actually extremely important to decide whether to take certain courses based on how practical they are. The entire original purpose of this community is research into AI, and while you can’t choose your own utility function, you can choose an AI’s. If this problem is practically insoluble, then we should design AIs with only bounded utility functions.
Tim seemed to be implying that it would be absurd for unlikely gods to be the most important motive for determining how to act, but I did not see how anything that he said showed that doing so is actually a bad idea.
What? I did not say that; I said that thinking about unlikely gods might just be one’s actual preference. I also pointed out that Tim did not prove that unlikely gods are more important than likely gods, so one who accepts most of his argument might still not motivated by “a million unlikely gods”.