I really care whether they can pay attention to an apple when they’re hungry to decide whether to eat the apple, as opposed to considering obscure consequences of how various possible unlikely gods might react to the eating of the apple.
Personal survival makes it a lot easier to please unlikely gods, so eating the apple is preferred.
Eating an apple was meant to be an example of a trivial thing. Inflating it to personal survival misses the point. Eating an apple should be connected to your own personal values concerning hunger and apples, and there should be a way to make a decision about eating an apple or not eating it and being slightly hungry based on your personal values about apples and hunger. If we have to think about unlikely gods to decide whether to eat an apple, something is broken.
For more general situations, some paths to infinity are much more probable than others. For example, perhaps we can build a god [SIAI pointer].
That’s a likely god, not an unlikely god, so it’s a little bit different. Even then, low-probability interactions between eating an apple and the nature of the likely god seem likely to lead to bizarre decision processes about apple-eating, unless you have bounded utilities.
Even then, low-probability interactions between eating an apple and the nature of the likely god seem likely to lead to bizarre decision processes about apple-eating, unless you have bounded utilities.
I don’t see why this is a problem. What causes you to find it so unlikely that our desires could work this way?
Even then, low-probability interactions between eating an apple and the nature of the likely god seem likely to lead to bizarre decision processes about apple-eating, unless you have bounded utilities.
What causes you to find it so unlikely that our desires could work this way?
Pay attention next time you eat something. Do you look at the food and eat what you like or what you think will improve your health, or do you try to prioritize eating the food against sending me money because I might be a god, and against giving all of the other unlikely gods what they might want?
We are human and cannot really do that. With unbounded utilities, there are an absurdly large number of possible ways that an ordinary action can have very low-probability influence on a wide variety of very high-utility things, and you have to take them all into account and balance them properly to do the right thing. If an AI is doing that, I have no confidence at all that it will weigh these things the way I would like, especially given that it’s not likely to search all of the space. Someone who thinks about a million unlikely gods to decide whether to eat an apple is broken. In practice, they won’t be able to do that, and their decision about whether to eat the apple will be driven by whatever unlikely gods have been brought to their attention in the last minute. (As I said before, an improbable minor change to a likely god is an unlikely god, for the purposes of this discussion.)
If utilities are bounded, then the number of alternatives you have to look at doesn’t grow so pathologically large, and you look at the apple to decide whether to eat the apple. The unlikely gods don’t enter into it because you don’t imagine that they can make enough of a difference to outweigh their unlikeliness.
Someone who thinks about a million unlikely gods to decide whether to eat an apple is broken. In practice, they won’t be able to do that, and their decision about whether to eat the apple will be driven by whatever unlikely gods have been brought to their attention in the last minute.
Why can’t they either estimate or prove that eating an apple has more expected utility (by please more gods overall than not eating an apple, say), without iterating over each god and considering them separately? And if for some reason you build an AI that does compute expected utility by brute force iteration of possibilities, then you obviously would not want it to consider only possibilities that “have been brought to their attention in the last minute”. That’s going to lead to trouble no matter what kind of utility function you give it.
(ETA: I think it’s likely that humans do have bounded utility functions (if we can be said to have utility functions at all) but your arguments here are not very good. BTW, have you seen The Lifespan Dilemma?)
Pay attention next time you eat something. Do you look at the food and eat what you like or what you think will improve your health, or do you try to prioritize eating the food against sending me money because I might be a god, and against giving all of the other unlikely gods what they might want?
I would like to do whichever of these two alternatives leads to more utility.
We are human and cannot really do that.
If an AI is doing that, I have no confidence at all that it will weigh these things the way I would like, especially given that it’s not likely to search all of the space.
Are you saying that we shouldn’t maximize utility because it’s too hard?
Someone who thinks about a million unlikely gods to decide whether to eat an apple is broken.
If your actual utility function is unbounded and thinking about a million “unlikely gods” is worth the computational resources that could be spent on likely gods (though you specified that small changes to likely gods are unlikely gods, there is a distinction in that there are not a metaphorical million of them), than that is your actual preference. The utility function is not up for grabs.
Your argument seems to be that maximizing an unbounded utility function is impractical, so we should maximize a bounded utility function instead. I find it improbable that you would make this argument, so, if I am missing anything, please clarify.
Yes, the utility function is not up for grabs, but introspection doesn’t tell you what it is either. In particular, the statement “endoself acts approximately consistently with utility function U” is an empirical statement for any given U (and any particular notion of “approximately”, but let’s skip that part for now). I believe I have provided fine arguments that you are not acting approximately consistently with an unbounded utility function, and that you will never be able to do so. If those arguments are valid, and you say you have an unbounded utility function, then you are wrong.
If those arguments are valid, and you say you want to have an unbounded utility function, then you’re wanting something impossible because you falsely believe it to be possible. The best I could do in that case if I were helping you would be to give you what you would want if you had true beliefs. I don’t know what that would be. What would you want from an unbounded utility function that you couldn’t get if the math turned out so that only bounded utility functions can be used in a decision procedure?
you specified that small changes to likely gods are unlikely gods, there is a distinction in that there are not a metaphorical million of them
There are many paths by which small actions taken today might in unlikely ways influence the details of how a likely god is built. If those paths have infinite utility, you have to analyze them to decide what to do.
I believe I have provided fine arguments that you are not acting approximately consistently with an unbounded utility function, and that you will never be able to do so.
I am currently researching logical uncertainty. I believe that the increased chance of FAI due to this research makes it the best way to act according to my utility function, taking into account the limits to my personal rationality (part of this is personal; I am particularly interested in logical uncertainty right now, so I am more likely to make progress in it than on other problems). This is because, among other things, an FAI will be far better at understanding the difficulties associated with unbounded utility functions than I am.
If those arguments are valid, and you say you want to have an unbounded utility function, then you’re wanting something impossible because you falsely believe it to be possible.
You have not demonstrated it to be impossible, you have just shown that the most obvious approach to it does not work. Given how questionable some of the axioms we use are, this is not particularly surprising.
What would you want from an unbounded utility function that you couldn’t get if the math turned out so that only bounded utility functions can be used in a decision procedure?
There are many paths by which small actions taken today might in unlikely ways influence the details of how a likely god is built. If those paths have infinite utility, you have to analyze them to decide what to do.
Some paths are far more likely than others. Actively researching FAI in a way that is unlikely to significantly increase the probability of UFAI provides far more expected utility than unlikely ways to help the development of FAI.
What would you want from an unbounded utility function that you couldn’t get if the math turned out so that only bounded utility functions can be used in a decision procedure?
An actual description of my preferences. I am unsure whether my utility function is actually unbounded but I find it probable that, for example, my utility function is linear in people. I don’t want to rule this out just because that current framework is insufficient for it.
Predicting your preferences requires specifying both the utility function and the framework, so offering a utility function without the framework as an explanation for your preferences does not actually explain them. I actually don’t know if my question was hypothetical or not. Do we have a decision procedure that gives reasonable results for an unbounded utility function?
The phrase “rule this out” seems interesting here. At any given time, you’ll have a set of explanations for your behavior. That doesn’t rule out coming up with better explanations later. Does the best explanation you have for your preferences that works with a known decision theory have bounded utility?
Perhaps I see what’s going on here—people who want unbounded utility are feeling loss when they imagine giving that up that unbounded goodness in order to avoid bugs like the one described in the OP. I, on the other hand, feel loss when people dither over difficult math problems when the actual issues confronting us have nothing to do with difficult math. Specifically, dealing effectively with the default future, in which one or more corporations make AI’s that optimize for something having no connection to the preferences of any individual human.
Do we have a decision procedure that gives reasonable results for an unbounded utility function?
Not one compatible with a Solomonoff prior. I agree that a utility function alone is not a full description of preferences.
Does the best explanation you have for your preferences that works with a known decision theory have bounded utility?
The best explanation that I have for my preferences does not, AFAICT, work with any known decision theory. However, I know enough of what such a decision theory would look like if it were possible to say that it would not have bounded utility.
I, on the other hand, feel loss when people dither over difficult math problems when the actual issues confronting us have nothing to do with difficult math.
I disagree that I am doing such. Whether or not the math is relevant to the issue is a question of values, not fact. Your estimates of your values do not find the math relevant; my estimates of my values do.
downvoted because you actually said “I would like to do whichever of these two alternatives leads to more utility.”
A) no one or almost no one thinks this way, and advice based on this sort of thinking is useless to almost everyone.
B) The entire point of the original post was that, if you try to do this, then you immediately get completely taken over by consideration of any gods you can imagine. When you say that thinking about unlikely gods is not “worth” the computational resources, you are sidestepping the very issue we are discussing. You have already decided it’s not worth thinking about tiny probabilities of huge returns.
I think he actually IS making the argument that you assign a low probability to, but instead of dismissing it I think it’s actually extremely important to decide whether to take certain courses based on how practical they are. The entire original purpose of this community is research into AI, and while you can’t choose your own utility function, you can choose an AI’s. If this problem is practically insoluble, then we should design AIs with only bounded utility functions.
downvoted because you actually said “I would like to do whichever of these two alternatives leads to more utility.”
Tim seemed to be implying that it would be absurd for unlikely gods to be the most important motive for determining how to act, but I did not see how anything that he said showed that doing so is actually a bad idea.
When you say that thinking about unlikely gods is not “worth” the computational resources, you are sidestepping the very issue we are discussing.
What? I did not say that; I said that thinking about unlikely gods might just be one’s actual preference. I also pointed out that Tim did not prove that unlikely gods are more important than likely gods, so one who accepts most of his argument might still not motivated by “a million unlikely gods”.
Eating an apple was meant to be an example of a trivial thing. Inflating it to personal survival misses the point. Eating an apple should be connected to your own personal values concerning hunger and apples, and there should be a way to make a decision about eating an apple or not eating it and being slightly hungry based on your personal values about apples and hunger. If we have to think about unlikely gods to decide whether to eat an apple, something is broken.
That’s a likely god, not an unlikely god, so it’s a little bit different. Even then, low-probability interactions between eating an apple and the nature of the likely god seem likely to lead to bizarre decision processes about apple-eating, unless you have bounded utilities.
I don’t see why this is a problem. What causes you to find it so unlikely that our desires could work this way?
Pay attention next time you eat something. Do you look at the food and eat what you like or what you think will improve your health, or do you try to prioritize eating the food against sending me money because I might be a god, and against giving all of the other unlikely gods what they might want?
We are human and cannot really do that. With unbounded utilities, there are an absurdly large number of possible ways that an ordinary action can have very low-probability influence on a wide variety of very high-utility things, and you have to take them all into account and balance them properly to do the right thing. If an AI is doing that, I have no confidence at all that it will weigh these things the way I would like, especially given that it’s not likely to search all of the space. Someone who thinks about a million unlikely gods to decide whether to eat an apple is broken. In practice, they won’t be able to do that, and their decision about whether to eat the apple will be driven by whatever unlikely gods have been brought to their attention in the last minute. (As I said before, an improbable minor change to a likely god is an unlikely god, for the purposes of this discussion.)
If utilities are bounded, then the number of alternatives you have to look at doesn’t grow so pathologically large, and you look at the apple to decide whether to eat the apple. The unlikely gods don’t enter into it because you don’t imagine that they can make enough of a difference to outweigh their unlikeliness.
Why can’t they either estimate or prove that eating an apple has more expected utility (by please more gods overall than not eating an apple, say), without iterating over each god and considering them separately? And if for some reason you build an AI that does compute expected utility by brute force iteration of possibilities, then you obviously would not want it to consider only possibilities that “have been brought to their attention in the last minute”. That’s going to lead to trouble no matter what kind of utility function you give it.
(ETA: I think it’s likely that humans do have bounded utility functions (if we can be said to have utility functions at all) but your arguments here are not very good. BTW, have you seen The Lifespan Dilemma?)
I would like to do whichever of these two alternatives leads to more utility.
Are you saying that we shouldn’t maximize utility because it’s too hard?
If your actual utility function is unbounded and thinking about a million “unlikely gods” is worth the computational resources that could be spent on likely gods (though you specified that small changes to likely gods are unlikely gods, there is a distinction in that there are not a metaphorical million of them), than that is your actual preference. The utility function is not up for grabs.
Your argument seems to be that maximizing an unbounded utility function is impractical, so we should maximize a bounded utility function instead. I find it improbable that you would make this argument, so, if I am missing anything, please clarify.
Yes, the utility function is not up for grabs, but introspection doesn’t tell you what it is either. In particular, the statement “endoself acts approximately consistently with utility function U” is an empirical statement for any given U (and any particular notion of “approximately”, but let’s skip that part for now). I believe I have provided fine arguments that you are not acting approximately consistently with an unbounded utility function, and that you will never be able to do so. If those arguments are valid, and you say you have an unbounded utility function, then you are wrong.
If those arguments are valid, and you say you want to have an unbounded utility function, then you’re wanting something impossible because you falsely believe it to be possible. The best I could do in that case if I were helping you would be to give you what you would want if you had true beliefs. I don’t know what that would be. What would you want from an unbounded utility function that you couldn’t get if the math turned out so that only bounded utility functions can be used in a decision procedure?
There are many paths by which small actions taken today might in unlikely ways influence the details of how a likely god is built. If those paths have infinite utility, you have to analyze them to decide what to do.
I am currently researching logical uncertainty. I believe that the increased chance of FAI due to this research makes it the best way to act according to my utility function, taking into account the limits to my personal rationality (part of this is personal; I am particularly interested in logical uncertainty right now, so I am more likely to make progress in it than on other problems). This is because, among other things, an FAI will be far better at understanding the difficulties associated with unbounded utility functions than I am.
You have not demonstrated it to be impossible, you have just shown that the most obvious approach to it does not work. Given how questionable some of the axioms we use are, this is not particularly surprising.
An actual description of my preferences. I am unsure whether my utility function is actually unbounded but I find it probable that, for example, my utility function is linear in people. I don’t want to rule this out just because that current framework is insufficient for it.
Some paths are far more likely than others. Actively researching FAI in a way that is unlikely to significantly increase the probability of UFAI provides far more expected utility than unlikely ways to help the development of FAI.
Predicting your preferences requires specifying both the utility function and the framework, so offering a utility function without the framework as an explanation for your preferences does not actually explain them. I actually don’t know if my question was hypothetical or not. Do we have a decision procedure that gives reasonable results for an unbounded utility function?
The phrase “rule this out” seems interesting here. At any given time, you’ll have a set of explanations for your behavior. That doesn’t rule out coming up with better explanations later. Does the best explanation you have for your preferences that works with a known decision theory have bounded utility?
Perhaps I see what’s going on here—people who want unbounded utility are feeling loss when they imagine giving that up that unbounded goodness in order to avoid bugs like the one described in the OP. I, on the other hand, feel loss when people dither over difficult math problems when the actual issues confronting us have nothing to do with difficult math. Specifically, dealing effectively with the default future, in which one or more corporations make AI’s that optimize for something having no connection to the preferences of any individual human.
Not one compatible with a Solomonoff prior. I agree that a utility function alone is not a full description of preferences.
The best explanation that I have for my preferences does not, AFAICT, work with any known decision theory. However, I know enough of what such a decision theory would look like if it were possible to say that it would not have bounded utility.
I disagree that I am doing such. Whether or not the math is relevant to the issue is a question of values, not fact. Your estimates of your values do not find the math relevant; my estimates of my values do.
downvoted because you actually said “I would like to do whichever of these two alternatives leads to more utility.”
A) no one or almost no one thinks this way, and advice based on this sort of thinking is useless to almost everyone.
B) The entire point of the original post was that, if you try to do this, then you immediately get completely taken over by consideration of any gods you can imagine. When you say that thinking about unlikely gods is not “worth” the computational resources, you are sidestepping the very issue we are discussing. You have already decided it’s not worth thinking about tiny probabilities of huge returns.
I think he actually IS making the argument that you assign a low probability to, but instead of dismissing it I think it’s actually extremely important to decide whether to take certain courses based on how practical they are. The entire original purpose of this community is research into AI, and while you can’t choose your own utility function, you can choose an AI’s. If this problem is practically insoluble, then we should design AIs with only bounded utility functions.
Tim seemed to be implying that it would be absurd for unlikely gods to be the most important motive for determining how to act, but I did not see how anything that he said showed that doing so is actually a bad idea.
What? I did not say that; I said that thinking about unlikely gods might just be one’s actual preference. I also pointed out that Tim did not prove that unlikely gods are more important than likely gods, so one who accepts most of his argument might still not motivated by “a million unlikely gods”.