I believe I have provided fine arguments that you are not acting approximately consistently with an unbounded utility function, and that you will never be able to do so.
I am currently researching logical uncertainty. I believe that the increased chance of FAI due to this research makes it the best way to act according to my utility function, taking into account the limits to my personal rationality (part of this is personal; I am particularly interested in logical uncertainty right now, so I am more likely to make progress in it than on other problems). This is because, among other things, an FAI will be far better at understanding the difficulties associated with unbounded utility functions than I am.
If those arguments are valid, and you say you want to have an unbounded utility function, then you’re wanting something impossible because you falsely believe it to be possible.
You have not demonstrated it to be impossible, you have just shown that the most obvious approach to it does not work. Given how questionable some of the axioms we use are, this is not particularly surprising.
What would you want from an unbounded utility function that you couldn’t get if the math turned out so that only bounded utility functions can be used in a decision procedure?
There are many paths by which small actions taken today might in unlikely ways influence the details of how a likely god is built. If those paths have infinite utility, you have to analyze them to decide what to do.
Some paths are far more likely than others. Actively researching FAI in a way that is unlikely to significantly increase the probability of UFAI provides far more expected utility than unlikely ways to help the development of FAI.
What would you want from an unbounded utility function that you couldn’t get if the math turned out so that only bounded utility functions can be used in a decision procedure?
An actual description of my preferences. I am unsure whether my utility function is actually unbounded but I find it probable that, for example, my utility function is linear in people. I don’t want to rule this out just because that current framework is insufficient for it.
Predicting your preferences requires specifying both the utility function and the framework, so offering a utility function without the framework as an explanation for your preferences does not actually explain them. I actually don’t know if my question was hypothetical or not. Do we have a decision procedure that gives reasonable results for an unbounded utility function?
The phrase “rule this out” seems interesting here. At any given time, you’ll have a set of explanations for your behavior. That doesn’t rule out coming up with better explanations later. Does the best explanation you have for your preferences that works with a known decision theory have bounded utility?
Perhaps I see what’s going on here—people who want unbounded utility are feeling loss when they imagine giving that up that unbounded goodness in order to avoid bugs like the one described in the OP. I, on the other hand, feel loss when people dither over difficult math problems when the actual issues confronting us have nothing to do with difficult math. Specifically, dealing effectively with the default future, in which one or more corporations make AI’s that optimize for something having no connection to the preferences of any individual human.
Do we have a decision procedure that gives reasonable results for an unbounded utility function?
Not one compatible with a Solomonoff prior. I agree that a utility function alone is not a full description of preferences.
Does the best explanation you have for your preferences that works with a known decision theory have bounded utility?
The best explanation that I have for my preferences does not, AFAICT, work with any known decision theory. However, I know enough of what such a decision theory would look like if it were possible to say that it would not have bounded utility.
I, on the other hand, feel loss when people dither over difficult math problems when the actual issues confronting us have nothing to do with difficult math.
I disagree that I am doing such. Whether or not the math is relevant to the issue is a question of values, not fact. Your estimates of your values do not find the math relevant; my estimates of my values do.
I am currently researching logical uncertainty. I believe that the increased chance of FAI due to this research makes it the best way to act according to my utility function, taking into account the limits to my personal rationality (part of this is personal; I am particularly interested in logical uncertainty right now, so I am more likely to make progress in it than on other problems). This is because, among other things, an FAI will be far better at understanding the difficulties associated with unbounded utility functions than I am.
You have not demonstrated it to be impossible, you have just shown that the most obvious approach to it does not work. Given how questionable some of the axioms we use are, this is not particularly surprising.
An actual description of my preferences. I am unsure whether my utility function is actually unbounded but I find it probable that, for example, my utility function is linear in people. I don’t want to rule this out just because that current framework is insufficient for it.
Some paths are far more likely than others. Actively researching FAI in a way that is unlikely to significantly increase the probability of UFAI provides far more expected utility than unlikely ways to help the development of FAI.
Predicting your preferences requires specifying both the utility function and the framework, so offering a utility function without the framework as an explanation for your preferences does not actually explain them. I actually don’t know if my question was hypothetical or not. Do we have a decision procedure that gives reasonable results for an unbounded utility function?
The phrase “rule this out” seems interesting here. At any given time, you’ll have a set of explanations for your behavior. That doesn’t rule out coming up with better explanations later. Does the best explanation you have for your preferences that works with a known decision theory have bounded utility?
Perhaps I see what’s going on here—people who want unbounded utility are feeling loss when they imagine giving that up that unbounded goodness in order to avoid bugs like the one described in the OP. I, on the other hand, feel loss when people dither over difficult math problems when the actual issues confronting us have nothing to do with difficult math. Specifically, dealing effectively with the default future, in which one or more corporations make AI’s that optimize for something having no connection to the preferences of any individual human.
Not one compatible with a Solomonoff prior. I agree that a utility function alone is not a full description of preferences.
The best explanation that I have for my preferences does not, AFAICT, work with any known decision theory. However, I know enough of what such a decision theory would look like if it were possible to say that it would not have bounded utility.
I disagree that I am doing such. Whether or not the math is relevant to the issue is a question of values, not fact. Your estimates of your values do not find the math relevant; my estimates of my values do.