This is a cop out. Obviously that specific situation can’t occur in reality, that’s not the point. If your decision algorithm fails in some extreme cases, at least confess that it’s not universal.
What does it mean for a decision algorithm to fail? I’ll give an answer later, but here I’ll point out that I do endorse that multiplication of reals is universal—that is, I don’t think multiplication breaks down when the numbers get extreme enough.
Same thing. Omega’s ability and honesty are premises.
And an unbelievable premise leads to an unbelievable conclusion. Don’t say that logic has broken down because someone gave the syllogism “All men are immortal, Socrates is a man, therefore Socrates is still alive.”
The point of the thought experiment is just to show that EU is required to trade away huge amounts of outcome-space for really good but improbable outcomes. This is a good strategy if you plan on making an infinite number of bets, but horrible if you don’t expect to live forever.
How does [logic work]? Eliezer puts it better than I can:
The power of logic is that it relates models and statements. … And here is the power of logic: For each syntactic step we do on our statements, we preserve the match to any model. In any model where our old collection of statements was true, the new statement will also be true. We don’t have to check all possible conforming models to see if the new statement is true in all of them. We can trust certain syntactic steps in general—not to produce truth, but to preserve truth.
EU is not “required” to trade away huge amounts of outcome-space for really good but improbable outcomes. EU applies preference models to novel situations, not to produce preferences but to preserve them. If you gave EU a preference model that matched your preferences, it will preserve the match and give you actions that best satisfy your preferences in underneath the uncertainty model of the universe you gave it.
And if it’s not true that you would trade away huge amounts of outcome-space for really good but improbable outcomes, this is a fact about your preference model that EU preserves! Remember, EU preference models map lists of outcomes to classes of lists of real numbers, but the inverse mapping is not guaranteed to have support over the entire reals.
I think a decision algorithm fails if it makes you predictably worse off than an alternative algorithm, and the chief ways to do so are 1) to do the math wrong and be inconsistent and 2) to make it more costly to express your preferences or world-model.
I don’t get your drug research analogy.
We have lots of hypotheses about low-probability, high-payout options, and if humans make mistakes, it is probably by overestimating the probability of low-probability events and overestimating how much we’ll enjoy the high payouts, both of which make us more likely to pursue those paths than a rational version of ourselves.
So it seems to me that if we have an algorithm that can correctly manage the budget of a pharmaceutical corporation, balancing R&D and marketing and production and so on, that requires solving this philosophical problem. But we have the mathematical tools to correctly manage the budget of a pharmaceutical corporation given a world-model, which says to me we should turn our attention to getting more precise and powerful world-models.
What does it mean for a decision algorithm to fail?
When it makes decisions that are undesirable. There is no point deciding to run a decision algorithm which is perfectly consistent but results in outcomes you don’t want.
In the case of the Omega’s-life-tickets scenario, one could argue it fails in an objective sense since it will never stop buying tickets until it dies. But that wasn’t even the point I was trying to make.
And an unbelievable premise leads to an unbelievable conclusion.
I don’t know if there is a name for this fallacy but there should be. Where someone objects to the premises of a hypothetical situation intended just to demonstrate a point. E.g. people who refuse to answer the trolley dilemma and instead say “but that will probably never happen!” It’s very frustrating.
EU is not “required” to trade away huge amounts of outcome-space for really good but improbable outcomes. EU applies preference models to novel situations, not to produce preferences but to preserve them. If you gave EU a preference model that matched your preferences, it will preserve the match and give you actions that best satisfy your preferences in underneath the uncertainty model of the universe you gave it.
This is very subtle circular reasoning. If you assume your goal is to maximize the expected value some utility function, then maximizing expected utility can do that if you specify the right utility function.
What I’ve been saying from the very beginning is that there isn’t any reason to believe there is any utility function that will produce desirable outcomes if fed to an expected utility maximizer.
I think a decision algorithm fails if it makes you predictably worse off than an alternative algorithm
Even if you are an EU maximizer, EU will make you “predictably” worse off, as in the majority of cases you will be worse off. A true EU maximizer doesn’t care so long as the utility of the very low probability outcomes is high enough.
There are good and bad reasons to fight the hypothetical. When it comes to these particular problems, though, the objections I’ve given are my true objections. The reason I’d only pay a tiny amount of money for the gamble in the St. Petersburg Paradox is that there is only so much financial value that the house can give up. One of the reasons I’m sure this is my true objection is because the richer the house, the more I would pay for such a gamble. (Because there are no infinitely rich houses, there is no one I would pay an infinite amount to for such a gamble.)
This is very subtle circular reasoning. If you assume your goal is to maximize the expected value some utility function, then maximizing expected utility can do that if you specify the right utility function.
I’m not sure why you think it’s subtle—I started off this conversation with:
This might sound silly, but it’s deeper than it looks: the reason why we use the expected value of utility (i.e. means) to determine the best of a set of gambles is because utility is defined as the thing that you maximize the expected value of.
But I don’t think it’s quite right to call it “circular,” for roughly the same reasons I don’t think it’s right to call logic “circular.”
What I’ve been saying from the very beginning is that there isn’t any reason to believe there is any utility function that will produce desirable outcomes if fed to an expected utility maximizer.
To make sure we’re talking about the same thing, I think an expected utility maximizer (EUM) is something that takes both a function u(O) that maps outcomes to utilities, a function p(A->O) that maps actions to probabilities of outcomes, and a set of possible actions, and then finds the action out of all possible A that has the maximum weighted sum of u(O)p(A->O) over all possible O.
So far, you have not been arguing that every possible EUM leads to pathological outcomes; you have been exhibiting particular combinations of u(O) and p(A->O) that lead to pathological outcomes, and I have been responding with “have you tried not using those u(O)s and p(A->O)s?”.
It doesn’t seem to me that this conversation is producing value for either of us, which suggests that we should either restart the conversation, take it to PMs, or drop it.
What does it mean for a decision algorithm to fail? I’ll give an answer later, but here I’ll point out that I do endorse that multiplication of reals is universal—that is, I don’t think multiplication breaks down when the numbers get extreme enough.
And an unbelievable premise leads to an unbelievable conclusion. Don’t say that logic has broken down because someone gave the syllogism “All men are immortal, Socrates is a man, therefore Socrates is still alive.”
How does [logic work]? Eliezer puts it better than I can:
EU is not “required” to trade away huge amounts of outcome-space for really good but improbable outcomes. EU applies preference models to novel situations, not to produce preferences but to preserve them. If you gave EU a preference model that matched your preferences, it will preserve the match and give you actions that best satisfy your preferences in underneath the uncertainty model of the universe you gave it.
And if it’s not true that you would trade away huge amounts of outcome-space for really good but improbable outcomes, this is a fact about your preference model that EU preserves! Remember, EU preference models map lists of outcomes to classes of lists of real numbers, but the inverse mapping is not guaranteed to have support over the entire reals.
I think a decision algorithm fails if it makes you predictably worse off than an alternative algorithm, and the chief ways to do so are 1) to do the math wrong and be inconsistent and 2) to make it more costly to express your preferences or world-model.
We have lots of hypotheses about low-probability, high-payout options, and if humans make mistakes, it is probably by overestimating the probability of low-probability events and overestimating how much we’ll enjoy the high payouts, both of which make us more likely to pursue those paths than a rational version of ourselves.
So it seems to me that if we have an algorithm that can correctly manage the budget of a pharmaceutical corporation, balancing R&D and marketing and production and so on, that requires solving this philosophical problem. But we have the mathematical tools to correctly manage the budget of a pharmaceutical corporation given a world-model, which says to me we should turn our attention to getting more precise and powerful world-models.
When it makes decisions that are undesirable. There is no point deciding to run a decision algorithm which is perfectly consistent but results in outcomes you don’t want.
In the case of the Omega’s-life-tickets scenario, one could argue it fails in an objective sense since it will never stop buying tickets until it dies. But that wasn’t even the point I was trying to make.
I don’t know if there is a name for this fallacy but there should be. Where someone objects to the premises of a hypothetical situation intended just to demonstrate a point. E.g. people who refuse to answer the trolley dilemma and instead say “but that will probably never happen!” It’s very frustrating.
This is very subtle circular reasoning. If you assume your goal is to maximize the expected value some utility function, then maximizing expected utility can do that if you specify the right utility function.
What I’ve been saying from the very beginning is that there isn’t any reason to believe there is any utility function that will produce desirable outcomes if fed to an expected utility maximizer.
Even if you are an EU maximizer, EU will make you “predictably” worse off, as in the majority of cases you will be worse off. A true EU maximizer doesn’t care so long as the utility of the very low probability outcomes is high enough.
One name is fighting the hypothetical, and it’s worth taking a look at the least convenient possible world and the true rejection as well.
There are good and bad reasons to fight the hypothetical. When it comes to these particular problems, though, the objections I’ve given are my true objections. The reason I’d only pay a tiny amount of money for the gamble in the St. Petersburg Paradox is that there is only so much financial value that the house can give up. One of the reasons I’m sure this is my true objection is because the richer the house, the more I would pay for such a gamble. (Because there are no infinitely rich houses, there is no one I would pay an infinite amount to for such a gamble.)
I’m not sure why you think it’s subtle—I started off this conversation with:
But I don’t think it’s quite right to call it “circular,” for roughly the same reasons I don’t think it’s right to call logic “circular.”
To make sure we’re talking about the same thing, I think an expected utility maximizer (EUM) is something that takes both a function u(O) that maps outcomes to utilities, a function p(A->O) that maps actions to probabilities of outcomes, and a set of possible actions, and then finds the action out of all possible A that has the maximum weighted sum of u(O)p(A->O) over all possible O.
So far, you have not been arguing that every possible EUM leads to pathological outcomes; you have been exhibiting particular combinations of u(O) and p(A->O) that lead to pathological outcomes, and I have been responding with “have you tried not using those u(O)s and p(A->O)s?”.
It doesn’t seem to me that this conversation is producing value for either of us, which suggests that we should either restart the conversation, take it to PMs, or drop it.