Speaking for myself, I sometimes use “EU maximization” as shorthand for one of the following concepts, depending on context:
The eventual intellectual descendant of EU maximization, i.e., the decision theory or theory of rationality that future philosophers will eventually accept as correct or ideal or normative, which presumably will have some kind of connection (even if only historical) to EU maximization.
The eventual decision procedure of a reflectively stable superintelligence.
The decision procedure of a very capable consequentialist AI, even if it’s not quite reflectively stable yet.
Hmm, I just did a search of my own LW content, and can’t actually find any instances of myself doing this, which makes me wonder why I was tempted to type the above. Perhaps what I actually do is if I see someone else mention “EU maximization”, I mentally steelman their argument by replacing the concept with one of the three above, if anyone of them would make a sensible substitution.
Do you have any actual examples of anyone talking about EU maximization lately, in connection with AI risk?
I note that EU maximization has this baggage of never strictly preferring a lottery over outcomes to the component outcomes, and you steelmen appear to me to not carry that baggage. I think that baggage is actually doing work in some people’s reasoning and intuitions.
I think you are referring to the case where an agent wishes to be unpredictable in an adversarial situation, right? (I genuinely do not feel confident I understand what you said.)
If so, isn’t this lottery on a different, let’s say ontological, level, instead of the level of “lotteries” that define its utility?
I parsed the Rob Bensinger tweet I linked in the OP as being about expected utility maximising when I read it, but others have pointed out that wasn’t necessarily a fair reading.
Speaking for myself, I sometimes use “EU maximization” as shorthand for one of the following concepts, depending on context:
The eventual intellectual descendant of EU maximization, i.e., the decision theory or theory of rationality that future philosophers will eventually accept as correct or ideal or normative, which presumably will have some kind of connection (even if only historical) to EU maximization.
The eventual decision procedure of a reflectively stable superintelligence.
The decision procedure of a very capable consequentialist AI, even if it’s not quite reflectively stable yet.
Hmm, I just did a search of my own LW content, and can’t actually find any instances of myself doing this, which makes me wonder why I was tempted to type the above. Perhaps what I actually do is if I see someone else mention “EU maximization”, I mentally steelman their argument by replacing the concept with one of the three above, if anyone of them would make a sensible substitution.
Do you have any actual examples of anyone talking about EU maximization lately, in connection with AI risk?
I note that EU maximization has this baggage of never strictly preferring a lottery over outcomes to the component outcomes, and you steelmen appear to me to not carry that baggage. I think that baggage is actually doing work in some people’s reasoning and intuitions.
Do you have any examples of this?
Hmm, examples are hard. Maybe the intuitions contribute to concept of edge instantiation?
I think you are referring to the case where an agent wishes to be unpredictable in an adversarial situation, right? (I genuinely do not feel confident I understand what you said.)
If so, isn’t this lottery on a different, let’s say ontological, level, instead of the level of “lotteries” that define its utility?
I parsed the Rob Bensinger tweet I linked in the OP as being about expected utility maximising when I read it, but others have pointed out that wasn’t necessarily a fair reading.