Yes, I am aware of that. The biggest trouble, as you have elaborately explained in your post, is that people think they can perform mathematical operations in VNM-utility-space to calculate utilities they have not explicitly defined in their system of ethics. I believe Eliezer has fallen into this trap, the sequences are full of that kind of thinking (e.g. torture vs dust specks) and while I realize it’s not supposed to be taken literally, “shut up and multiply” is symptomatic.
Another problem is that you can only use VNM when talking about complete world states. A day where you get a tasty sandwich might be better than a normal day, or it might not be, depending on the world state. If you know there’s a wizard who’ll give you immortality for $1, you’ll chose $1 over any probability<1 of $2, and if the wizard wants $2, the opposite applies.
VNM isn’t bad, it’s just far, far, far too limited. It’s somewhat useful when probabilities are involved, but otherwise it’s literally just the concept of well-ordering your options by preferability.
Assuming you assign utility to lifetime as a function of life quality in such a way that for any constant quality longer life has strictly higher (or lower) utility than shorter life, then either you can’t assign any utility to actually infinite immortality, or you can’t differentiate between higher-quality and lower-quality immortality, or you can’t represent utility as a real number.
Turns out this is not actually true: 1 day is 1, 2 days is 1.5, 3 days is 1.75, etc, immortality is 2, and then you can add quality. Not very surprising in fact, considering immortality is effectively infinity and |ℕ| < |ℝ|. Still, I’m pretty sure the set of all possible world states is of higher cardinality than ℝ, so...
(Also it’s a good illustration why simply assigning utility to 1 day of life and then scaling up is not a bright idea.)
Another problem is that you can only use VNM when talking about complete world states.
You can talk about probability distributions over world-states as well. When I say “tasty sandwich day minus normal day” I mean to refer to the expected marginal utility of the sandwich, including the possibilities with wizards and stuff. This simplifies things a bit, but goes to hell as soon as you include probability updating, or actually have to find that value.
Yes, I am aware of that. The biggest trouble, as you have elaborately explained in your post, is that people think they can perform mathematical operations in VNM-utility-space to calculate utilities they have not explicitly defined in their system of ethics. I believe Eliezer has fallen into this trap, the sequences are full of that kind of thinking (e.g. torture vs dust specks) and while I realize it’s not supposed to be taken literally, “shut up and multiply” is symptomatic.
Another problem is that you can only use VNM when talking about complete world states. A day where you get a tasty sandwich might be better than a normal day, or it might not be, depending on the world state. If you know there’s a wizard who’ll give you immortality for $1, you’ll chose $1 over any probability<1 of $2, and if the wizard wants $2, the opposite applies.
VNM isn’t bad, it’s just far, far, far too limited. It’s somewhat useful when probabilities are involved, but otherwise it’s literally just the concept of well-ordering your options by preferability.
Turns out this is not actually true: 1 day is 1, 2 days is 1.5, 3 days is 1.75, etc, immortality is 2, and then you can add quality. Not very surprising in fact, considering immortality is effectively infinity and |ℕ| < |ℝ|. Still, I’m pretty sure the set of all possible world states is of higher cardinality than ℝ, so...
(Also it’s a good illustration why simply assigning utility to 1 day of life and then scaling up is not a bright idea.)
You can talk about probability distributions over world-states as well. When I say “tasty sandwich day minus normal day” I mean to refer to the expected marginal utility of the sandwich, including the possibilities with wizards and stuff. This simplifies things a bit, but goes to hell as soon as you include probability updating, or actually have to find that value.