And, speaking of Conway games, the closely related Conway numbers (“surreal numbers”, as they are usually called) show that even if your preferences are totally ordered it’s not obvious that they can be embedded into the real numbers.
If you have lexically ordered ‘orders’ of utility, only the highest order will ever affect your actions in non-toy situations, and you might as well use reals.
I think that’s debatable. For instance, consider Eliezer’s “torture versus dust specks” question from way back on OB. (In case you weren’t reading OB then or have forgotten: let N be a vastly unimaginably huge number (Eliezer chose 3^^^3 in Knuth arrow notation), and ask: which is worse, for one person to be tortured horribly for 50 years or for N people each to get a small speck of dust in their eye, just enough to be marginally annoying, but to suffer no longer-term consequences from it? I claim that having separate “orders” of utility is at most as irrational as choosing SPECKS rather than TORTURE, and that it’s at least arguable that SPECKS is a defensible answer.
I claim that having separate “orders” of utility is at most as irrational...
The point isn’t about the (ir)rationality of separate “orders” of utility. It’s a “without loss of generality” argument. Preferences not found at the highest order are effectively irrelevant, so you don’t lose any expressive power by restricting yourself to the reals.
Er, sorry, I was unclear. (I wrote unclearly because I wasn’t thinking clearly enough. It’s annoying how that happens.) So, the point I was trying to make but didn’t actually get around to writing down because I forgot about it while writing down what I did :-) is that those people for whom dust specks and torture are incommensurable—which I think they have to be, to prefer 3^^^3.SPECK to 1.TORTURE—don’t, so far as I can tell, generally spend their entire lives estimating how many people are going to get tortured-or-worse on account of their actions, neither do they entirely ignore minor inconveniences; so it doesn’t seem to be the case that having that sort of utility function implies ignoring everything but the highest order.
[EDITED above, about a day after posting, to fix a formatting glitch that I hadn’t noticed before.]
Arguably it would do if those people were perfectly consistent—one of the more convincing arguments for preferring TORTURE to SPECKS consists of exhibiting a series of steps between SPECK and TORTURE of length, say, at most 100 in which no step appears to involve a worse than, say, 100:1 difference in badness, so maybe preferring TORTURE to SPECKS almost always involves intransitivity or something like that. And maybe some similar charge could be brought against anyone who has separate “orders” but still gives any consideration to the lower ones. Hence my remark that the one doesn’t seem more irrational than the other.
If you have lexically ordered ‘orders’ of utility, only the highest order will ever affect your actions in non-toy situations, and you might as well use reals.
I think that’s debatable. For instance, consider Eliezer’s “torture versus dust specks” question from way back on OB. (In case you weren’t reading OB then or have forgotten: let N be a vastly unimaginably huge number (Eliezer chose 3^^^3 in Knuth arrow notation), and ask: which is worse, for one person to be tortured horribly for 50 years or for N people each to get a small speck of dust in their eye, just enough to be marginally annoying, but to suffer no longer-term consequences from it? I claim that having separate “orders” of utility is at most as irrational as choosing SPECKS rather than TORTURE, and that it’s at least arguable that SPECKS is a defensible answer.
The point isn’t about the (ir)rationality of separate “orders” of utility. It’s a “without loss of generality” argument. Preferences not found at the highest order are effectively irrelevant, so you don’t lose any expressive power by restricting yourself to the reals.
Er, sorry, I was unclear. (I wrote unclearly because I wasn’t thinking clearly enough. It’s annoying how that happens.) So, the point I was trying to make but didn’t actually get around to writing down because I forgot about it while writing down what I did :-) is that those people for whom dust specks and torture are incommensurable—which I think they have to be, to prefer 3^^^3.SPECK to 1.TORTURE—don’t, so far as I can tell, generally spend their entire lives estimating how many people are going to get tortured-or-worse on account of their actions, neither do they entirely ignore minor inconveniences; so it doesn’t seem to be the case that having that sort of utility function implies ignoring everything but the highest order.
[EDITED above, about a day after posting, to fix a formatting glitch that I hadn’t noticed before.]
Arguably it would do if those people were perfectly consistent—one of the more convincing arguments for preferring TORTURE to SPECKS consists of exhibiting a series of steps between SPECK and TORTURE of length, say, at most 100 in which no step appears to involve a worse than, say, 100:1 difference in badness, so maybe preferring TORTURE to SPECKS almost always involves intransitivity or something like that. And maybe some similar charge could be brought against anyone who has separate “orders” but still gives any consideration to the lower ones. Hence my remark that the one doesn’t seem more irrational than the other.