Robin, of course it’s not obvious. It’s only an obvious conclusion if the global utility function from the dust specks is an additive function of the individual utilities, and since we know that utility functions must be bounded to avoid Dutch books, we know that the global utility function cannot possibly be additive—otherwise you could break the bound by choosing a large enough number of people (say, 3^^^3).
From a more metamathematical perspective, you can also question whether 3^^3 is a number at all. It’s perfectly straightforward to construct a perfectly consistent mathematics that rejects the axiom of infinity. Besides the philosophical justification for ultrafinitism (ie, infinite sets don’t really exist), these theories corresponds to various notions of bounded computation (such as logspace or polytime). This is a natural requirement, if we want to require moral judgements to be made quickly enough to be relevant to decision making—and that rules out seriously computing with numbers like 3^^^3.
Robin, of course it’s not obvious. It’s only an obvious conclusion if the global utility function from the dust specks is an additive function of the individual utilities, and since we know that utility functions must be bounded to avoid Dutch books, we know that the global utility function cannot possibly be additive—otherwise you could break the bound by choosing a large enough number of people (say, 3^^^3).
From a more metamathematical perspective, you can also question whether 3^^3 is a number at all. It’s perfectly straightforward to construct a perfectly consistent mathematics that rejects the axiom of infinity. Besides the philosophical justification for ultrafinitism (ie, infinite sets don’t really exist), these theories corresponds to various notions of bounded computation (such as logspace or polytime). This is a natural requirement, if we want to require moral judgements to be made quickly enough to be relevant to decision making—and that rules out seriously computing with numbers like 3^^^3.
I once read the following story about a Russian mathematician. I can’t find the source right now.
Cast: Russian mathematician RM, other guy OG
RM: “Truly large numbers don’t really exist in the same sense that small ones do.”
OG: “That’s ridiculous. Consider the powers of two. Does 2ˆ1 exist?”″
RM: “Yes.”
OG: “OK, does 2ˆ2 exist?”
RM: ”.Yes.”
OG: “So you’d agree that 2ˆ3 exists?”
RM: ”...Yes.”
OG: “How about 2ˆ4?”
RM: ”.......Yes.”
OG: “So this is silly. Where would you ever draw the boundary?”
RM: ”..............................................................................................................................................”