How does this work with Clippy (the only paperclipper in known existence) being tempted with 3^^^^3 paperclips?
Clippy has some sort of prior over the number of paperclips that could possibly exist. Let this number be P. Conditioned on each value of P, Clippy evaluates the utility of the offer and the probability that it comes true.
In particular, for P < 3^^^^3, the conditional probability that the offer of 3^^^^3 paperclips is legit is 0. If some large number of paperclips exists, e.g. P = 2*3^^^^3, the offer might actually be viable with non-negligible probability, while its utility would be given by 3^^^^3/P. Note that this is always at most 1.
However, unless Clippy lives in a very strange universe, it thinks that P >= 3^^^^3 is very unlikely. So the expected utility will be bounded by Pr[P >= 3^^^^3] and will end up being very small.
Clippy has some sort of prior over the number of paperclips that could possibly exist. Let this number be P. Conditioned on each value of P, Clippy evaluates the utility of the offer and the probability that it comes true.
In particular, for P < 3^^^^3, the conditional probability that the offer of 3^^^^3 paperclips is legit is 0. If some large number of paperclips exists, e.g. P = 2*3^^^^3, the offer might actually be viable with non-negligible probability, while its utility would be given by 3^^^^3/P. Note that this is always at most 1.
However, unless Clippy lives in a very strange universe, it thinks that P >= 3^^^^3 is very unlikely. So the expected utility will be bounded by Pr[P >= 3^^^^3] and will end up being very small.