A bigger problem is your ability to hand out arbitrarily large amounts of utility. Suppose the universe can be simulated by an N state Turing machine, this limits the number of possible states it can occupy to a finite (but probably very large) number. This in turn bounds the amount of utility you can offer me, since each state has finite utility and the maximum of a finite set of finite numbers is finite. (The reason why this doesn’t automatically imply a bounded utility function is that we are uncertain of N.)
As a result of this:
P(you can offer me k utility) > 0 for any fixed k
but
P(you can offer me x utility for any x) = 0
To be honest thought, I’m not really comfortable with this, and I think Solomonoff needs to be fixed (I don’t feel like I believe with certainty that the universe is computable). The real reason why you haven’t seen any of my money is that I think the maths is bullshit, as I have mentioned elsewhere.
Thinking about it more, this isn’t a serious problem for the dilemma. While P(you can offer me k utility) goes to zero as k goes to infinity but there’s no reason to suppose it goes faster then 1/n does.
This means you can still set a similar dilemma, with a probability of you being able to offer me 2^n utility eventually becoming greater than (1/2)^n for sufficiently large n, satisfying the conditions for a St Petersburg Lottery.
Thinking about it more, this isn’t a serious problem for the dilemma. While P(you can offer me k utility) goes to zero as k goes to infinity but there’s no reason to suppose it goes faster then 1/n does.
A bigger problem is your ability to hand out arbitrarily large amounts of utility. Suppose the universe can be simulated by an N state Turing machine, this limits the number of possible states it can occupy to a finite (but probably very large) number. This in turn bounds the amount of utility you can offer me, since each state has finite utility and the maximum of a finite set of finite numbers is finite. (The reason why this doesn’t automatically imply a bounded utility function is that we are uncertain of N.)
As a result of this:
P(you can offer me k utility) > 0 for any fixed k
but
P(you can offer me x utility for any x) = 0
To be honest thought, I’m not really comfortable with this, and I think Solomonoff needs to be fixed (I don’t feel like I believe with certainty that the universe is computable). The real reason why you haven’t seen any of my money is that I think the maths is bullshit, as I have mentioned elsewhere.
Thinking about it more, this isn’t a serious problem for the dilemma. While P(you can offer me k utility) goes to zero as k goes to infinity but there’s no reason to suppose it goes faster then 1/n does.
This means you can still set a similar dilemma, with a probability of you being able to offer me 2^n utility eventually becoming greater than (1/2)^n for sufficiently large n, satisfying the conditions for a St Petersburg Lottery.
That’s just Pascal’s mugging, though; the problem that “the utility of a Turing machine can grow much faster than its prior probability shrinks”.