Okay, since I’m one year wiser now, here is a New and improved utility formalization
1) Torture human, and then wipe their memory of the event.
2) Pleasure human, and then wipe their memory of the event.
3) Torture human, and then do not wipe their memory of the event.
4) Pleasure human, and do not wipe their memory of the event.
Rank these in terms of preference from best to work. My rank is 4, 2, 1, 3. You must share my preference ranking for this to work.
You must also accept the following proposition: Death is roughly analogous to a memory wipe.
In January, I tried to escape anthropic panic by saying that “death” by trying to design a utility function which simply ignored possibilities that met certain criteria, while acknowledging that problems arise when you do this.
Today, I’ll say death / memory wipe reduce the extent to which the preceding actions matter because said actions no longer have long term repercussions.
So under Stuart_Armstrong’s hypothetical, we still continue behaving more or less as normal because if we must all die soon, our actions now matter a great deal less than if we do not die soon. So the sliver of chance in which we do not die must influence our actions a great deal more...an arbitrarily large number more, than the high chance that we do die.
Under this utility function, we can not completely freak out and stop all long term investments if we find out that we are under a false-vacuum which can collapse at any moment and that we’ve just been lucky so far.
Increasing near-term preference when faced with certain Doom is now fair game, taking into account that Doom decreases the weight of all preferences...so if there is any chance you aren’t Doomed, don’t throw away your resources on near-term stuff.
So...what happens to the cake now? If the cake doubles in tastiness each minute, but your expectation of being alive to eat it halves each minute, the expected value of the cake remains constant. However, if the pleasure of eating the cake+the memory of the cake lasts longer than an arbitrarily short epsilon time, then if you eat the cake sooner, you’ll expect feel the pleasure longer (As in, you are less likely to die before you even get a chance to finish the damn cake and feel satisfied about it)...so you aught to eat the cake immediately. If the rate of pleasure-doubling is lower than half, you don’t even need to resort to fancy arguments before you eat the cake.
However, you can still be persuaded to wait forever if the cake increases in tastiness above a certain rate, overpowering both the chance of death and any residual post-cake satisfactions.
Okay, since I’m one year wiser now, here is a New and improved utility formalization
1) Torture human, and then wipe their memory of the event.
2) Pleasure human, and then wipe their memory of the event.
3) Torture human, and then do not wipe their memory of the event.
4) Pleasure human, and do not wipe their memory of the event.
Rank these in terms of preference from best to work. My rank is 4, 2, 1, 3. You must share my preference ranking for this to work.
You must also accept the following proposition: Death is roughly analogous to a memory wipe.
In January, I tried to escape anthropic panic by saying that “death” by trying to design a utility function which simply ignored possibilities that met certain criteria, while acknowledging that problems arise when you do this.
Today, I’ll say death / memory wipe reduce the extent to which the preceding actions matter because said actions no longer have long term repercussions.
So under Stuart_Armstrong’s hypothetical, we still continue behaving more or less as normal because if we must all die soon, our actions now matter a great deal less than if we do not die soon. So the sliver of chance in which we do not die must influence our actions a great deal more...an arbitrarily large number more, than the high chance that we do die.
Under this utility function, we can not completely freak out and stop all long term investments if we find out that we are under a false-vacuum which can collapse at any moment and that we’ve just been lucky so far.
Increasing near-term preference when faced with certain Doom is now fair game, taking into account that Doom decreases the weight of all preferences...so if there is any chance you aren’t Doomed, don’t throw away your resources on near-term stuff.
So...what happens to the cake now? If the cake doubles in tastiness each minute, but your expectation of being alive to eat it halves each minute, the expected value of the cake remains constant. However, if the pleasure of eating the cake+the memory of the cake lasts longer than an arbitrarily short epsilon time, then if you eat the cake sooner, you’ll expect feel the pleasure longer (As in, you are less likely to die before you even get a chance to finish the damn cake and feel satisfied about it)...so you aught to eat the cake immediately. If the rate of pleasure-doubling is lower than half, you don’t even need to resort to fancy arguments before you eat the cake.
However, you can still be persuaded to wait forever if the cake increases in tastiness above a certain rate, overpowering both the chance of death and any residual post-cake satisfactions.