Perhaps a little reframing is in order. Let’s go with shminux’s suggestion. You’re locked in a room with a cake and a vial of deadly gas. The vial is quantumly unstable, and has a 1⁄2 measure of breaking every minute. Do you eat the cake now, or later, if you know you’d enjoy it a little more if you ate it in 10 minutes?
“Now” corresponds to not thinking you have quantum immortality ”Later” corresponds to thinking you have quantum immortality
Thee reason I think a reframing like this is better is because it doesn’t by construction have a negligible probability if you choose #1 - humans are bad at hypotheticals, particularly when the hypothetical is nearly impossible.
I still pick “later”, despite not thinking I have quantum immortality.
As in, my behavior in this scenario won’t differ depending on whether my fate hangs on quantum vials or classical coin flips.
Of course, when I say “me” I mean the utility function describing myself that I outlined in the other comment in this thread… In real life I can’t really imagine eating cake at a moment like that. I’ll try to come up with a more realistic dilemma later to see if my gut instinct matches my utility function model....
I still pick “later”, despite not thinking I have quantum immortality
I dunno, your choice is walking in a duck-like manner. Of course, intuition pumps are tricky, and we’re usually not consistent with respect to them. For example, if I asked you to lock yourself in that room for ten dollars per minute, you’d refuse. Perhaps some conceptual weirdness stems from the fact that in the thought experiment, there’s no point at which you’re “let out to find out if you’re dead or not.”
There may well be an inconsistency, but that particular example doesn’t seem to exploit it yet...
U=aPresent+bFutureExpectations
agree, die
U1=aPresent+b*[Expected Future with me Dead]
agree, live
U2=present+b*[alive and $10 richer]
refuse
U3=present+b*[Expected Future with me Alive]
U3 > [.5U2 + .5U1] do not take the deal,
With the cake, however, ExpectedValueFromFutureCake = Null in the case that I am dead, which renders the entire utility function irrelevant. (within the system of the other comment)
Eat cake now (dead or alive)
Ux = CakeRightNow + 0
Eat cake later
Uy = 0 + ExpectedValueFromFutureCake
Die without eating cake
Uz = 0 + null, therefore irrelevant
Ux < Uy so do not eat the cake
What I didn’t mention before—as I’ve outlined it, this utility function won’t ever get to eat the cake, since the expected future value is always greater. So there’s that flaw. I’m not sure whether this signals that the utility function is silly, or that the cake is silly...maybe both.
However, my utility function is only silly in that you can’t even eat the cake before near certain death—I’m guessing your model would have you eat the cake as soon as the probability of your death crossed a certain threshold. But if you were immortal and the cake was just sitting in front of you in all its ever-increasing utility, when would you eat it? The cake will generate a paradox—you always expect more from the cake in the future, yet you will never eat the cake (and once you realize this, your expectations from the cake should drop down to zero—which means you might as well eat it now, but if you wait just a bit longer...)
I think the cake breaks everything and we aught to not use it.
Dying without eating cake surely has a utility. I mean, suppose I know I’m going to die tomorrow. I still assign different utilities to different ways I could spend today, I don’t say the utility of today is null in all cases.
Or are you saying that it’s possible to have a silly utility function that doesn’t assign any value to eating the cake before dying compared to not eating the cake and then dying at the same time? Sure, but that utility function is silly.
Okay, since I’m one year wiser now, here is a New and improved utility formalization
1) Torture human, and then wipe their memory of the event.
2) Pleasure human, and then wipe their memory of the event.
3) Torture human, and then do not wipe their memory of the event.
4) Pleasure human, and do not wipe their memory of the event.
Rank these in terms of preference from best to work. My rank is 4, 2, 1, 3. You must share my preference ranking for this to work.
You must also accept the following proposition: Death is roughly analogous to a memory wipe.
In January, I tried to escape anthropic panic by saying that “death” by trying to design a utility function which simply ignored possibilities that met certain criteria, while acknowledging that problems arise when you do this.
Today, I’ll say death / memory wipe reduce the extent to which the preceding actions matter because said actions no longer have long term repercussions.
So under Stuart_Armstrong’s hypothetical, we still continue behaving more or less as normal because if we must all die soon, our actions now matter a great deal less than if we do not die soon. So the sliver of chance in which we do not die must influence our actions a great deal more...an arbitrarily large number more, than the high chance that we do die.
Under this utility function, we can not completely freak out and stop all long term investments if we find out that we are under a false-vacuum which can collapse at any moment and that we’ve just been lucky so far.
Increasing near-term preference when faced with certain Doom is now fair game, taking into account that Doom decreases the weight of all preferences...so if there is any chance you aren’t Doomed, don’t throw away your resources on near-term stuff.
So...what happens to the cake now? If the cake doubles in tastiness each minute, but your expectation of being alive to eat it halves each minute, the expected value of the cake remains constant. However, if the pleasure of eating the cake+the memory of the cake lasts longer than an arbitrarily short epsilon time, then if you eat the cake sooner, you’ll expect feel the pleasure longer (As in, you are less likely to die before you even get a chance to finish the damn cake and feel satisfied about it)...so you aught to eat the cake immediately. If the rate of pleasure-doubling is lower than half, you don’t even need to resort to fancy arguments before you eat the cake.
However, you can still be persuaded to wait forever if the cake increases in tastiness above a certain rate, overpowering both the chance of death and any residual post-cake satisfactions.
TL:DR: You’re probably right.. I was attempting to create a utility function that wouldn’t freak out if the evidence said that we were doomed and anthropics rendered the “we haven’t died yet so we’re probably safe” argument out the window, without resorting to anthropic immortality, while also behaving more or less normally in most scenarios. I doubt I succeeded at this. I’ve got a long winded explanation below, but feel free not to read it.
It’s been a year since I made this comment, and there’s a lot of context to catch up on. Reading this over, here is what I think this is going on in this conversation.
In the other comment I tried to make the case that our behavior would not change even if we constantly had a 99% chance of total universe annihilation every day, regardless of whether it was quantum or classical. As in, we shouldn’t all switch to near-term thinking just because physicists tell us that we face a high chance of total annihilation every day and there is nothing we can do about it.
Why? Because in the universe where everything is annihilated, it doesn’t matter what we do now. What we do now only matters in the case that we survive. Thus, even if there is only a small chance of survival, we should behave under the assumption that we survive.
Now, the above isn’t necessarily true for all utility functions, or even true for you. It’s just how I would wish behave if I was told that false vacuum has a 99% chance of occurring every day, and the only reason we’re still here is classical luck / quantum immortality. I wouldn’t want to switch to near-mode thinking and rush around fulfilling short term goals just because I heard that news. My intuition says that, when faced with the probability of unavoidable doom, it should not alter my behavior no matter how large that probability, particularly in the case where the Doom is a chronic, continuing phenomenon. If you knew you had a high chance of dying, you’d tell your loved ones that you loved them, because they’d live on and those words would have an effect. But if everyone, including those loved ones, is probably doomed all the time...well, it seems best to just continuing on as usual.
So now, I have to attempt to formalize my preferences in terms of a utility function. The essence that must be represented is that in the event that certain scenarios occur, certain decisions become irrelevant, and so for the purpose of deciding between certain choices you should just pretend those scenarios can’t happen.
A real world example where there is a chance of something that can render a choice irrelevant: You need to decide whether to bet or fold your hand in poker game. You do not need to consider the probability that the fire alarm will ring and disrupt the game, because in the event that this happens your choice is irrelevant anyway. (This does not identical to the cake/false vacuum scenario, because both folding and betting are future orientated actions. It is only an example of where a choice becomes irrelevant)
I attempted to do this by assigning “null” value to those events. By doing this, my intuition was aligned with my formalized utility function in the case of Stuart Armstrong’s. Then Manfred comes along and creates a new scenario. Like the previous scenario, it examines how the likelihood of one’s death effects the choice between a short term preference and a long term preference. It is, in theory, identical to the first scenario. Therefore, the utility function formalized in the other comment should in theory behave the same way, right? That utility function would choose to eat the cake later.
I pointed this out to Manfred. He then claimed:
I dunno, your choice is walking in a duck-like manner. Of course, intuition pumps are tricky, and we’re usually not consistent with respect to them. For example, if I asked you to lock yourself in that room for ten dollars per minute, you’d refuse. Perhaps some conceptual weirdness stems from the fact that in the thought experiment, there’s no point at which you’re “let out to find out if you’re dead or not.”
What Manfred means is that, if I always choose my actions under the assumption that I survive, then I must be willing to put myself in danger (since I attempted to consider the situations where I die / humanity ends as irrelevant to my decision making).
My comment you replied to points out that Manfred is wrong about this because the utility function as formalized still prefers a future where I do not end / humanity does not end to the alternative.
Your criticism
I still assign different utilities to different ways I could spend today
is completely valid. actually I acknowledged this in the comment you replied to:
my utility function is only silly in that you can’t even eat the cake before near certain death
Meaning, yes, there are some major problems with this (which you pointed out) but Manfred’s criticism that this utility function puts itself in dangerous situations is not one of them. Also, it’s worth noting that in the absence of certain death, no one can ever eat the cake...so I’m not sure if the weirdness is due to my formalization.. or if it’s just inheriting weird behavior from the pathological properties of the cake.
So now you understand the problem: I was trying to create a utility function which would cause someone to not significantly change their behavior in Stuart_Armstrong’s scenario without making appeals to spooky anthropic immortality.
But as you pointed out, it’s flawed because real humans do increase weight on short term hedons when they think they are about to die. (Although you might argue this preference is based off the false belief that death = sensory deprivation)
So, just to clarify, here is the “normal” utility function which Manfred implicitly assumes
Eat cake now: U(now) = CurrentCakeTastiness
Postpone eating cake for one minute: U(postpone) = .5futureCakeTastiness + .5(zero, because you are dead and can’t eat it)
This utility function has a tendency to eat the cake right away—though if the difference between the future cake and current cake is drastic enough, it could be persuaded to wait forever.
This utility function also rushes around and fulfills short term goals if physicists tell us that our universe has a 99% chance of ending every day. This behavior is NOT my preferences.
Here is the utility function which I postulated
Eat cake now: U(now) = CurrentCakeTastiness
Postpone eating cake for one minute: U(postpone) = futureCakeTastiness + (null, because you’ll be dead and it won’t matter anyway. The probability of this option is discounted.)
This utility function has a tendency to not eat the cake right away.I do not know if these actions mirror my preferences—the situation is too alien for my intuition.. However, this utility function also has the property of behaving normally if physicists tell us that our universe has a 99% chance of ending every day, and I consider this to mirror my preferences**.
Let’s make the cake less alien by replacing the cake with something I’d actually care about in this situation.
Eat Cake = save C people from torture. There are a finite number of humans. If you don’t save them, they get tortured until they die.
Every minute of not eating cake, the number of people you can save from torture increases (linearly? Exponentially? Asymptotically approaching a number? Depending on the function you might be able to save everyone from torture.)
X% chance of everyone involved dying every minute. When humans die via some other cause, a new human is born to replace them (and be tortured, or not, depending on your choices.) The population remains constant.
Now, the cake example perfectly mirrors Stuart Armstrong’s example, without forcing you to accept a cake which you can never eat into your utility function.
If X=0%, I think you’d want to wait until you could save everyone if it was possible to do so. Failing that (say, C never exceeds 1⁄2 the human population) you’d want to at least wait until C was pretty damn close to the maximum.
If X = 100%, my formalized utility function says you’d eat the cake right away. That seems intuitive enough.
If X was between 0% and 100%, how would you behave?
My formalized utility function says that you would behave identically to if X was 0%. Is this silly and inhuman? Maybe… probably. I’m not certain that it’s silly because I haven’t come up with an example where it is obviously silly, but this is probably due to lack of thinking about it sufficiently. (Related question: Is the human increase in short-term preference under high chance of death a real preference or just an artifact of thinking that death in analogy to “going away”?)
Manfred’s implicit utility function says your behavior would take some intermediate form, unless you believed in Quantum Immortality and thought X% was decided by a quantum dice, in which case you would behave identically to if X was 0%. I think the quantum portion of this is silly—even under Many Worlds, current-you aught to multiply your enjoyment by the number of future-you’s that are there to enjoy it. Is it still silly in the classical scenario, where you start shifting to short-term for all preferences which become unfulfilled after death? I don’t know, but it leads to some conclusions I don’t like.
It is, of course, possible that I’m actually engaging in biased thinking here—as in, the reason I think that I prefer to ignore the possibility that we live in a universe where there is a false vacuum that might collapse at any moment because behaving as if this is true is stressful.
Perhaps a little reframing is in order. Let’s go with shminux’s suggestion. You’re locked in a room with a cake and a vial of deadly gas. The vial is quantumly unstable, and has a 1⁄2 measure of breaking every minute. Do you eat the cake now, or later, if you know you’d enjoy it a little more if you ate it in 10 minutes?
“Now” corresponds to not thinking you have quantum immortality
”Later” corresponds to thinking you have quantum immortality
Thee reason I think a reframing like this is better is because it doesn’t by construction have a negligible probability if you choose #1 - humans are bad at hypotheticals, particularly when the hypothetical is nearly impossible.
I still pick “later”, despite not thinking I have quantum immortality.
As in, my behavior in this scenario won’t differ depending on whether my fate hangs on quantum vials or classical coin flips.
Of course, when I say “me” I mean the utility function describing myself that I outlined in the other comment in this thread… In real life I can’t really imagine eating cake at a moment like that. I’ll try to come up with a more realistic dilemma later to see if my gut instinct matches my utility function model....
I dunno, your choice is walking in a duck-like manner. Of course, intuition pumps are tricky, and we’re usually not consistent with respect to them. For example, if I asked you to lock yourself in that room for ten dollars per minute, you’d refuse. Perhaps some conceptual weirdness stems from the fact that in the thought experiment, there’s no point at which you’re “let out to find out if you’re dead or not.”
There may well be an inconsistency, but that particular example doesn’t seem to exploit it yet...
U=aPresent+bFutureExpectations
agree, die
U1=aPresent+b*[Expected Future with me Dead]
agree, live
U2=present+b*[alive and $10 richer]
refuse
U3=present+b*[Expected Future with me Alive]
U3 > [.5U2 + .5U1] do not take the deal,
With the cake, however, ExpectedValueFromFutureCake = Null in the case that I am dead, which renders the entire utility function irrelevant. (within the system of the other comment)
Eat cake now (dead or alive) Ux = CakeRightNow + 0
Eat cake later Uy = 0 + ExpectedValueFromFutureCake
Die without eating cake Uz = 0 + null, therefore irrelevant
Ux < Uy so do not eat the cake
What I didn’t mention before—as I’ve outlined it, this utility function won’t ever get to eat the cake, since the expected future value is always greater. So there’s that flaw. I’m not sure whether this signals that the utility function is silly, or that the cake is silly...maybe both.
However, my utility function is only silly in that you can’t even eat the cake before near certain death—I’m guessing your model would have you eat the cake as soon as the probability of your death crossed a certain threshold. But if you were immortal and the cake was just sitting in front of you in all its ever-increasing utility, when would you eat it? The cake will generate a paradox—you always expect more from the cake in the future, yet you will never eat the cake (and once you realize this, your expectations from the cake should drop down to zero—which means you might as well eat it now, but if you wait just a bit longer...)
I think the cake breaks everything and we aught to not use it.
Dying without eating cake surely has a utility. I mean, suppose I know I’m going to die tomorrow. I still assign different utilities to different ways I could spend today, I don’t say the utility of today is null in all cases.
Or are you saying that it’s possible to have a silly utility function that doesn’t assign any value to eating the cake before dying compared to not eating the cake and then dying at the same time? Sure, but that utility function is silly.
Okay, since I’m one year wiser now, here is a New and improved utility formalization
1) Torture human, and then wipe their memory of the event.
2) Pleasure human, and then wipe their memory of the event.
3) Torture human, and then do not wipe their memory of the event.
4) Pleasure human, and do not wipe their memory of the event.
Rank these in terms of preference from best to work. My rank is 4, 2, 1, 3. You must share my preference ranking for this to work.
You must also accept the following proposition: Death is roughly analogous to a memory wipe.
In January, I tried to escape anthropic panic by saying that “death” by trying to design a utility function which simply ignored possibilities that met certain criteria, while acknowledging that problems arise when you do this.
Today, I’ll say death / memory wipe reduce the extent to which the preceding actions matter because said actions no longer have long term repercussions.
So under Stuart_Armstrong’s hypothetical, we still continue behaving more or less as normal because if we must all die soon, our actions now matter a great deal less than if we do not die soon. So the sliver of chance in which we do not die must influence our actions a great deal more...an arbitrarily large number more, than the high chance that we do die.
Under this utility function, we can not completely freak out and stop all long term investments if we find out that we are under a false-vacuum which can collapse at any moment and that we’ve just been lucky so far.
Increasing near-term preference when faced with certain Doom is now fair game, taking into account that Doom decreases the weight of all preferences...so if there is any chance you aren’t Doomed, don’t throw away your resources on near-term stuff.
So...what happens to the cake now? If the cake doubles in tastiness each minute, but your expectation of being alive to eat it halves each minute, the expected value of the cake remains constant. However, if the pleasure of eating the cake+the memory of the cake lasts longer than an arbitrarily short epsilon time, then if you eat the cake sooner, you’ll expect feel the pleasure longer (As in, you are less likely to die before you even get a chance to finish the damn cake and feel satisfied about it)...so you aught to eat the cake immediately. If the rate of pleasure-doubling is lower than half, you don’t even need to resort to fancy arguments before you eat the cake.
However, you can still be persuaded to wait forever if the cake increases in tastiness above a certain rate, overpowering both the chance of death and any residual post-cake satisfactions.
TL:DR: You’re probably right.. I was attempting to create a utility function that wouldn’t freak out if the evidence said that we were doomed and anthropics rendered the “we haven’t died yet so we’re probably safe” argument out the window, without resorting to anthropic immortality, while also behaving more or less normally in most scenarios. I doubt I succeeded at this. I’ve got a long winded explanation below, but feel free not to read it.
It’s been a year since I made this comment, and there’s a lot of context to catch up on. Reading this over, here is what I think this is going on in this conversation.
In the other comment I tried to make the case that our behavior would not change even if we constantly had a 99% chance of total universe annihilation every day, regardless of whether it was quantum or classical. As in, we shouldn’t all switch to near-term thinking just because physicists tell us that we face a high chance of total annihilation every day and there is nothing we can do about it.
Why? Because in the universe where everything is annihilated, it doesn’t matter what we do now. What we do now only matters in the case that we survive. Thus, even if there is only a small chance of survival, we should behave under the assumption that we survive.
Now, the above isn’t necessarily true for all utility functions, or even true for you. It’s just how I would wish behave if I was told that false vacuum has a 99% chance of occurring every day, and the only reason we’re still here is classical luck / quantum immortality. I wouldn’t want to switch to near-mode thinking and rush around fulfilling short term goals just because I heard that news. My intuition says that, when faced with the probability of unavoidable doom, it should not alter my behavior no matter how large that probability, particularly in the case where the Doom is a chronic, continuing phenomenon. If you knew you had a high chance of dying, you’d tell your loved ones that you loved them, because they’d live on and those words would have an effect. But if everyone, including those loved ones, is probably doomed all the time...well, it seems best to just continuing on as usual.
So now, I have to attempt to formalize my preferences in terms of a utility function. The essence that must be represented is that in the event that certain scenarios occur, certain decisions become irrelevant, and so for the purpose of deciding between certain choices you should just pretend those scenarios can’t happen.
A real world example where there is a chance of something that can render a choice irrelevant: You need to decide whether to bet or fold your hand in poker game. You do not need to consider the probability that the fire alarm will ring and disrupt the game, because in the event that this happens your choice is irrelevant anyway. (This does not identical to the cake/false vacuum scenario, because both folding and betting are future orientated actions. It is only an example of where a choice becomes irrelevant)
I attempted to do this by assigning “null” value to those events. By doing this, my intuition was aligned with my formalized utility function in the case of Stuart Armstrong’s. Then Manfred comes along and creates a new scenario. Like the previous scenario, it examines how the likelihood of one’s death effects the choice between a short term preference and a long term preference. It is, in theory, identical to the first scenario. Therefore, the utility function formalized in the other comment should in theory behave the same way, right? That utility function would choose to eat the cake later.
I pointed this out to Manfred. He then claimed:
What Manfred means is that, if I always choose my actions under the assumption that I survive, then I must be willing to put myself in danger (since I attempted to consider the situations where I die / humanity ends as irrelevant to my decision making).
My comment you replied to points out that Manfred is wrong about this because the utility function as formalized still prefers a future where I do not end / humanity does not end to the alternative.
Your criticism
is completely valid. actually I acknowledged this in the comment you replied to:
Meaning, yes, there are some major problems with this (which you pointed out) but Manfred’s criticism that this utility function puts itself in dangerous situations is not one of them. Also, it’s worth noting that in the absence of certain death, no one can ever eat the cake...so I’m not sure if the weirdness is due to my formalization.. or if it’s just inheriting weird behavior from the pathological properties of the cake.
So now you understand the problem: I was trying to create a utility function which would cause someone to not significantly change their behavior in Stuart_Armstrong’s scenario without making appeals to spooky anthropic immortality.
But as you pointed out, it’s flawed because real humans do increase weight on short term hedons when they think they are about to die. (Although you might argue this preference is based off the false belief that death = sensory deprivation)
So, just to clarify, here is the “normal” utility function which Manfred implicitly assumes
Eat cake now: U(now) = CurrentCakeTastiness
Postpone eating cake for one minute: U(postpone) = .5futureCakeTastiness + .5(zero, because you are dead and can’t eat it)
This utility function has a tendency to eat the cake right away—though if the difference between the future cake and current cake is drastic enough, it could be persuaded to wait forever.
This utility function also rushes around and fulfills short term goals if physicists tell us that our universe has a 99% chance of ending every day. This behavior is NOT my preferences.
Here is the utility function which I postulated
Eat cake now: U(now) = CurrentCakeTastiness
Postpone eating cake for one minute: U(postpone) = futureCakeTastiness + (null, because you’ll be dead and it won’t matter anyway. The probability of this option is discounted.)
This utility function has a tendency to not eat the cake right away. I do not know if these actions mirror my preferences—the situation is too alien for my intuition.. However, this utility function also has the property of behaving normally if physicists tell us that our universe has a 99% chance of ending every day, and I consider this to mirror my preferences**.
Let’s make the cake less alien by replacing the cake with something I’d actually care about in this situation.
Eat Cake = save C people from torture. There are a finite number of humans. If you don’t save them, they get tortured until they die.
Every minute of not eating cake, the number of people you can save from torture increases (linearly? Exponentially? Asymptotically approaching a number? Depending on the function you might be able to save everyone from torture.)
X% chance of everyone involved dying every minute. When humans die via some other cause, a new human is born to replace them (and be tortured, or not, depending on your choices.) The population remains constant.
Now, the cake example perfectly mirrors Stuart Armstrong’s example, without forcing you to accept a cake which you can never eat into your utility function.
If X=0%, I think you’d want to wait until you could save everyone if it was possible to do so. Failing that (say, C never exceeds 1⁄2 the human population) you’d want to at least wait until C was pretty damn close to the maximum.
If X = 100%, my formalized utility function says you’d eat the cake right away. That seems intuitive enough.
If X was between 0% and 100%, how would you behave?
My formalized utility function says that you would behave identically to if X was 0%. Is this silly and inhuman? Maybe… probably. I’m not certain that it’s silly because I haven’t come up with an example where it is obviously silly, but this is probably due to lack of thinking about it sufficiently. (Related question: Is the human increase in short-term preference under high chance of death a real preference or just an artifact of thinking that death in analogy to “going away”?)
Manfred’s implicit utility function says your behavior would take some intermediate form, unless you believed in Quantum Immortality and thought X% was decided by a quantum dice, in which case you would behave identically to if X was 0%. I think the quantum portion of this is silly—even under Many Worlds, current-you aught to multiply your enjoyment by the number of future-you’s that are there to enjoy it. Is it still silly in the classical scenario, where you start shifting to short-term for all preferences which become unfulfilled after death? I don’t know, but it leads to some conclusions I don’t like.
It is, of course, possible that I’m actually engaging in biased thinking here—as in, the reason I think that I prefer to ignore the possibility that we live in a universe where there is a false vacuum that might collapse at any moment because behaving as if this is true is stressful.