somewhat confident of Omega’s prediction
51% confidence would suffice.
Two-box expected value: 0.51 $1K + 0.49 $1.001M = $491000
One-box expected value: 0.51 $1M + 0.49 $0 = $510000
thrawnca
Karma: 8
At some point you’ll predictably approach death
I’m pretty sure that decision theories are not designed on that basis. We don’t want an AI to start making different decisions based on the probability of an upcoming decommission. We don’t want it to become nihilistic and stop making decisions because it predicted the heat death of the universe and decided that all paths have zero value. If death is actually tied to the decision in some way, then sure, take that into account, but otherwise, I don’t think a decision theory should have “death is inevitably coming for us all” as a factor.
How do you resolve that tension?
Well, as previously stated, my view is that the scenario as stated (single-shot with no precommitment) is not the most helpful hypothetical for designing a decision theory. An iterated version would actually be more relevant, since we want to design an AI that can make more than one decision. And in the iterated version, the tension is largely resolved, because there is a clear motivation to stick with the decision: we still hope for the next coin to come down heads.
what happens when the consequences grow large? Say 1 person to save 500, or 1 to save 3^^^^3?
If 3^^^^3 lives are at stake, and we assume that we are running on faulty or even hostile hardware, then it becomes all the more important not to rely on potentially-corrupted “seems like this will work”.
Well, humans can build calculators. That they can’t be the calculators that they create doesn’t demand an unusual explanation.
Yes, but don’t these articles emphasise how evolution doesn’t do miracles, doesn’t get everything right at once, and takes a very long time to do anything awesome? The fact that humans can do so much more than the normal evolutionary processes can marks us as a rather significant anomaly.
Your decision is a result of your decision theory
I get that that could work for a computer, because a computer can be bound by an overall decision theory without attempting to think about whether that decision theory still makes sense in the current situation.
I don’t mind predictors in eg Newcomb’s problem. Effectively, there is a backward causal arrow, because whatever you choose causes the predictor to have already acted differently. Unusual, but reasonable.
However, in this case, yes, your choice affects the predictor’s earlier decision—but since the coin never came down heads, who cares any more how the predictor would have acted? Why care about being the kind of person who will pay the counterfactual mugger, if there will never again be any opportunity for it to pay off?
Humans can do things that evolutions probably can’t do period over the expected lifetime of the universe.
This does beg the question, How, then, did an evolutionary process produce something so much more efficient than itself?
(And if we are products of evolutionary processes, then all our actions are basically facets of evolution, so isn’t that sentence self-contradictory?)
there is no distinction between making the decision ahead of time or not
Except that even if you make the decision, what would motivate you to stick to it once it can no longer pay up?
Your only motivation to pay is the hope of obtaining the $10000. If that hope does not exist, what reason would you have to abide by the decision that you make now?
I didn’t mean to suggest that the existence of suffering is evidence that there is a God. What I meant was, the known fact of “shared threat → people come together” makes the reality of suffering less powerful evidence against the existence of a God.
we want a rigorous, formal explanation of exactly how, when, and why you should or should not stick to your precommitment
Well, if we’re designing an AI now, then we have the capability to make a binding precommitment, simply by writing code. And we are still in a position where we can hope for the coin to come down heads. So yes, in that privileged position, we should bind the AI to pay up.
However, to the question as stated, “is the decision to give up $100 when you have no real benefit from it, only counterfactual benefit, an example of winning?” I would still answer, “No, you don’t achieve your goals/utility by paying up.” We’re specifically told that the coin has already been flipped. Losing $100 has negative utility, and positive utility isn’t on the table.
Alternatively, since it’s asking specifically about the decision, I would answer, If you haven’t made the decision until after the coin comes down tails, then paying is the wrong decision. Only if you’re deciding in advance (when you still hope for heads) can a decision to pay have the best expected value.
Even if deciding in advance, though, it’s still not a guaranteed win, but rather a gamble. So I don’t see any inconsistency in saying, on the one hand, “You should make a binding precommitment to pay”, and on the other hand, “If the coin has already come down tails without a precommitment, you shouldn’t pay.”
If there were a lottery where the expected value of a ticket was actually positive, and someone comes to you offering to sell you their ticket (at cost price), then it would make sense in advance to buy it, but if you didn’t, and then the winners were announced and that ticket didn’t win, then buying it no longer makes sense.
We are told no such thing. We are told it’s a fair coin and that can only mean that if you divide up worlds by their probability density, you win in half of them. This is defined.
No, take another look:
in the overwhelming measure of the MWI worlds it gives the same outcome. You don’t care about a fraction that sees a different result, in all reality the result is that Omega won’t even consider giving you $10000, it only asks for your $100.
I wouldn’t trust myself to accurately predict the odds of another repetition, so I don’t think it would unravel for me. But this comes back to my earlier point that you really need some external motivation, some precommitment, because “I want the 10K” loses its power as soon as the coin comes down tails.
if your decision theory pays up, then if he flips tails, you pay $100 for no possible benefit.
But in the single-shot scenario, after it comes down tails, what motivation does an ideal game theorist have to stick to the decision theory?
Like Parfit’s hitchhiker, although in advance you might agree that it’s a worthwhile deal, when it comes to the point of actually paying up, your motivation is gone, unless you have bound yourself in some other way.
It should also be possible to milk the scenario for publicity: “Our opponents sold out to the evil plutocrat and passed horrible legislation so he would bankroll them!”
I wish I were more confident that that strategy would actually work...
is the decision to give up $100 when you have no real benefit from it, only counterfactual benefit, an example of winning?
No, it’s a clear loss.
The only winning scenario is, “the coin comes down heads and you have an effective commitment to have paid if it came down tails.”
By making a binding precommitment, you effectively gamble that the coin will come down heads. If it comes down tails instead, clearly you have lost the gamble. Giving the $100 when you didn’t even make the precommitment would just be pointlessly giving away money.
The beggars-and-gods formulation is the same problem.
I don’t think so; I think the element of repetition substantially alters it—but in a good way, one that makes it more useful in designing a real-world agent. Because in reality, we want to design decision theories that will solve problems multiple times.
At the point of meeting a beggar, although my prospects of obtaining a gold coin this time around are gone, nonetheless my overall commitment is not meaningless. I can still think, “I want to be the kind of person who gives pennies to beggars, because overall I will come out ahead”, and this thought remains applicable. I know that I can average out my losses with greater wins, and so I still want to stick to the algorithm.
In the single-shot scenario, however, my commitment becomes worthless once the coin comes down tails. There will never be any more 10K; there is no motivation any more to give 100. Following my precommitment, unless it is externally enforced, no longer makes any sense.
So the scenarios are significantly different.
Sorry, but I’m not in the habit of taking one for the quantum superteam. And I don’t think that it really helps to solve the problem; it just means that you don’t necessarily care so much about winning any more. Not exactly the point.
Plus we are explicitly told that the coin is deterministic and comes down tails in the majority of worlds.
I think that what really does my head in about this problem is, although I may right now be motivated to make a commitment, because of the hope of winning the 10K, nonetheless my commitment cannot rely on that motivation, because when it comes to the crunch, that possibility has evaporated and the associated motivation is gone. I can only make an effective commitment if I have something more persistent—like the suggested $1000 contract with a third party. Without that, I cannot trust my future self to follow through, because the reasons that I would currently like it to follow through will no longer apply.
MBlume stated that if you want to be known as the sort of person who’ll do X given Y, then when Y turns up, you’d better do X. That’s a good principle—but it too can’t apply, unless at the point of being presented with the request for $100, you still care about being known as that sort of person—in other words, you expect a later repetition of the scenario in some form or another. This applies as well to Eliezer’s reasoning about how to design a self-modifying decision agent—which will have to make many future decisions of the same kind.
Just wanting the 10K isn’t enough to make an effective precommitment. You need some motivation that will persist in the face of no longer having the possibility of the 10K.
This is an attempt to examine the consequences of that.
Yes, but if the artificial scenario doesn’t reflect anything in the real world, then even if we get the right answer, therefore what? It’s like being vaccinated against a fictitious disease; even if you successfully develop the antibodies, what good do they do?
It seems to me that the “beggars and gods” variant mentioned earlier in the comments, where the opportunity repeats itself each day, is actually a more useful study. Sure, it’s much more intuitive; it doesn’t tie our brains up in knots, trying to work out a way to intend to do something at a point when all our motivation to do so has evaporated. But reality doesn’t have to be complicated. Sometimes you just have to learn to throw in the pebble.
If this was the only chance you ever get to determine your lifespan—then choose B.
In the real world, it would probably be a better idea to discard both options and use your natural lifespan to search for alternative paths to immortality.