For now, let me just reply to your incidental concluding point, because that’s brief.
I disagree that the red/green problem is unsolvable. I’d say the solution is that, with respect to the available information, both choices have equal (low) utility, so it’s simply a toss-up. A correct decision algorithm will just flip a coin or whatever.
Having done so, will a correct decision algorithm try to revise its choice in light of its (tentative) new knowledge of what its choice is? Only if it has nothing more productive to do with its remaining time.
Actually, one can do even better than that. As (I think), Eliezer implied, the key is Omega saying those words. (about the simulated you getting it wrong)
Did the simulated version receive that message too? (if yes, and if we assume Omega is always truthful, this implies an infinite recursion of simulations… let us not go invoking infinite nested computations willy-nilly.) If there was only a single layer of simulation, them Omega either gave that statement as input to it or did not. If yes, Omega is untruthful, which throws pretty much all of the standard reasoning about Omega out the window and we can simply take into account the possibility that Omega is blatantly lying.
If Omega is truthful, even to the simulations, then the simulation would not have received that prefix message. In which case you are in a different state than simulated you was. So all you have to do is make the decision opposite to what you would have done if you hadn’t heard that particular extra message. This may be guessed by simply one iteration of “I automatically want to guess color1… but wait, simulated me got it wrong, so I’ll guess color2 instead” since “actual” you has the knowledge that the previous version of you got it wrong.
If Omega lies to simulations and tells truth to “actuals” (and can somehow simulate without the simulation being conscious, so there’s no ambiguity about which you are, yet still be accurate… (am skeptical but confused on that point)), then we have an issue. But then it would require Omega to take a risk: if when telling the lie to the simulation, the simulation then gets it right, then what does Omega tell “actual” you?
(“actual” in quotes because I honestly don’t know whether or not one could be modeled with sufficient accuracy, however indirectly, without the model being conscious. I’m actually kind of skeptical of the prospect of a perfectly accurate model not being conscious, although a model that can determine some properties/approximations of the person without being conscious is probably possible)
TL;DR: even without access to coinflips beyond Omega’s predictive power, one might be able to do better in the red/green problem simply by noting that the nature of the additional information Omega provided you opens up the possibility that Omega’s simulation of you was a bit different than the actual situation you are in.
“Simulate telling the human that they got the answer wrong. If in this case they get the answer wrong, actually tell them that they get the answer wrong. Otherwise say nothing.”
This ought to make it relatively easy for Omega to truthfully put you in a “you’re screwed” situation a fair amount of the time. Albeit, if you know that this is Omega’s procedure, the rest of the time you should figure out what you would have done if Omega said “you’re wrong” and then do that.
This kind of thinking is, I think, outside the domain of current TDT, because it involves strategies that depend on actions you would have taken in counterfactual branches. I think it may even be outside the domain of current UDT for the same reason.
I don’t see why this is outside of UDT’s domain. It seems straightforward to model and solve the decision problem in UDT1. Here’s the world program:
def P(color):
outcome = "die"
if Omega_Predict(S, "you're wrong") == color:
if S("") == color:
outcome = "live"
else:
if S("you're wrong") == color:
outcome = "live"
Assuming a preference to maximize the occurrence of outcome=”live” averaged over P(“green”) and P(“red”), UDT1 would conclude that the optimal S returns a constant, either “green” or “red”, and do that.
BTW, do you find this “world program” style analysis useful? I don’t want to over-do them and get people annoyed. (I refrained from doing this for the problem described in Gary’s post, since it doesn’t mention UDT at all, and therefore I’m assuming you want to find a TDT-only solution.)
(I refrained from doing this for the problem described in Gary’s post, since it doesn’t mention UDT at all, and therefore I’m assuming you want to find a TDT-only solution.)
Yes, I was focusing on a specific difficulty in TDT, But I certainly have no objection to bringing UDT into the thread too. (I myself haven’t yet gotten around to giving UDT the attention I think it deserves.)
I was modeling what Eliezer wrote in the comment that I was responding to:
“Simulate telling the human that they got the answer wrong. If in this case they get the answer wrong, actually tell them that they get the answer wrong. Otherwise say nothing.”
BTW, if you add a tab in front of each line of your program listing, it will get formatted correctly.
Ah, I see. Then it seems that you are really solving the problem of minimizing the probability that Omega presents this problem in the first place.
What about the scenario, where Omega uses the strategy: Simulate telling the human that they got the answer wrong. Define the resulting answer as wrong, and the other as right.
This is what I modeled.
BTW, if you add a tab in front of each line of your program listing, it will get formatted correctly.
Thanks. Is there an easier way to get a tab into the comment input box than copy paste from an outside editor?
What about the scenario, where Omega uses the strategy: Simulate telling the human that they got the answer wrong. Define the resulting answer as wrong, and the other as right.
Is there an easier way to get a tab into the comment input box than copy paste from an outside editor?
Not that I’m aware of.
Are you guys talking about getting code to indent properly? You can do that by typing four spaces in front of each line. Each quadruple of spaces produces a further indentation.
Are you guys talking about getting code to indent properly? You can do that by typing four spaces in front of each line.
Spaces? Think of the wasted negentropy! I say we make tab the official Less Wrong indention symbol, and kick out anyone who disagrees. Who’s with me? :-)
Hm, I think the difference in our model programs indicates something that I don’t understand about UDT, like a wrong assumption that justified an optimization. But it seems they both produce the same result for P(S(“you’re wrong”)), which is outcome=”die” for all S.
Do you agree that this problem is, and should remain, unsolvable? (I understand “should remain unsolvable” to mean that any supposed solution must represent some sort of confusion about the problem.)
The input to P is supposed to contain the physical randomness in the problem, so P(S(“you’re wrong”)) doesn’t make sense to me. The idea is that both P(“green”) and P(“red”) get run, and we can think of them as different universes in a multiverse. Actually in this case I should have wrote “def P():” since there is no random correct color.
wrong assumption that justified an optimization
I’m not quite sure what you mean here, but in general I suggest just translating the decision problem directly into a world program without trying to optimize it.
Do you agree that this problem is, and should remain, unsolvable? (I understand “should remain unsolvable” to mean that any supposed solution must represent some sort of confusion about the problem.)
No, like I said, it seems pretty straightforward to solve in UDT. It’s just that even in the optimal solution you still die.
The input to P is supposed to contain the physical randomness in the problem, so P(S(“you’re wrong”)) doesn’t make sense to me. The idea is that both P(“green”) and P(“red”) get run, and we can think of them as different universes in a multiverse. Actually in this case I should have wrote “def P():” since there is no random correct color.
Ok, now I understood why you wrote your program the way you did.
It’s just that even in the optimal solution you still die.
By solve, I meant find a way to win. I think that after getting past different word use, we agree on the nature of the problem.
I’m not sure the algorithm you describe here is necessarily outside current TDT though. The counterfactual still corresponds to an actual thing Omega simulated. It’d be more like this: Omega did not add the “you are wrong” prefix. Therefore, conditioning on the idea that Omega always tries simulating with that prefix and only states the prefix if I (or whoever Omega is offering the challenge to) was wrong in that simulation, the simulation in question then did not produce the wrong answer.
Therefore a sufficient property for a good answer (one with higher expected utility) is that it should have the same output as that simulation. Therefore determine what that output was...
ie, TDT shouldn’t have much more problem (in principle) with that than with being told that it needs to guess the Nth digit of Pi. If possible, it would simply compute the Nth digit of Pi. In this case, it has to simply compute the outcome of a certain different algorithm which happens to be equivalent to its own decision algorithm when faced with a certain situation. I don’t THINK this would be inherently outside of current TDT as I understand it
I may be completely wrong on this, though, but that’s the way it seems to me.
As far as stuff like the problem in the OP, I suspect though that the Right Way for dealing with things analogous to counterfactual mugging (and extended to the problem in the OP) and such amounts to a very general precommitment… Or a retroactive precommitment.
My thinking here is rather fuzzy. I do suspect though that the Right Way probably looks something like the the TDT, in advance, doing a very general precommitment to be the sort of being that tends to have high expected utility when faced with counterfactual muggers and whatnot… (Or retroactively deciding to be the sort of being that effectively has the logical implication of being mathematically “precommited” to be such.)
By “unsolvable” I mean that you’re screwed over in final outcomes, not that TDT fails to have an output.
The interesting part of the problem is that, whatever you decide, you deduce facts about the background such that you know that what you are doing is the wrong thing. However, if you do anything differently, you would have to make a different deduction about the background facts, and again know that what you were doing was the wrong thing. Since we don’t believe that our decision is capable of affecting the background facts, the background facts ought to be a fixed constant, and we should be able to alter our decision without affecting the background facts… however, as soon as we do so, our inference about the unalterable background facts changes. It’s not 100% clear how to square this with TDT.
Actually, there is an optimal solution to this dilemma. Rather than use any internal process to decide, using a truly random process gives a 50% chance of survival. If you base your decision on a quantum randomness source, in principle no simulation can predict your choice (or rather, a complete simulation would correctly predict you fail in 50% of possible worlds).
Knowing how to use randomness against an intelligent adversary is important.
Gary postulated an infallible simulator, which presumably includes your entire initial state and all pseudorandom algorithms you might run. Known quantum randomness methods can only amplify existing entropy, not manufacture it ab initio. So you have no recourse to coinflips.
EDIT: Oops! pengvado is right. I was thinking of the case discussed here, where the random bits are provided by some quantum black box.
Quantum coinflips work even if Omega can predict them. It’s like a branch-both-ways instruction. Just measure some quantum variable, then measure a noncommuting variable, and voila, you’ve been split into two or more branches that observe different results and thus can perform different strategies. Omega’s perfect predictor tells it that you will do both strategies, each with half of your original measure. There is no arrangement of atoms (encoding the right answer) that Omega can choose in advance that would make both of you wrong.
If Omega wants to smack down the use of randomness, I can’t stop it. But there are a number of game theoretic situations where the optimal response is random play, and any decision theory that can’t respond correctly is broken.
A black box RNG is still useless despite being based on a quantum mechanism, or
That a quantum device will necessarily manufacture random bits.
Counterexamples to 2 are pretty straightforward (quantum computers), so I’m assuming you mean 1. I’m operating at the edge of my knowledge here (as my original mistake shows), but I think the entire point of Pironio et al’s paper was that you can verify random bits obtained from an adversary, subject to the conditions:
Bell inequality violations are observable (i.e., it’s a quantum generator).
The adversary can’t predict your measurement strategy.
By “unsolvable” I mean that you’re screwed over in final outcomes, not that TDT fails to have an output.
Oh ok. So it’s unsolvable in the same sense that “Choose red or green. Then I’ll shoot you.” is unsolvable. Sometimes choice really is futile. :) [EDIT: Oops, I probably misunderstood what you’re referring to by “screwed over”.]
The interesting part of the problem is that, whatever you decide, you deduce facts about the background such that you know that what you are doing is the wrong thing.
Yes, assuming that you’re the sort of algorithm that can (without inconsistency) know its own choice here before the choice is executed.
If you’re the sort of algorithm that may revise its intended action in response to the updated deduction, and if you have enough time left to perform the updated deduction, then the (previously) intended action may not be reliable evidence of what you will actually do, so it fails to provide sound reason for the update in the first place.
For now, let me just reply to your incidental concluding point, because that’s brief.
I disagree that the red/green problem is unsolvable. I’d say the solution is that, with respect to the available information, both choices have equal (low) utility, so it’s simply a toss-up. A correct decision algorithm will just flip a coin or whatever.
Having done so, will a correct decision algorithm try to revise its choice in light of its (tentative) new knowledge of what its choice is? Only if it has nothing more productive to do with its remaining time.
Actually, one can do even better than that. As (I think), Eliezer implied, the key is Omega saying those words. (about the simulated you getting it wrong)
Did the simulated version receive that message too? (if yes, and if we assume Omega is always truthful, this implies an infinite recursion of simulations… let us not go invoking infinite nested computations willy-nilly.) If there was only a single layer of simulation, them Omega either gave that statement as input to it or did not. If yes, Omega is untruthful, which throws pretty much all of the standard reasoning about Omega out the window and we can simply take into account the possibility that Omega is blatantly lying.
If Omega is truthful, even to the simulations, then the simulation would not have received that prefix message. In which case you are in a different state than simulated you was. So all you have to do is make the decision opposite to what you would have done if you hadn’t heard that particular extra message. This may be guessed by simply one iteration of “I automatically want to guess color1… but wait, simulated me got it wrong, so I’ll guess color2 instead” since “actual” you has the knowledge that the previous version of you got it wrong.
If Omega lies to simulations and tells truth to “actuals” (and can somehow simulate without the simulation being conscious, so there’s no ambiguity about which you are, yet still be accurate… (am skeptical but confused on that point)), then we have an issue. But then it would require Omega to take a risk: if when telling the lie to the simulation, the simulation then gets it right, then what does Omega tell “actual” you?
(“actual” in quotes because I honestly don’t know whether or not one could be modeled with sufficient accuracy, however indirectly, without the model being conscious. I’m actually kind of skeptical of the prospect of a perfectly accurate model not being conscious, although a model that can determine some properties/approximations of the person without being conscious is probably possible)
TL;DR: even without access to coinflips beyond Omega’s predictive power, one might be able to do better in the red/green problem simply by noting that the nature of the additional information Omega provided you opens up the possibility that Omega’s simulation of you was a bit different than the actual situation you are in.
Omega can use the following algorithm:
“Simulate telling the human that they got the answer wrong. If in this case they get the answer wrong, actually tell them that they get the answer wrong. Otherwise say nothing.”
This ought to make it relatively easy for Omega to truthfully put you in a “you’re screwed” situation a fair amount of the time. Albeit, if you know that this is Omega’s procedure, the rest of the time you should figure out what you would have done if Omega said “you’re wrong” and then do that.
This kind of thinking is, I think, outside the domain of current TDT, because it involves strategies that depend on actions you would have taken in counterfactual branches. I think it may even be outside the domain of current UDT for the same reason.
I don’t see why this is outside of UDT’s domain. It seems straightforward to model and solve the decision problem in UDT1. Here’s the world program:
Assuming a preference to maximize the occurrence of outcome=”live” averaged over P(“green”) and P(“red”), UDT1 would conclude that the optimal S returns a constant, either “green” or “red”, and do that.
BTW, do you find this “world program” style analysis useful? I don’t want to over-do them and get people annoyed. (I refrained from doing this for the problem described in Gary’s post, since it doesn’t mention UDT at all, and therefore I’m assuming you want to find a TDT-only solution.)
Yes, I was focusing on a specific difficulty in TDT, But I certainly have no objection to bringing UDT into the thread too. (I myself haven’t yet gotten around to giving UDT the attention I think it deserves.)
The world program I would use to model this scenario is:
The else branch seems unreachable, given color = S(“your’e wrong) and the usual assumptions about Omega.
I don’t understand what your nested if statements are modeling.
I was modeling what Eliezer wrote in the comment that I was responding to:
BTW, if you add a tab in front of each line of your program listing, it will get formatted correctly.
Ah, I see. Then it seems that you are really solving the problem of minimizing the probability that Omega presents this problem in the first place.
What about the scenario, where Omega uses the strategy: Simulate telling the human that they got the answer wrong. Define the resulting answer as wrong, and the other as right.
This is what I modeled.
Thanks. Is there an easier way to get a tab into the comment input box than copy paste from an outside editor?
In that case it should be modeled like this:
Not that I’m aware of.
Are you guys talking about getting code to indent properly? You can do that by typing four spaces in front of each line. Each quadruple of spaces produces a further indentation.
http://daringfireball.net/projects/markdown/syntax#precode
Spaces? Think of the wasted negentropy! I say we make tab the official Less Wrong indention symbol, and kick out anyone who disagrees. Who’s with me? :-)
Hm, I think the difference in our model programs indicates something that I don’t understand about UDT, like a wrong assumption that justified an optimization. But it seems they both produce the same result for P(S(“you’re wrong”)), which is outcome=”die” for all S.
Do you agree that this problem is, and should remain, unsolvable? (I understand “should remain unsolvable” to mean that any supposed solution must represent some sort of confusion about the problem.)
The input to P is supposed to contain the physical randomness in the problem, so P(S(“you’re wrong”)) doesn’t make sense to me. The idea is that both P(“green”) and P(“red”) get run, and we can think of them as different universes in a multiverse. Actually in this case I should have wrote “def P():” since there is no random correct color.
I’m not quite sure what you mean here, but in general I suggest just translating the decision problem directly into a world program without trying to optimize it.
No, like I said, it seems pretty straightforward to solve in UDT. It’s just that even in the optimal solution you still die.
Ok, now I understood why you wrote your program the way you did.
By solve, I meant find a way to win. I think that after getting past different word use, we agree on the nature of the problem.
Fair enough.
I’m not sure the algorithm you describe here is necessarily outside current TDT though. The counterfactual still corresponds to an actual thing Omega simulated. It’d be more like this: Omega did not add the “you are wrong” prefix. Therefore, conditioning on the idea that Omega always tries simulating with that prefix and only states the prefix if I (or whoever Omega is offering the challenge to) was wrong in that simulation, the simulation in question then did not produce the wrong answer.
Therefore a sufficient property for a good answer (one with higher expected utility) is that it should have the same output as that simulation. Therefore determine what that output was...
ie, TDT shouldn’t have much more problem (in principle) with that than with being told that it needs to guess the Nth digit of Pi. If possible, it would simply compute the Nth digit of Pi. In this case, it has to simply compute the outcome of a certain different algorithm which happens to be equivalent to its own decision algorithm when faced with a certain situation. I don’t THINK this would be inherently outside of current TDT as I understand it
I may be completely wrong on this, though, but that’s the way it seems to me.
As far as stuff like the problem in the OP, I suspect though that the Right Way for dealing with things analogous to counterfactual mugging (and extended to the problem in the OP) and such amounts to a very general precommitment… Or a retroactive precommitment.
My thinking here is rather fuzzy. I do suspect though that the Right Way probably looks something like the the TDT, in advance, doing a very general precommitment to be the sort of being that tends to have high expected utility when faced with counterfactual muggers and whatnot… (Or retroactively deciding to be the sort of being that effectively has the logical implication of being mathematically “precommited” to be such.)
By “unsolvable” I mean that you’re screwed over in final outcomes, not that TDT fails to have an output.
The interesting part of the problem is that, whatever you decide, you deduce facts about the background such that you know that what you are doing is the wrong thing. However, if you do anything differently, you would have to make a different deduction about the background facts, and again know that what you were doing was the wrong thing. Since we don’t believe that our decision is capable of affecting the background facts, the background facts ought to be a fixed constant, and we should be able to alter our decision without affecting the background facts… however, as soon as we do so, our inference about the unalterable background facts changes. It’s not 100% clear how to square this with TDT.
This is like trying to decide whether this statement is true:
“You will decide that this statement is false.”
There is nothing paradoxical about this statement. It is either true or false. The only problem is that you can’t get it right.
Actually, there is an optimal solution to this dilemma. Rather than use any internal process to decide, using a truly random process gives a 50% chance of survival. If you base your decision on a quantum randomness source, in principle no simulation can predict your choice (or rather, a complete simulation would correctly predict you fail in 50% of possible worlds).
Knowing how to use randomness against an intelligent adversary is important.
Gary postulated an infallible simulator, which presumably includes your entire initial state and all pseudorandom algorithms you might run. Known quantum randomness methods can only amplify existing entropy, not manufacture it ab initio. So you have no recourse to coinflips.
EDIT: Oops! pengvado is right. I was thinking of the case discussed here, where the random bits are provided by some quantum black box.
Quantum coinflips work even if Omega can predict them. It’s like a branch-both-ways instruction. Just measure some quantum variable, then measure a noncommuting variable, and voila, you’ve been split into two or more branches that observe different results and thus can perform different strategies. Omega’s perfect predictor tells it that you will do both strategies, each with half of your original measure. There is no arrangement of atoms (encoding the right answer) that Omega can choose in advance that would make both of you wrong.
I agree, and for this reason whenever I make descriptions I make Omega’s response to quantum smart-asses and other randomisers explicit and negative.
If Omega wants to smack down the use of randomness, I can’t stop it. But there are a number of game theoretic situations where the optimal response is random play, and any decision theory that can’t respond correctly is broken.
Does putting the ‘quantum’ in a black box change anything?
Not sure I know which question you’re asking:
A black box RNG is still useless despite being based on a quantum mechanism, or
That a quantum device will necessarily manufacture random bits.
Counterexamples to 2 are pretty straightforward (quantum computers), so I’m assuming you mean 1. I’m operating at the edge of my knowledge here (as my original mistake shows), but I think the entire point of Pironio et al’s paper was that you can verify random bits obtained from an adversary, subject to the conditions:
Bell inequality violations are observable (i.e., it’s a quantum generator).
The adversary can’t predict your measurement strategy.
Am I misunderstanding something?
Oh ok. So it’s unsolvable in the same sense that “Choose red or green. Then I’ll shoot you.” is unsolvable. Sometimes choice really is futile. :) [EDIT: Oops, I probably misunderstood what you’re referring to by “screwed over”.]
Yes, assuming that you’re the sort of algorithm that can (without inconsistency) know its own choice here before the choice is executed.
If you’re the sort of algorithm that may revise its intended action in response to the updated deduction, and if you have enough time left to perform the updated deduction, then the (previously) intended action may not be reliable evidence of what you will actually do, so it fails to provide sound reason for the update in the first place.