There’s no need to break causality. You are a being implemented in chaotic wetware. However, there’s no reason to think we couldn’t have rational agents implemented in much more predictable form, as python routines for example, so that any being with superior computation power could simply inspect the source and determine what the output would be.
In such a case, Newcomb-like problems would arise, perfectly lawfully, under normal physics.
In fact, Newcomb-like problems fall naturally out of any ability to simulate and predict the actions of other agents. Omega as described is essentially the limit as predictive power goes to infinity.
This gives me the intuition that trying to decide whether to one-box or two box on newcomb is like trying to decide what 0^0 is; you get your intuition by following a limit process, but that limit process produces different results depending on the path you take.
It would be interesting to look at finitely good predictors. Perhaps we can find something analogous to the result that lim_(x, y -->0) (x^y) is path dependent.
If we define an imperfect predictor as a perfect predictor plus noise, i.e. produces the correct prediction with probability p regardless of the cognition algorithm it’s trying to predict, then Newcomb-like problems are very robust to imperfect prediction: for any p > .5 there is some payoff ratio great enough to preserve the paradox, and the required ratio goes down as the prediction improves. e.g. if 1-boxing gets 100 utilons and 2-boxing gets 1 utilon, then the predictor only needs to be more than 50.5% accurate. So the limit in that direction favors 1-boxing.
What other direction could there be? If the prediction accuracy depends on the algorithm-to-be-predicted (as it would in the real world), then you could try to be an algorithm that is mispredicted in your favor… but a misprediction in your favor can only occur if you actually 2-box, so it only takes a modicum of accuracy before a 1-boxer who tries to be predictable is better off than a 2-boxer who tries to be unpredictable.
I can’t see any other way for the limit to turn out.
If you have two agents trying to precommit not to be blackmailed by each other / precommit not to pay attention to the others precommitment, then any attempt to take a limit of this Newcomblike problem does depend on how you approach the limit. (I don’t know how to solve this problem.)
The value(s) for which the limit is being taken here is unidirectional predictive power, which is loosely a function of the difference in intelligence between the two agents; intuitively, I think a case could be made that (assuming ideal rationality) the total accuracy of mutual behavior prediction between two agents is conserved in some fashion, that doubling the predictive power of one unavoidably would roughly halve the predictive power of the other. Omega represents an entity with a delta-g so large vs. us that predictive power is essentially completely one-sided.
From that basis, allowing the unidirectional predictive power of both agents to go to infinity is probably inherently ill-defined and there’s no reason to expect the problem to have a solution.
there’s no reason to think we couldn’t have rational agents implemented in much more predictable form, as python routines for example, so that any being with superior computation power could simply inspect the source and determine what the output would be.
Such a being would be different from a human in fundamental ways. Imagine knowing with certainty that your actions can be predicted perfectly by the guy next door, even taking into account that you are trying to be hard to predict?
A (quasi)rational agent with access to genuine randomness (such as a human) is a different matter. A superintelligence could almost perfectly predict the probability distribution over my actions, but by quantum entanglement it would not be able to predict my actual actions.
A (quasi)rational agent with access to genuine randomness (such as a human)
Whaddaya mean humans are rational agents with access to genuine randomness? That’s what we’re arguing about in the first place!
A superintelligence could almost perfectly predict the probability distribution over my actions, but by quantum entanglement it would not be able to predict my actual actions.
Perhaps Omega is entangled with your brain such that in all the worlds in which you would choose to one-box, he would predict that you one-box, and all the worlds in which you would choose to two-box, he would predict that you two-box?
Imagine knowing with certainty that your actions can be predicted perfectly by the guy next door, even taking into account that you are trying to be hard to predict?
You wouldn’t know this with certainty* because it wouldn’t be true.
(*unless you were delusional)
The guy next door is on roughly your mental level. Thus, the guy next door can’t predict your actions perfectly, because he can’t run a perfect simulation of your mind that’s faster than you. He doesn’t have the capacity.
And he certainly doesn’t have the capacity to simulate the environment, including other people, while doing so.
A (quasi)rational agent with access to genuine randomness (such as a human) is a different matter.
Humans may or may not generally have access to genuine randomness.
It’s as yet unknown whether we even have run on quantum randomness; and its also unprovable that quantum randomness is actually genuine randomness, and not just based on effects we don’t yet understand, as so many other types of randomness have been.
You wouldn’t know this with certainty* because it wouldn’t be true.
You’re not taking this in the least convenient possible world. Surely it’s not impossible in principle that your neighbor can simulate you and your environment. Perhaps your neighbor is superintelligent?
It’s ALSO not impossible in principle in the real world. A superintelligent entity could, in principle, perfectly predict my actions.
Remember, in the Least Convenient Possible World quantum “randomness” isn’t random.
As such, this ISN’T a fundamental difference between humans and “such beings”.
Which was all I set out to demonstrate.
I was using the “most plausible world” on the basis that it seemed pretty clear that that was the one Roko intended. (Where your neighbour isn’t in fact Yahweh in disguise).
EDIT: Probably should specify worlds for things in this kind of environment. Thanks, the critical environment here is helping me think about how I think/argue.
It’s as yet unknown whether we even have run on quantum randomness; and its also unprovable that quantum randomness is actually genuine randomness, and not just based on effects we don’t yet understand, as so many other types of randomness have been.
If you believe the Many Worlds Interpretation, then quantum randomness just creates copies in a deterministic way.
You cannot do that without breaking Rice’s theorem. If you assume you can find out the answer from someone else’s source code → instant contradiction.
You cannot work around Rice’s theorem or around causality by specifying 50.5% accuracy independently of modeled system, any accuracy higher than 50%+epsilon is equivalent to indefinitely good accuracy by repeatedly predicting (standard cryptographic result), and 50%+epsilon doesn’t cause the paradox.
Give me one serious math model of Newcomb-like problems where the paradox emerges while preserving causality. Here are some examples. Then you model it, you either get trivial solution to one-box, or causality break, or omega loses.
You decide first what you would do in every situation, omega decides second, and now you only implement your initial decision table and are not allowed to switch. Game theory says you should implement one-boxing.
You decide first what you would do in every situation, omega decides second, and now you are allowed to switch. Game theory says you should precommit to one-box, then implement two-boxing, omega loses.
You decide first what you would do in every situation, omega decides second, and now you are allowed to switch. If omega always decides correctly, then he bases his decision on your switch, which either turns it into model #1 (you cannot really switch, precommitment is binding), or breaks causality.
Rice’s theorem says you can’t predict every possible algorithm in general. Plenty of particular algorithms can be predictable. If you’re running on a classical computer and Omega has a copy of you, you are perfectly predictable.
And all of your choices are just as real as they ever were, see the OB sequence on free will (I think someone referred to it already).
And the argument that omega just needs predictive power of 50.5% to cause the paradox only works if it works against ANY arbitrary algorithm. Having that power against any arbitrary algorithm breaks Rice’s Theorem, having that power (or even 100%) against just limited subset of algorithms doesn’t cause the paradox.
If you take strict decision tree precommitment interpretation, then you fix causality. You decide first, omega decides second, game theory says one-box, problem solved.
Decision tree precommitment is never a problem in game theory, as precommitment of the entire tree commutes with decisions by other agents:
A decides what f(X), f(Y) to do if B does X or Y. B does X. A does f(X)
B does X. A decides what f(X), f(Y) to do if B does X or Y. A does f(X)
are identical, as B cannot decide based on f. So the changing your mind problem never occurs.
With omega:
A decides what f(X), f(Y) to do if B does X or Y. B does X. A does f(X) - B can answer depending on f
B does X. A decides what f(X), f(Y) to do if B does X or Y. A does f(X) - somehow not allowed any more
I don’t think the paradox exist in any plausible mathematization of the problem. It looks to me like another of those philosophical problems that exist because of sloppiness of natural language and very little more, I’m just surprised that OB/LW crowd cares about this one and not about others. OK, I admit I really enjoyed it the first time I saw it but just as something fun, nothing more than that.
There’s no need to break causality. You are a being implemented in chaotic wetware. However, there’s no reason to think we couldn’t have rational agents implemented in much more predictable form, as python routines for example, so that any being with superior computation power could simply inspect the source and determine what the output would be.
In such a case, Newcomb-like problems would arise, perfectly lawfully, under normal physics.
In fact, Newcomb-like problems fall naturally out of any ability to simulate and predict the actions of other agents. Omega as described is essentially the limit as predictive power goes to infinity.
This gives me the intuition that trying to decide whether to one-box or two box on newcomb is like trying to decide what 0^0 is; you get your intuition by following a limit process, but that limit process produces different results depending on the path you take.
It would be interesting to look at finitely good predictors. Perhaps we can find something analogous to the result that lim_(x, y -->0) (x^y) is path dependent.
If we define an imperfect predictor as a perfect predictor plus noise, i.e. produces the correct prediction with probability p regardless of the cognition algorithm it’s trying to predict, then Newcomb-like problems are very robust to imperfect prediction: for any p > .5 there is some payoff ratio great enough to preserve the paradox, and the required ratio goes down as the prediction improves. e.g. if 1-boxing gets 100 utilons and 2-boxing gets 1 utilon, then the predictor only needs to be more than 50.5% accurate. So the limit in that direction favors 1-boxing.
What other direction could there be? If the prediction accuracy depends on the algorithm-to-be-predicted (as it would in the real world), then you could try to be an algorithm that is mispredicted in your favor… but a misprediction in your favor can only occur if you actually 2-box, so it only takes a modicum of accuracy before a 1-boxer who tries to be predictable is better off than a 2-boxer who tries to be unpredictable.
I can’t see any other way for the limit to turn out.
If you have two agents trying to precommit not to be blackmailed by each other / precommit not to pay attention to the others precommitment, then any attempt to take a limit of this Newcomblike problem does depend on how you approach the limit. (I don’t know how to solve this problem.)
The value(s) for which the limit is being taken here is unidirectional predictive power, which is loosely a function of the difference in intelligence between the two agents; intuitively, I think a case could be made that (assuming ideal rationality) the total accuracy of mutual behavior prediction between two agents is conserved in some fashion, that doubling the predictive power of one unavoidably would roughly halve the predictive power of the other. Omega represents an entity with a delta-g so large vs. us that predictive power is essentially completely one-sided.
From that basis, allowing the unidirectional predictive power of both agents to go to infinity is probably inherently ill-defined and there’s no reason to expect the problem to have a solution.
Such a being would be different from a human in fundamental ways. Imagine knowing with certainty that your actions can be predicted perfectly by the guy next door, even taking into account that you are trying to be hard to predict?
A (quasi)rational agent with access to genuine randomness (such as a human) is a different matter. A superintelligence could almost perfectly predict the probability distribution over my actions, but by quantum entanglement it would not be able to predict my actual actions.
Whaddaya mean humans are rational agents with access to genuine randomness? That’s what we’re arguing about in the first place!
Perhaps Omega is entangled with your brain such that in all the worlds in which you would choose to one-box, he would predict that you one-box, and all the worlds in which you would choose to two-box, he would predict that you two-box?
In the original formulation, if Omega expects you to flip a coin, he leaves box B empty.
You wouldn’t know this with certainty* because it wouldn’t be true.
(*unless you were delusional)
The guy next door is on roughly your mental level. Thus, the guy next door can’t predict your actions perfectly, because he can’t run a perfect simulation of your mind that’s faster than you. He doesn’t have the capacity.
And he certainly doesn’t have the capacity to simulate the environment, including other people, while doing so.
Humans may or may not generally have access to genuine randomness.
It’s as yet unknown whether we even have run on quantum randomness; and its also unprovable that quantum randomness is actually genuine randomness, and not just based on effects we don’t yet understand, as so many other types of randomness have been.
You’re not taking this in the least convenient possible world. Surely it’s not impossible in principle that your neighbor can simulate you and your environment. Perhaps your neighbor is superintelligent?
It’s ALSO not impossible in principle in the real world. A superintelligent entity could, in principle, perfectly predict my actions. Remember, in the Least Convenient Possible World quantum “randomness” isn’t random.
As such, this ISN’T a fundamental difference between humans and “such beings”. Which was all I set out to demonstrate.
I was using the “most plausible world” on the basis that it seemed pretty clear that that was the one Roko intended. (Where your neighbour isn’t in fact Yahweh in disguise). EDIT: Probably should specify worlds for things in this kind of environment. Thanks, the critical environment here is helping me think about how I think/argue.
If you believe the Many Worlds Interpretation, then quantum randomness just creates copies in a deterministic way.
You cannot do that without breaking Rice’s theorem. If you assume you can find out the answer from someone else’s source code → instant contradiction.
You cannot work around Rice’s theorem or around causality by specifying 50.5% accuracy independently of modeled system, any accuracy higher than 50%+epsilon is equivalent to indefinitely good accuracy by repeatedly predicting (standard cryptographic result), and 50%+epsilon doesn’t cause the paradox.
Give me one serious math model of Newcomb-like problems where the paradox emerges while preserving causality. Here are some examples. Then you model it, you either get trivial solution to one-box, or causality break, or omega loses.
You decide first what you would do in every situation, omega decides second, and now you only implement your initial decision table and are not allowed to switch. Game theory says you should implement one-boxing.
You decide first what you would do in every situation, omega decides second, and now you are allowed to switch. Game theory says you should precommit to one-box, then implement two-boxing, omega loses.
You decide first what you would do in every situation, omega decides second, and now you are allowed to switch. If omega always decides correctly, then he bases his decision on your switch, which either turns it into model #1 (you cannot really switch, precommitment is binding), or breaks causality.
Rice’s theorem says you can’t predict every possible algorithm in general. Plenty of particular algorithms can be predictable. If you’re running on a classical computer and Omega has a copy of you, you are perfectly predictable.
And all of your choices are just as real as they ever were, see the OB sequence on free will (I think someone referred to it already).
And the argument that omega just needs predictive power of 50.5% to cause the paradox only works if it works against ANY arbitrary algorithm. Having that power against any arbitrary algorithm breaks Rice’s Theorem, having that power (or even 100%) against just limited subset of algorithms doesn’t cause the paradox.
If you take strict decision tree precommitment interpretation, then you fix causality. You decide first, omega decides second, game theory says one-box, problem solved.
Decision tree precommitment is never a problem in game theory, as precommitment of the entire tree commutes with decisions by other agents:
A decides what f(X), f(Y) to do if B does X or Y. B does X. A does f(X)
B does X. A decides what f(X), f(Y) to do if B does X or Y. A does f(X)
are identical, as B cannot decide based on f. So the changing your mind problem never occurs.
With omega:
A decides what f(X), f(Y) to do if B does X or Y. B does X. A does f(X) - B can answer depending on f
B does X. A decides what f(X), f(Y) to do if B does X or Y. A does f(X) - somehow not allowed any more
I don’t think the paradox exist in any plausible mathematization of the problem. It looks to me like another of those philosophical problems that exist because of sloppiness of natural language and very little more, I’m just surprised that OB/LW crowd cares about this one and not about others. OK, I admit I really enjoyed it the first time I saw it but just as something fun, nothing more than that.
I don’t know why nobody mentioned this at the time, but that’s hardly an unpopular view around here (as I’m sure you’ve noticed by now).
The interesting thing about Newcomb had nothing to do with thinking it was a genuine paradox—just counterintuitive for some.