I think it’s implicit in the Newcomb’s problem scenario that it takes place within the constraints of the universe as we know it. Obviously we have to make an exception for AIXI itself, but I don’t see a reason to make any further exceptions after that point. Additionally, it is explicitly stated in the problem setup that the contents of the box are supposed to be predetermined, and that the agent is made aware of this aspect of the setup. As far as the epistemic states are concerned, this would imply that AIXI has been presented with a number of prior observations that provide very strong evidential support for this fact.
I agree that AIXI’s universe programs are general Turing machines rather than explicit physics simulations, but I don’t think that’s a particularly big problem. Unless we’re talking about a particularly immature AIXI agent, it should already be aware of the obvious physics-like nature of the real world; it seems to me that the majority of AIXI’s probability mass should be occupied by physics-like Turing machines rather than by thunking. Why would AIXI come up with world programs that involve Omega making money magically appear or disappear after being presented significant evidence to the contrary?
I can agree that in the general case it would be rather difficult indeed to predict AIXI, but in many specific instances I think it’s rather straightforward. In particular, I think Newcomb’s problem is one of those cases.
I guess that in general Omega could be extremely complex, but unless there is a reason Omega needs to be that complex, isn’t it much more sensible to interpret the problem in a way that is more likely to comport with our knowledge of reality? Insofar as there exist simpler explanations for Omega’s predictive power, those simpler explanations should be preferred.
I guess you could say that AIXI itself cannot exist in our reality and so we need to reinterpret the problem in that context, but that seems like a flawed approach to me. After all, the whole point of AIXI is to reason about its performance relative to other agents, so I don’t think it makes sense to posit a different problem setup for AIXI than we would for any other agent.
If AIXI has been presented with sufficient evidence that the Newcomb’s problem works as advertised, then it must be assigning most of its model probability mass to programs where the content of the box, however internally represented, is correlated to the next decision. Such programs exist in the model ensemble, hence the question is how much probability mass does AIXI assign to them. If it not enough to dominate its choice, then by definition AIXI has not been presented with enough evidence.
What do you mean by “programs where the content of the box, however internally represented, is correlated to the next decision”? Do you mean world programs that output $1,000,000 when the input is “one-box” and output $1000 when the input is “two-box”? That seems to contradict the setup of Newcomb’s to me; in order for Newcomb’s problem to work, the content of the box has to be correlated to the actual next decision, not to counterfactual next decisions that don’t actually occur.
As such, as far as I can see it’s important for AIXI’s probability mass to focus down to models where the box already contains a million dollars and/or models where the box is already empty, rather than models in which the contents of the box are determined by the input to the world program at the moment AIXI makes its decision.
AIXI world programs have no inputs, they just run and produce sequences of triples in the form: (action, percept, reward).
So, let’s say AIXI has been just subjected to Newcomb’s problem. Assuming that the decision variable is always binary (“OneBox” vs “TwoBox”), of all the programs which produce a sequence consistent with the observed history, we distinguish five classes of programs, depending on the next triple they produce: 1: (“OneBox”, “Opaque box contains $1,000,000“, 1,000,000) 2: (“TwoBox”, “Opaque box is empty”, 1,000) 3: (“OneBox”, “Opaque box is empty”, 0) 4: (“TwoBox”, “Opaque box contains $1,000,000”, 1,001,000) 5: Anything else (eg. (“OneBox”, “A pink elephant appears”, 42)).
Class 5 should have a vanishing probability, since we assume that the agent already knows physics. Therefore: E(“OneBox”) = (1,000,000 p(class1) + 0 p(class3)) / (p(class1) + p(class3)) E(“TwoBox”) = (1,000 p(class2) + 1,001,000 p(class4)) / (p(class2) + p(class4))
Classes 1 and 2 are consistent with the setup of Newcomb’s problem, while classes 3 and 4 aren’t. Hence I would say that if AIXI has been presented with enough evidence to believe that it is facing Newcomb’s problem, then by definition of “enough evidence”, p(class1) >> p(class3) and p(class2) >> p(class4), implying that AIXI will OneBox.
AIXI world programs have no inputs, they just run and produce sequences of triples in the form: (action, percept, reward).
No, that isn’t true. See, for example, page 7 of this article. The environments (q) accept inputs from the agent and output the agent’s percepts.
As such (as per my discussion with private_messaging), there are only three relevant classes of world programs: (1) Opaque box contains $1,000,000 (2) Opaque box is empty (3) Contents of the box are determined by my action input
For any and all such environment programs that are consistent with AIXI’s observations to date, AIXI will evaluate the reward for both the OneBox and TwoBox actions. As long as classes (1) and (2) win out over class (3), which they should due to being simpler, AIXI will determine that the E(TwoBox) > E(OneBox) and therefore AIXI will TwoBox. In fact, as long as AIXI is smart enough to predict Omega’s reasoning, world programs of type (2) should win out over type (1) as well, and so AIXI will already be pretty sure that the opaque box is empty when it two-boxes.
The environments (q) accept inputs from the agent and output the agent’s percepts.
Yes, but the programs that AIXI maintains internally in its model ensemble are defined as input-less programs that generate all the possible histories. AIXI filters them for the one observed history and then evaluates the expected (discounted) reward over the future histories, for each possible choice of its next action. Anyway, that’s a technical detail.
As long as classes (1) and (2) win out over class (3), which they should due to being simpler
How can they be simpler, given that you have explained to AIXI what Newcomb’s problem is and provided it with enough evidence so that it really believes that it is going to face it?
Maybe Newcomb’s problem is simply inconceivable to AIXI, in a way that no amount of evidence can ever lead it to expect that the content of the box, and thus the reward, is correlated to its action. That’s a possibility, but I find it not very plausible: AIXI world programs contain embeddings of all human minds, and all super-human computable AIs. If we assume that the agent is experienced, world programs embedding these very very smart AIs will get most of probability mass, since they are very good sequence predictors. So if a human can understand Newcomb’s problem, I think that a super-human AI would understand it as well.
Anyway, if we stipulate that it is indeed possible to provide AIXI with enough evidence that it is facing Newcomb’s problem, then it seems to me that it will OneBox.
Maybe Newcomb’s problem is simply inconceivable to AIXI, in a way that no amount of evidence can ever lead it to expect that the content of the box, and thus the reward, is correlated to its action.
AIXI does recognise this correlation; it two-boxes and with a reasonable amount of evidence it also believes (correctly) that Omega predicted it would two-box.
That’s a possibility, but I find it not very plausible: AIXI world programs contain embeddings of all human minds, and all super-human computable AIs. If we assume that the agent is experienced, world programs embedding these very very smart AIs will get most of probability mass, since they are very good sequence predictors. So if a human can understand Newcomb’s problem, I think that a super-human AI would understand it as well.
The problem is that AIXI cannot recognise the kinds of models in which AIXI’s own action and Omega’s prediction of its action have a common cause (i.e. the AIXI equation). A better agent would be capable of recognising that dependency.
If you always exclude certain kinds of models then it doesn’t matter how smart you are, some explanations are simply never going to occur to you.
If you always exclude certain kinds of models then it doesn’t matter how smart you are, some explanations are simply never going to occur to you.
Actually, these models exist in AIXI world program ensemble. In order to support your point, you have to argue that they are more complex than models which make an incorrect prediction, no matter how much evidence for Newcomb’s problem AIXI has been presented with.
Yes, but the programs that AIXI maintains internally in its model ensemble are defined as input-less programs that generate all the possible histories.
Please clarify this and/or give a reference. Every time I’ve seen the equation AIXI’s actions are inputs to the environment program.
How can they be simpler, given that you have explained to AIXI what Newcomb’s problem is and provided it with enough evidence so that it really believes that it is going to face it?
The point of Newcomb’s problem is that the contents of the box are already predetermined; it’s stipulated that as part of the problem setup you are given enough evidence of this. In general, any explanation that involves AIXI’s action directly affecting the contents of the box will be more complex because it bypasses the physics-like explanation that AIXI would have for everything else.
When I am facing Newcomb’s problem I don’t believe that the box magically changes contents as the result of my action—that would be stupid. I believe that the box already has the million dollars because I’m predictably a one-boxer, and then I one-box.
Similarly, if AIXI is facing Newcomb’s then it should, without a particularly large amount of evidence, also narrow its environment programs down to ones that already either contains the million, and ones that already do not.
EDIT: Wait, perhaps we agree re. the environment programs.
AIXI filters them for the one observed history and then evaluates the expected (discounted) reward over the future histories, for each possible choice of its next action.
Yes, for each possible choice. As such, if AIXI has an environment program “q” in which Omega already predicted one-boxing and put the million dollars in, AIXI will check the outcome of OneBox as well as the outcome of TwoBox with that same “q”.
Please clarify this and/or give a reference. Every time I’ve seen the equation AIXI’s actions are inputs to the environment program.
Eq 22 in the paper you linked, trace the definitions back to eq. 16, which describes Solomonoff induction. It uses input-less programs to obtain the joint probability distribution, then it divides it by the marginal distribution to obtain the conditional probability distribution it needs.
(Anyway, Hutter’s original papers are somewhat difficult to read due to their heavy notation, I find Shane Legg’s PhD thesis more readable.)
The point of Newcomb’s problem is that the contents of the box are already predetermined; it’s stipulated that as part of the problem setup you are given enough evidence of this. In general, any explanation that involves AIXI’s action directly affecting the contents of the box will be more complex because it bypasses the physics-like explanation that AIXI would have for everything else.
If you tell AIXI: “Look, the transparent box contains $1,000 and the opaque box may contain $0 or $1,000,000. Do you want to take the content only of the opaque box or both?”, then AIXI will two-box, just as you would. Clearly the scenario where there is no Omega and the content of the opaque box is independent on your action is simpler than Newcomb’s problem.
But if you convince AIXI that it’s actually facing Newcomb’s problem, then its surviving world-programs must model the action of Omega somewhere in their “physics modules”. The simplest way of doing that is probably to assume that there is some physical variable which determines AIXI next action (remember, the world programs predict actions as well as the inputs), and Omega can observe it and use it to set the content of the opaque box. Or maybe they can assume that Omega has a time machine or something. Different programs in the ensemble will model Omega in a different way, but the point is that in order to be epistemically correct, the probability mass of programs that model Omega must be greater than the probability mass of programs that don’t.
Eq 22 in the paper you linked, trace the definitions back to eq. 16, which describes Solomonoff induction.
It uses input-less programs to obtain the joint probability distribution, then it divides it by the marginal distribution to obtain the conditional probability distribution it needs.
Nope, the environment q is a chronological program; it takes AIXI’s action sequence and outputs an observation sequence, with the restriction that observations cannot be dependent upon future actions.
Basically, it is assumed that the universal Turing machine U is fed both the environment program q, and AIXI’s action sequence y, and outputs AIXI’s observation sequence x by running the program q with input y. Quoting from the paper I linked: ”Reversely, if q already is a binary string we define q(y):=U(q,y)”
In the paper I linked, see Eq. 21:
%20=%20\sum_{q:q(y_{1:k})=x_{1:k}}2%5E{-l(q)}) and the the term
) from Eq. 22.
In other words, any program q that matches AIXI’s observations to date when given AIXI’s actions to date will be part of the ensemble. In order to evaluate different future action sequences, AIXI then evaluates the different future actions it could take by feeding them to its program ensemble, and summing over different possible future rewards conditional on the environments that output those rewards.
If you tell AIXI: “Look, the transparent box contains $1,000 and the opaque box may contain $0 or $1,000,000. Do you want to take the content only of the opaque box or both?”, then AIXI will two-box, just as you would.
Clearly the scenario where there is no Omega and the content of the opaque box is independent on your action is simpler than Newcomb’s problem.
The CDT agent can correctly argue that Omega already left the million dollars out of the box when the CDT agent was presented the choice, but that doesn’t mean that it’s correct to be a CDT agent. My argument is that AIXI suffers from the same flaw, and so a different algorithm is needed.
But if you convince AIXI that it’s actually facing Newcomb’s problem, then its surviving world-programs must model the action of Omega somewhere in their “physics modules”.
Correct. My point is that AIXI’s surviving world-programs boil down to “Omega predicted I would two-box, and didn’t put the million dollars in”, but it’s the fault of the AIXI algorithm that this happens.
The simplest way of doing that is probably to assume that there is some physical variable which determines AIXI next action (remember, the world programs predict actions as well as the inputs), and Omega can observe it and use it to set the content of the opaque box. Or maybe they can assume that Omega has a time machine or something.
As per the AIXI equations, this is incorrect. AIXI cannot recognize the presence of a physical variable determining its next action because for any environment program AIXI’s evaluation stage is always going to try both the OneBox and TwoBox actions. Given the three classes of programs above, the only way AIXI can justify one-boxing is if the class (3) programs, in which its action somehow causes the contents of the box, win out.
“Reversely, if q already is a binary string we define q(y):=U(q,y)”
Ok, missed that. I don’t think it matters to the rest of the argument, though.
As per the AIXI equations, this is incorrect. AIXI cannot recognize the presence of a physical variable determining its next action because for any environment program AIXI’s evaluation stage is always going to try both the OneBox and TwoBox actions. Given the three classes of programs above, the only way AIXI can justify one-boxing is if the class (3) programs, in which its action somehow causes the contents of the box, win out.
An environment program can just assume a value for the physical variable and then abort by failing to halt if the next action doesn’t match it. Or it can assume that the physical simulation branches at time t0, when Omega prepares the box, simulate each branch it until t < t1, when the next AIXI action occurs, and then kill off the branch corresponding to the wrong action. Or, as it has already been proposed by somebody else, it could internally represent the physical world as a set of constraints and then run a constraint solver on it, without the need of performing a step-by-step chronological simulation.
So it seems that there are plenty of environment programs that can represent the action of Omega without assuming that it violates the known laws of physics. But even if it had to, what is the problem? AIXI doesn’t assume that the laws of physics forbid retro-causality.
An environment program can just assume a value for the physical variable and then abort by failing to halt if the next action doesn’t match it.
Why would AIXI come up with something like that? Any such program is clearly more complex than one that does the same thing but doesn’t fail to halt.
Or it can assume that the physical simulation branches at time t0, when Omega prepares the box, simulate each branch it until t < t1, when the next AIXI action occurs, and then kill off the branch corresponding to the wrong action.
Once again, possible but unnecessarily complex to explain AIXI’s observations.
Or, as it has already been proposed by somebody else, it could internally represent the physical world as a set of constraints and then run a constraint solver on it, without the need of performing a step-by-step chronological simulation.
Sure, but the point is that those constraints would still be physics-like in nature. Omega’s prediction accuracy is much better explained by constraints that are physics-like rather than an extra constraint that says “Omega is always right”. if you assume a constraint of the latter kind, you are still forced to explain all the other aspects of Omega—things like Omega walking, Omega speaking, and Omega thinking, or more generally Omega doing all those things that ze does. Also, if Omega isn’t always right, but is instead right only 99% of the time, then the constraint-based approach is penalized further.
So it seems that there are plenty of environment programs that can represent the action of Omega without assuming that it violates the known laws of physics. But even if it had to, what is the problem? AIXI doesn’t assume that the laws of physics forbid retro-causality.
It doesn’t assume that, no, but because it assumes that its observations cannot be affected by its future actions AIXI is still very much restricted in that regard.
My point is a simple one:
If AIXI is going to end-up one-boxing, the simplest model of Omega will be one that used its prediction method and already predicted that AIXI would one-box.
If AIXI is going to end up two-boxing, the simplest model of Omega will be one that used its prediction method and already predicted that AIXI would two-box.
However, if Omega predicted one-boxing and AIXI realized that this was the case, AIXI would still evaluate that the two-boxing action results in AIXI getting more money than the one-boxing action, which means that AIXI would two-box.
As long as Omega is capable of reaching this relatively simple logical conclusion, Omega thereby knows that a prediction of one-boxing would turn out to be wrong, and hence Omega should predict two-boxing; this will, of course, turn out to be correct.
The kinds of models you’re suggesting, with branching etc. are significantly more complex and don’t really serve to explain anything.
It doesn’t assume that, no, but because it assumes that its observations cannot be affected by its future actions AIXI is still very much restricted in that regards.
But this doesn’t matter for Newcomb’s problem, since AIXI observes the content of the opaque box only after it has made its decision.
However, if Omega predicted one-boxing and AIXI realized that this was the case, AIXI would still evaluate that the two-boxing action results in AIXI getting more money than the one-boxing action, which means that AIXI would two-box.
Which means that the epistemic model was flawed with high probability. You are insisting that the flawed model is simpler that the correct one. This may be true for certain states of evidence where AIXI is not convinced that Omega works as advertised, but you haven’t shown that this is true for all possible states of evidence.
The kinds of models you’re suggesting, with branching etc. are significantly more complex and don’t really serve to explain anything.
They may be more complex only up to a small constant overhead (how many bits does it take to include a condition “if OmegaPrediction != NextAction then loop forever”?), therefore, a constant amount of evidence should be sufficient to select them.
Which means that the epistemic model was flawed with high probability.
You are insisting that the flawed model is simpler that the correct one. This may be true for certain states of evidence where AIXI is not convinced that Omega works as advertised, but you haven’t shown that this is true for all possible states of evidence.
Yes, AIXI’s epistemic model will be flawed. This is necessarily true because AIXI is not capable of coming up with the true model of Newcomb’s problem, which is one in which its action and Omega’s prediction of its action share a common cause. Since AIXI isn’t capable of having a self-model, the only way it could possibly replicate the behaviour of the true model is by inserting retrocausality and/or magic into its environment.
They may be more complex only up to a small constant overhead (how many bits does it take to include a condition “if OmegaPrediction != NextAction then loop forever”?), therefore, a constant amount of evidence should be sufficient to select them.
I’m not even sure AIXI is capable of considering programs of this kind, but even if it is, what kind of evidence can AIXI have received that would justify the condition “if OmegaPrediction != NextAction then loop forever”? What evidence would justify such a model over a strictly simpler version without that condition?
Essentially, you’re arguing that rather than coming up with a correct model of its environment (e.g. one in which Omega makes a prediction on the basis of the AIXI equation), AIXI will somehow make up for its inability to self-model by coming up with an inaccurate and obviously false retrocausal/magical model of its environment instead.
However, I don’t see why this would be the case. It’s quite clear that either Omega has already predicted one-boxing, or Omega has already predicted two-boxing. At the very least, the evidence should narrow things down to models of either kind, although I think that AIXI should easily have sufficient evidence to work out which of them is actually true (that being the two-boxing one).
I agree that AIXI’s universe programs are general Turing machines rather than explicit physics simulations, but I don’t think that’s a particularly big problem. Unless we’re talking about a particularly immature AIXI agent, it should already be aware of the obvious physics-like nature of the real world; it seems to me that the majority of AIXI’s probability mass should be occupied by physics-like Turing machines rather than by thunking. Why would AIXI come up with world programs that involve Omega making money magically appear or disappear after being presented significant evidence to the contrary?
The problem is not “programs that make money magically (dis)appear for the box after the fact” but rather programs that don’t explicitly represent the presence or nonpresence of money at all until it is known. For example, a constraint solver that seeks a proof of AIXI’s observations when they are called for (using a logic that expresses normal physics). This gives all the right answers, and is fairly simple but does allow the content of the box to be controlled by the decision.
Such models would generally not offer good explanations for why Omega is so good at predicting all those other agents who aren’t AIXI, and would be penalized for this. On the other hand, any model that explains Omega’s general predictive power would be made more complex by adding a special case just for AIXI.
I don’t understand what you mean by “a constraint solver that seeks a proof of AIXI’s observations when they are called for.” Can you explain it further?
A proof system that starts with some axioms describing the physical world (excluding the AIXI machine itself), and when run with input a_1 .. a_m being AIXI’s actions so far, plugs them in as axioms about AIXI’s control wires, and attempts to prove a statement of the form ‘AIXI’s input wire observes a 1 at time t’ or ‘AIXI’s input wire observes a 0 at time t’. And returns the first answer it finds.
It takes, as input, a description of the agent it’s predicting; typically source code, but in the case of AIXI, it gets the AIXI equation and a sequence of prior observations for AIXI.
As for what it does, it spends some period of time (maybe a very long one) on whatever kind of deductive and/or inductive reasoning it chooses to do in order to establish with a reasonable level of confidence what the agent it’s trying to predict will do.
Yes, AIXI being uncomputable means that Omega can’t simply run the equation for itself, but there is no need for a perfect prediction here. On the whole, it just needs to be able to come up with a well-reasoned argument for why AIXI will take a particular action, or perhaps run an approximation of AIXI for a while. Moreover, anyone in this thread arguing for either one-boxing or two-boxing has already implicitly agreed with this assumption.
Yes, AIXI being uncomputable means that Omega can’t simply run the equation for itself, but there is no need for a perfect prediction here. On the whole, it just needs to be able to come up with a well-reasoned argument for why AIXI will take a particular action, or perhaps run an approximation of AIXI for a while.
This opens up the possibility that AIXI figures out that Omega is going to mispredict it, which would make TwoBoxing the best decision.
Moreover, anyone in this thread arguing for either one-boxing or two-boxing has already implicitly agreed with this assumption.
I think it is generally assumed that, even if Omega is not a perfect predictor, the agent can’t outsmart it and predict its errors. But if Omega is computable and the agent is uncomputable, this doesn’t necessarily hold true.
Moreover, anyone in this thread arguing for either one-boxing or two-boxing has already implicitly agreed with this assumption.
I’m not so sure this is true now. People in this thread arguing that AIXI does something at least have the advantage that AIXI’s decision is not going to depend on how they do the arguing. The fact that AIXI can simulate Omega with perfect fidelity (assuming Omega is not also a hypercomputer) and will make its decision based on the simulation seems like it might impact Omega’s ability to make a good prediction.
I think it’s implicit in the Newcomb’s problem scenario that it takes place within the constraints of the universe as we know it. Obviously we have to make an exception for AIXI itself, but I don’t see a reason to make any further exceptions after that point. Additionally, it is explicitly stated in the problem setup that the contents of the box are supposed to be predetermined, and that the agent is made aware of this aspect of the setup. As far as the epistemic states are concerned, this would imply that AIXI has been presented with a number of prior observations that provide very strong evidential support for this fact.
I agree that AIXI’s universe programs are general Turing machines rather than explicit physics simulations, but I don’t think that’s a particularly big problem. Unless we’re talking about a particularly immature AIXI agent, it should already be aware of the obvious physics-like nature of the real world; it seems to me that the majority of AIXI’s probability mass should be occupied by physics-like Turing machines rather than by thunking. Why would AIXI come up with world programs that involve Omega making money magically appear or disappear after being presented significant evidence to the contrary?
I can agree that in the general case it would be rather difficult indeed to predict AIXI, but in many specific instances I think it’s rather straightforward. In particular, I think Newcomb’s problem is one of those cases.
I guess that in general Omega could be extremely complex, but unless there is a reason Omega needs to be that complex, isn’t it much more sensible to interpret the problem in a way that is more likely to comport with our knowledge of reality? Insofar as there exist simpler explanations for Omega’s predictive power, those simpler explanations should be preferred.
I guess you could say that AIXI itself cannot exist in our reality and so we need to reinterpret the problem in that context, but that seems like a flawed approach to me. After all, the whole point of AIXI is to reason about its performance relative to other agents, so I don’t think it makes sense to posit a different problem setup for AIXI than we would for any other agent.
If AIXI has been presented with sufficient evidence that the Newcomb’s problem works as advertised, then it must be assigning most of its model probability mass to programs where the content of the box, however internally represented, is correlated to the next decision.
Such programs exist in the model ensemble, hence the question is how much probability mass does AIXI assign to them. If it not enough to dominate its choice, then by definition AIXI has not been presented with enough evidence.
What do you mean by “programs where the content of the box, however internally represented, is correlated to the next decision”? Do you mean world programs that output $1,000,000 when the input is “one-box” and output $1000 when the input is “two-box”? That seems to contradict the setup of Newcomb’s to me; in order for Newcomb’s problem to work, the content of the box has to be correlated to the actual next decision, not to counterfactual next decisions that don’t actually occur.
As such, as far as I can see it’s important for AIXI’s probability mass to focus down to models where the box already contains a million dollars and/or models where the box is already empty, rather than models in which the contents of the box are determined by the input to the world program at the moment AIXI makes its decision.
AIXI world programs have no inputs, they just run and produce sequences of triples in the form: (action, percept, reward).
So, let’s say AIXI has been just subjected to Newcomb’s problem. Assuming that the decision variable is always binary (“OneBox” vs “TwoBox”), of all the programs which produce a sequence consistent with the observed history, we distinguish five classes of programs, depending on the next triple they produce:
1: (“OneBox”, “Opaque box contains $1,000,000“, 1,000,000)
2: (“TwoBox”, “Opaque box is empty”, 1,000)
3: (“OneBox”, “Opaque box is empty”, 0)
4: (“TwoBox”, “Opaque box contains $1,000,000”, 1,001,000)
5: Anything else (eg. (“OneBox”, “A pink elephant appears”, 42)).
Class 5 should have a vanishing probability, since we assume that the agent already knows physics.
Therefore:
E(“OneBox”) = (1,000,000 p(class1) + 0 p(class3)) / (p(class1) + p(class3))
E(“TwoBox”) = (1,000 p(class2) + 1,001,000 p(class4)) / (p(class2) + p(class4))
Classes 1 and 2 are consistent with the setup of Newcomb’s problem, while classes 3 and 4 aren’t.
Hence I would say that if AIXI has been presented with enough evidence to believe that it is facing Newcomb’s problem, then by definition of “enough evidence”, p(class1) >> p(class3) and p(class2) >> p(class4), implying that AIXI will OneBox.
EDIT: math.
No, that isn’t true. See, for example, page 7 of this article. The environments (q) accept inputs from the agent and output the agent’s percepts.
As such (as per my discussion with private_messaging), there are only three relevant classes of world programs:
(1) Opaque box contains $1,000,000
(2) Opaque box is empty
(3) Contents of the box are determined by my action input
For any and all such environment programs that are consistent with AIXI’s observations to date, AIXI will evaluate the reward for both the OneBox and TwoBox actions. As long as classes (1) and (2) win out over class (3), which they should due to being simpler, AIXI will determine that the E(TwoBox) > E(OneBox) and therefore AIXI will TwoBox. In fact, as long as AIXI is smart enough to predict Omega’s reasoning, world programs of type (2) should win out over type (1) as well, and so AIXI will already be pretty sure that the opaque box is empty when it two-boxes.
Yes, but the programs that AIXI maintains internally in its model ensemble are defined as input-less programs that generate all the possible histories.
AIXI filters them for the one observed history and then evaluates the expected (discounted) reward over the future histories, for each possible choice of its next action.
Anyway, that’s a technical detail.
How can they be simpler, given that you have explained to AIXI what Newcomb’s problem is and provided it with enough evidence so that it really believes that it is going to face it?
Maybe Newcomb’s problem is simply inconceivable to AIXI, in a way that no amount of evidence can ever lead it to expect that the content of the box, and thus the reward, is correlated to its action.
That’s a possibility, but I find it not very plausible: AIXI world programs contain embeddings of all human minds, and all super-human computable AIs. If we assume that the agent is experienced, world programs embedding these very very smart AIs will get most of probability mass, since they are very good sequence predictors. So if a human can understand Newcomb’s problem, I think that a super-human AI would understand it as well.
Anyway, if we stipulate that it is indeed possible to provide AIXI with enough evidence that it is facing Newcomb’s problem, then it seems to me that it will OneBox.
AIXI does recognise this correlation; it two-boxes and with a reasonable amount of evidence it also believes (correctly) that Omega predicted it would two-box.
The problem is that AIXI cannot recognise the kinds of models in which AIXI’s own action and Omega’s prediction of its action have a common cause (i.e. the AIXI equation). A better agent would be capable of recognising that dependency.
If you always exclude certain kinds of models then it doesn’t matter how smart you are, some explanations are simply never going to occur to you.
Actually, these models exist in AIXI world program ensemble. In order to support your point, you have to argue that they are more complex than models which make an incorrect prediction, no matter how much evidence for Newcomb’s problem AIXI has been presented with.
Please clarify this and/or give a reference. Every time I’ve seen the equation AIXI’s actions are inputs to the environment program.
The point of Newcomb’s problem is that the contents of the box are already predetermined; it’s stipulated that as part of the problem setup you are given enough evidence of this. In general, any explanation that involves AIXI’s action directly affecting the contents of the box will be more complex because it bypasses the physics-like explanation that AIXI would have for everything else.
When I am facing Newcomb’s problem I don’t believe that the box magically changes contents as the result of my action—that would be stupid. I believe that the box already has the million dollars because I’m predictably a one-boxer, and then I one-box.
Similarly, if AIXI is facing Newcomb’s then it should, without a particularly large amount of evidence, also narrow its environment programs down to ones that already either contains the million, and ones that already do not.
EDIT: Wait, perhaps we agree re. the environment programs.
Yes, for each possible choice. As such, if AIXI has an environment program “q” in which Omega already predicted one-boxing and put the million dollars in, AIXI will check the outcome of OneBox as well as the outcome of TwoBox with that same “q”.
Eq 22 in the paper you linked, trace the definitions back to eq. 16, which describes Solomonoff induction.
It uses input-less programs to obtain the joint probability distribution, then it divides it by the marginal distribution to obtain the conditional probability distribution it needs.
(Anyway, Hutter’s original papers are somewhat difficult to read due to their heavy notation, I find Shane Legg’s PhD thesis more readable.)
If you tell AIXI: “Look, the transparent box contains $1,000 and the opaque box may contain $0 or $1,000,000. Do you want to take the content only of the opaque box or both?”, then AIXI will two-box, just as you would.
Clearly the scenario where there is no Omega and the content of the opaque box is independent on your action is simpler than Newcomb’s problem.
But if you convince AIXI that it’s actually facing Newcomb’s problem, then its surviving world-programs must model the action of Omega somewhere in their “physics modules”.
The simplest way of doing that is probably to assume that there is some physical variable which determines AIXI next action (remember, the world programs predict actions as well as the inputs), and Omega can observe it and use it to set the content of the opaque box. Or maybe they can assume that Omega has a time machine or something.
Different programs in the ensemble will model Omega in a different way, but the point is that in order to be epistemically correct, the probability mass of programs that model Omega must be greater than the probability mass of programs that don’t.
Nope, the environment q is a chronological program; it takes AIXI’s action sequence and outputs an observation sequence, with the restriction that observations cannot be dependent upon future actions. Basically, it is assumed that the universal Turing machine U is fed both the environment program q, and AIXI’s action sequence y, and outputs AIXI’s observation sequence x by running the program q with input y. Quoting from the paper I linked:
”Reversely, if q already is a binary string we define q(y):=U(q,y)”
In the paper I linked, see Eq. 21:
and the the term
In other words, any program q that matches AIXI’s observations to date when given AIXI’s actions to date will be part of the ensemble. In order to evaluate different future action sequences, AIXI then evaluates the different future actions it could take by feeding them to its program ensemble, and summing over different possible future rewards conditional on the environments that output those rewards.
The CDT agent can correctly argue that Omega already left the million dollars out of the box when the CDT agent was presented the choice, but that doesn’t mean that it’s correct to be a CDT agent. My argument is that AIXI suffers from the same flaw, and so a different algorithm is needed.
Correct. My point is that AIXI’s surviving world-programs boil down to “Omega predicted I would two-box, and didn’t put the million dollars in”, but it’s the fault of the AIXI algorithm that this happens.
As per the AIXI equations, this is incorrect. AIXI cannot recognize the presence of a physical variable determining its next action because for any environment program AIXI’s evaluation stage is always going to try both the OneBox and TwoBox actions. Given the three classes of programs above, the only way AIXI can justify one-boxing is if the class (3) programs, in which its action somehow causes the contents of the box, win out.
Ok, missed that. I don’t think it matters to the rest of the argument, though.
An environment program can just assume a value for the physical variable and then abort by failing to halt if the next action doesn’t match it.
Or it can assume that the physical simulation branches at time t0, when Omega prepares the box, simulate each branch it until t < t1, when the next AIXI action occurs, and then kill off the branch corresponding to the wrong action.
Or, as it has already been proposed by somebody else, it could internally represent the physical world as a set of constraints and then run a constraint solver on it, without the need of performing a step-by-step chronological simulation.
So it seems that there are plenty of environment programs that can represent the action of Omega without assuming that it violates the known laws of physics. But even if it had to, what is the problem? AIXI doesn’t assume that the laws of physics forbid retro-causality.
Why would AIXI come up with something like that? Any such program is clearly more complex than one that does the same thing but doesn’t fail to halt.
Once again, possible but unnecessarily complex to explain AIXI’s observations.
Sure, but the point is that those constraints would still be physics-like in nature. Omega’s prediction accuracy is much better explained by constraints that are physics-like rather than an extra constraint that says “Omega is always right”. if you assume a constraint of the latter kind, you are still forced to explain all the other aspects of Omega—things like Omega walking, Omega speaking, and Omega thinking, or more generally Omega doing all those things that ze does. Also, if Omega isn’t always right, but is instead right only 99% of the time, then the constraint-based approach is penalized further.
It doesn’t assume that, no, but because it assumes that its observations cannot be affected by its future actions AIXI is still very much restricted in that regard.
My point is a simple one: If AIXI is going to end-up one-boxing, the simplest model of Omega will be one that used its prediction method and already predicted that AIXI would one-box. If AIXI is going to end up two-boxing, the simplest model of Omega will be one that used its prediction method and already predicted that AIXI would two-box.
However, if Omega predicted one-boxing and AIXI realized that this was the case, AIXI would still evaluate that the two-boxing action results in AIXI getting more money than the one-boxing action, which means that AIXI would two-box. As long as Omega is capable of reaching this relatively simple logical conclusion, Omega thereby knows that a prediction of one-boxing would turn out to be wrong, and hence Omega should predict two-boxing; this will, of course, turn out to be correct.
The kinds of models you’re suggesting, with branching etc. are significantly more complex and don’t really serve to explain anything.
But this doesn’t matter for Newcomb’s problem, since AIXI observes the content of the opaque box only after it has made its decision.
Which means that the epistemic model was flawed with high probability.
You are insisting that the flawed model is simpler that the correct one. This may be true for certain states of evidence where AIXI is not convinced that Omega works as advertised, but you haven’t shown that this is true for all possible states of evidence.
They may be more complex only up to a small constant overhead (how many bits does it take to include a condition “if OmegaPrediction != NextAction then loop forever”?), therefore, a constant amount of evidence should be sufficient to select them.
Yes, AIXI’s epistemic model will be flawed. This is necessarily true because AIXI is not capable of coming up with the true model of Newcomb’s problem, which is one in which its action and Omega’s prediction of its action share a common cause. Since AIXI isn’t capable of having a self-model, the only way it could possibly replicate the behaviour of the true model is by inserting retrocausality and/or magic into its environment.
I’m not even sure AIXI is capable of considering programs of this kind, but even if it is, what kind of evidence can AIXI have received that would justify the condition “if OmegaPrediction != NextAction then loop forever”? What evidence would justify such a model over a strictly simpler version without that condition?
Essentially, you’re arguing that rather than coming up with a correct model of its environment (e.g. one in which Omega makes a prediction on the basis of the AIXI equation), AIXI will somehow make up for its inability to self-model by coming up with an inaccurate and obviously false retrocausal/magical model of its environment instead.
However, I don’t see why this would be the case. It’s quite clear that either Omega has already predicted one-boxing, or Omega has already predicted two-boxing. At the very least, the evidence should narrow things down to models of either kind, although I think that AIXI should easily have sufficient evidence to work out which of them is actually true (that being the two-boxing one).
The problem is not “programs that make money magically (dis)appear for the box after the fact” but rather programs that don’t explicitly represent the presence or nonpresence of money at all until it is known. For example, a constraint solver that seeks a proof of AIXI’s observations when they are called for (using a logic that expresses normal physics). This gives all the right answers, and is fairly simple but does allow the content of the box to be controlled by the decision.
Such models would generally not offer good explanations for why Omega is so good at predicting all those other agents who aren’t AIXI, and would be penalized for this. On the other hand, any model that explains Omega’s general predictive power would be made more complex by adding a special case just for AIXI.
I don’t understand what you mean by “a constraint solver that seeks a proof of AIXI’s observations when they are called for.” Can you explain it further?
A proof system that starts with some axioms describing the physical world (excluding the AIXI machine itself), and when run with input
a_1 .. a_mbeing AIXI’s actions so far, plugs them in as axioms about AIXI’s control wires, and attempts to prove a statement of the form ‘AIXI’s input wire observes a 1 at timet’ or ‘AIXI’s input wire observes a 0 at timet’. And returns the first answer it finds.Alternatively, what about a version of Newcomb’s problem where the predictor’s source code is shown to AIXI before it makes its decision?
What would the source code of an Omega able to predict an AIXI look like?
It won’t have source code per se, but one can posit the existence of a halting oracle without generating an inconsistency.
It takes, as input, a description of the agent it’s predicting; typically source code, but in the case of AIXI, it gets the AIXI equation and a sequence of prior observations for AIXI.
As for what it does, it spends some period of time (maybe a very long one) on whatever kind of deductive and/or inductive reasoning it chooses to do in order to establish with a reasonable level of confidence what the agent it’s trying to predict will do.
Yes, AIXI being uncomputable means that Omega can’t simply run the equation for itself, but there is no need for a perfect prediction here. On the whole, it just needs to be able to come up with a well-reasoned argument for why AIXI will take a particular action, or perhaps run an approximation of AIXI for a while. Moreover, anyone in this thread arguing for either one-boxing or two-boxing has already implicitly agreed with this assumption.
This opens up the possibility that AIXI figures out that Omega is going to mispredict it, which would make TwoBoxing the best decision.
I think it is generally assumed that, even if Omega is not a perfect predictor, the agent can’t outsmart it and predict its errors. But if Omega is computable and the agent is uncomputable, this doesn’t necessarily hold true.
I’m not so sure this is true now. People in this thread arguing that AIXI does something at least have the advantage that AIXI’s decision is not going to depend on how they do the arguing. The fact that AIXI can simulate Omega with perfect fidelity (assuming Omega is not also a hypercomputer) and will make its decision based on the simulation seems like it might impact Omega’s ability to make a good prediction.