) is your player and PD_alpha is the payoff of the first player in the Prisoner Dilemma with the (1,5,6) payoff matrix.
Omega’s players end up playing C regardless of A. The agent can either understand this or at least fail to find a strong dependence of the logical probabilities of Omega’s players’ strategy on either their input (the agent’s source) or the conditions in the expectation values it is evaluating (since the conditions are of the form
=X) which seems to be correlated with
)) only in the obvious way i.e. by determining the input to B_i).
Therefore, the highest expectation values will be computed for conditions of the form
=DefectBot). Therefore the agent will defect.
I guess it wasn’t mentioned explicitly in that discussion, but it’s how I’ve come to think of the problem. Perhaps the most relevant part of that discussion is Eliezer’s direct reply, here.
I see. However, your problem doesn’t seem to be a realistic model of acausal bargaining with agents in other universes, since in such bargaining you know who you’re cooperating with. For example, when an agent considers filling its universe with human utility, it does it in order to cooperate with a human FAI, not in order to cooperate with a paperclip maximizer (which would require a very different strategy namely filling its universe with paperclips).
It’s more of a model for FAI meeting an alien AI in space. Suppose each side then has the choice of doing an arms buildup or not, and the payoffs for these choices are analogous to PD. (If one side builds up while the other doesn’t, it can attack and conquer the other. If both sides build up, it’s a stalemate and just wastes resources.) BTW, what was your “UDT anti-Newcomb problem” intended to be a model for?
I guess if both sides are using your decision theory, then whether the human FAI plays C or D against the alien AI depends on how much logical correlation the FAI thinks exists between its human designer and the alien AI’s designer, which does make sense (assuming we solve the problem of the FAI just simulating its designer and already knowing what their output is).
It’s more of a model for FAI meeting an alien AI in space.
Makes sense.
BTW, what was your “UDT anti-Newcomb problem” intended to be a model for?
Frankly, I didn’t have a specific realistic scenario in mind. I came up with the anti-Newcomb problem as simple semi-artificial problem demonstrating the problems with quining in UDT. The reason I started thinking about these problems is that it doesn’t seem “classical” UDT can be translated to realistic AGI architecture. UDT takes a finite number of bits and produces a finite number of bits whereas a realistic AGI has continuous input and output streams. Such an AGI has to somehow take into account a formal specification of its own hardware, and the natural way of introducing such a specification seems to me to go through introducing a precursor, specifically a precursor which is a Solomonoff average over all “theories of physics” containing the formal specification.
Your problem can be written as
where B_1 and B_2 are Omega’s players,
Omega’s players end up playing C regardless of A. The agent can either understand this or at least fail to find a strong dependence of the logical probabilities of Omega’s players’ strategy on either their input (the agent’s source) or the conditions in the expectation values it is evaluating (since the conditions are of the form
Therefore, the highest expectation values will be computed for conditions of the form
I see. However, your problem doesn’t seem to be a realistic model of acausal bargaining with agents in other universes, since in such bargaining you know who you’re cooperating with. For example, when an agent considers filling its universe with human utility, it does it in order to cooperate with a human FAI, not in order to cooperate with a paperclip maximizer (which would require a very different strategy namely filling its universe with paperclips).
It’s more of a model for FAI meeting an alien AI in space. Suppose each side then has the choice of doing an arms buildup or not, and the payoffs for these choices are analogous to PD. (If one side builds up while the other doesn’t, it can attack and conquer the other. If both sides build up, it’s a stalemate and just wastes resources.) BTW, what was your “UDT anti-Newcomb problem” intended to be a model for?
I guess if both sides are using your decision theory, then whether the human FAI plays C or D against the alien AI depends on how much logical correlation the FAI thinks exists between its human designer and the alien AI’s designer, which does make sense (assuming we solve the problem of the FAI just simulating its designer and already knowing what their output is).
Makes sense.
Frankly, I didn’t have a specific realistic scenario in mind. I came up with the anti-Newcomb problem as simple semi-artificial problem demonstrating the problems with quining in UDT. The reason I started thinking about these problems is that it doesn’t seem “classical” UDT can be translated to realistic AGI architecture. UDT takes a finite number of bits and produces a finite number of bits whereas a realistic AGI has continuous input and output streams. Such an AGI has to somehow take into account a formal specification of its own hardware, and the natural way of introducing such a specification seems to me to go through introducing a precursor, specifically a precursor which is a Solomonoff average over all “theories of physics” containing the formal specification.