I’ve been watching for a while, but have never commented, so this may be horribly flawed, opaque or otherwise unhelpful.
I think the problem is entirely caused by the use of the wrong sets of belief, and that anything holding to Eliezer’s 1-line summary of TDT or alternatively UDT should get this right.
Suppose that you’re a rational agent. Since you are instantiated in multiple identical circumstances (green rooms) and asked identical questions, your answers should be identical. Hence if you wake up in a green room and you’re asked to steal from the red rooms and give to the green rooms, you either commit a group of 2 of you to a loss of 52 or commit a group of 18 of you to a gain of 12.
This committal is what you wish to optimise over from TDT/UDT, and clearly this requires knowledge about the likelyhood of different decision making groups. The distribution of sizes of random groups is not the same as the distribution of sizes of groups that a random individual is in. The probabilities of being in a group are upweighted by the size of the group and normalised. This is why Bostrom’s suggested 1/n split of responsibility works; it reverses the belief about where a random individual is in a set of decision making groups to a belief about the size of a random decision making group.
By the construction of the problem the probability that a random (group of all the people in green rooms) has size 18 is 0.5, and similarly for 2 the probability is 0.5. Hence the expected utility is (0.512)+(0.5-52)=-20.
If you’re asked to accept a bet on there being 18 people in green rooms, and you’re told that only you’re being offered it, then the decision commits exactly one instance of you to a specific loss or gain, regardless of the group you’re in. Hence you can’t do better than the 0.9 and 0.1 beliefs.
If you’re told that the bet is being offered to everyone in a green room, then you are committing to n times the outcome in any group of n people. In this case gains are conditional on group size, and so you have to use the 0.5-0.5 belief about the distribution of groups. It doesn’t matter because the larger groups have the larger multiplier and thus shutting up and multiplying yields the same answers as a single-shot bet.
ETA: At some level this is just choosing an optimal output for your calculation of what to do, given that the result is used variably widely.
This committal is what you wish to optimise over from TDT/UDT, and clearly this requires knowledge about the likelyhood of different decision making groups.
I was influenced by the OP and used to think that way. However I think now, that this is not the root problem.
What if the agents get more complicated decision problems: for example, rewards depending on the parity of the agents voting certain way, etc.?
I think, what essential is that the agents have to think globally (categorical imperative, hmmm?)
Practically: if the agent recognizes that there is a collective decision, then it should model all available conceivable protocols (but making apriori sure that all cooperating agents perform the same or compatible analysis, if they can’t communicate) and then they should choose the protocol with best overall total gain. In the case of the OP: the second calculation in the OP. (Not messing around with correction factors based on responsibilities, etc.)
Special considerations based on group sizes etc. may be incidentally correct in certain situations, but this is just not general enough. The crux is that the ultimate test is simply the expected value computation for the protocol of the whole group.
Between non communicating copies of your decision algorithm, it’s forced that every instance comes to the same answers/distributions to all questions, as otherwise Eliezer can make money betting between different instances of the algorithm. It’s not really a categorical imperative, beyond demanding consistency.
The crux of the OP is asking for a probability assessment of the world, not whether the DT functions.
I’m not postulating 1/n allocation of responsibility; I’m stating that the source of the confusion is over:
P(A random individual is in a world of class A_i | Data) with
P(A random world is of class A_i | Data)
And that these are not equal if the number of individuals with access to Data are different in distinct classes of world.
Hence in this case, there are 2 classes of world, A_1 with 18 Green rooms and 2 Reds, and A_2 with 2 Green rooms and 18 Reds.
P(Random individual is in the A_1 class | Woke up in a green room) = 0.9 by anthropic update.
P(Random world is in the A_1 class | Some individual woke up in a green room) = 0.5
Why? Because in A_1 there 18⁄20 individuals fit the description “Woke up in a green room”, but in A_2 only 2⁄20 do.
The crux of the OP is that neither a 90⁄10 nor 50⁄50 split seem acceptable, if betting on
“Which world-class an individual in a Green room is in” and
“Which world-class the (set of all individuals in Green rooms which contains this individual) is in”
are identical. I assert that they are not. The first case is 0.9/0.1 A_1/A_2, the second is 0.5/0.5 A_1/A_2.
Consider a similar question where a random Green room will be asked. If you’re in that room, you update both on (Green walls) and (I’m being asked) and recover the 0.5/0.5, correctly. This is close to the OP as if we wildly assert that you and only you have free will and force the others, then you are special. Equally in cases where everyone is asked and plays separately, you have 18 or 2 times the benefits depending on whether you’re in A_1 or A_2.
If each individual Green room played separately, then you update on (Green walls), but P(I’m being asked|Green) = 1 in either case. This is betting on whether there are 18 people in green rooms or 2, and you get the correct 0.9/0.1 split. To reproduce the OP the offers would need to be +1/18 to Greens and −3/18 from Reds in A_1, and +1/2 to Greens and −3/2 from Reds in A_2, and then you’d refuse to play, correctly.
And how would they decide which protocol had the best overall total gain? For instance, could you define a protocol complexity measure, and then use this complexity measure to decide? And are you even dealing with ordinary Bayesian reasoning any more, or is this the first hint of some new more general type of rationality?
It’s not about complexity, it is just expected total gain. Simply the second calculation of the OP.
I just argued, that the second calculation is right and that is what the agents should do in general. (unless they are completely egoistic for their special copies)
This was a simple situation. I’m suggesting a ‘big picture’ idea for the general case.
According to Wei Dei and Nesov above, the anthropic-like puzzles can be re-interpreted as ‘agent co-ordination’ problems (multiple agents trying to coordinate their decision making). And you seemed to have a similiar interpretation. Am I right?
If Dei and Nesov’s interpretation is right, it seems the puzzles could be reinterpreted as being about groups of agents tring to agree in advance about a ‘decision making protocol’.
But now I ask is this not equivalent to trying to find a ‘communication protocol’ which enables them to best coordinate their decision making? And rather than trying to directly calculate the results of every possible protocol (which would be impractical for all but simple problems), I was suggesting trying to use information theory to apply a complexity measure to protocols, in order to rank them.
Indeed I ask whether this is actually the correct way to interpret Occam’s Razor/Complexity Priors? i.e, My suggestion is to re-interpret Occam/Priors as referring to copies of agents trying to co-ordinate their decision making using some communication protocol, such that they seek to minimize the complexity of this protocol.
“Hence if you wake up in a green room and you’re asked to steal from the red rooms and give to the green rooms, you either commit a group of 2 of you to a loss of 52 or commit a group of 18 of you to a gain of 12.”
In the example you care equally about the red room and green room dwellers.
Hence if there are 2 instances of your decision algorithm in Green rooms, there are 2 runs of your decision algorithm, and if they vote to steal there is a loss of 3 from each red and gain 1 for each green, for a total gain of 12-318 = − 52.
If there are 18 instances in Green rooms, there are 18 runs of your decision algorithm, and if they vote to steal there is a loss of 3 from each red and a gain of 1for each green, for a total gain of 118-23 = 12
The “committal of a group of” is noting that there are 2 or 18 runs of your decision algorithm that are logically forced by the decision made this specific instance of the decision algorithm in a green room.
I’ve been watching for a while, but have never commented, so this may be horribly flawed, opaque or otherwise unhelpful.
I think the problem is entirely caused by the use of the wrong sets of belief, and that anything holding to Eliezer’s 1-line summary of TDT or alternatively UDT should get this right.
Suppose that you’re a rational agent. Since you are instantiated in multiple identical circumstances (green rooms) and asked identical questions, your answers should be identical. Hence if you wake up in a green room and you’re asked to steal from the red rooms and give to the green rooms, you either commit a group of 2 of you to a loss of 52 or commit a group of 18 of you to a gain of 12.
This committal is what you wish to optimise over from TDT/UDT, and clearly this requires knowledge about the likelyhood of different decision making groups. The distribution of sizes of random groups is not the same as the distribution of sizes of groups that a random individual is in. The probabilities of being in a group are upweighted by the size of the group and normalised. This is why Bostrom’s suggested 1/n split of responsibility works; it reverses the belief about where a random individual is in a set of decision making groups to a belief about the size of a random decision making group.
By the construction of the problem the probability that a random (group of all the people in green rooms) has size 18 is 0.5, and similarly for 2 the probability is 0.5. Hence the expected utility is (0.512)+(0.5-52)=-20.
If you’re asked to accept a bet on there being 18 people in green rooms, and you’re told that only you’re being offered it, then the decision commits exactly one instance of you to a specific loss or gain, regardless of the group you’re in. Hence you can’t do better than the 0.9 and 0.1 beliefs.
If you’re told that the bet is being offered to everyone in a green room, then you are committing to n times the outcome in any group of n people. In this case gains are conditional on group size, and so you have to use the 0.5-0.5 belief about the distribution of groups. It doesn’t matter because the larger groups have the larger multiplier and thus shutting up and multiplying yields the same answers as a single-shot bet.
ETA: At some level this is just choosing an optimal output for your calculation of what to do, given that the result is used variably widely.
I was influenced by the OP and used to think that way. However I think now, that this is not the root problem.
What if the agents get more complicated decision problems: for example, rewards depending on the parity of the agents voting certain way, etc.?
I think, what essential is that the agents have to think globally (categorical imperative, hmmm?)
Practically: if the agent recognizes that there is a collective decision, then it should model all available conceivable protocols (but making apriori sure that all cooperating agents perform the same or compatible analysis, if they can’t communicate) and then they should choose the protocol with best overall total gain. In the case of the OP: the second calculation in the OP. (Not messing around with correction factors based on responsibilities, etc.)
Special considerations based on group sizes etc. may be incidentally correct in certain situations, but this is just not general enough. The crux is that the ultimate test is simply the expected value computation for the protocol of the whole group.
Between non communicating copies of your decision algorithm, it’s forced that every instance comes to the same answers/distributions to all questions, as otherwise Eliezer can make money betting between different instances of the algorithm. It’s not really a categorical imperative, beyond demanding consistency.
The crux of the OP is asking for a probability assessment of the world, not whether the DT functions.
I’m not postulating 1/n allocation of responsibility; I’m stating that the source of the confusion is over: P(A random individual is in a world of class A_i | Data) with P(A random world is of class A_i | Data) And that these are not equal if the number of individuals with access to Data are different in distinct classes of world.
Hence in this case, there are 2 classes of world, A_1 with 18 Green rooms and 2 Reds, and A_2 with 2 Green rooms and 18 Reds.
P(Random individual is in the A_1 class | Woke up in a green room) = 0.9 by anthropic update. P(Random world is in the A_1 class | Some individual woke up in a green room) = 0.5
Why? Because in A_1 there 18⁄20 individuals fit the description “Woke up in a green room”, but in A_2 only 2⁄20 do.
The crux of the OP is that neither a 90⁄10 nor 50⁄50 split seem acceptable, if betting on “Which world-class an individual in a Green room is in” and “Which world-class the (set of all individuals in Green rooms which contains this individual) is in” are identical. I assert that they are not. The first case is 0.9/0.1 A_1/A_2, the second is 0.5/0.5 A_1/A_2.
Consider a similar question where a random Green room will be asked. If you’re in that room, you update both on (Green walls) and (I’m being asked) and recover the 0.5/0.5, correctly. This is close to the OP as if we wildly assert that you and only you have free will and force the others, then you are special. Equally in cases where everyone is asked and plays separately, you have 18 or 2 times the benefits depending on whether you’re in A_1 or A_2.
If each individual Green room played separately, then you update on (Green walls), but P(I’m being asked|Green) = 1 in either case. This is betting on whether there are 18 people in green rooms or 2, and you get the correct 0.9/0.1 split. To reproduce the OP the offers would need to be +1/18 to Greens and −3/18 from Reds in A_1, and +1/2 to Greens and −3/2 from Reds in A_2, and then you’d refuse to play, correctly.
And how would they decide which protocol had the best overall total gain? For instance, could you define a protocol complexity measure, and then use this complexity measure to decide? And are you even dealing with ordinary Bayesian reasoning any more, or is this the first hint of some new more general type of rationality?
MJG—The Black Swan is Near!
It’s not about complexity, it is just expected total gain. Simply the second calculation of the OP.
I just argued, that the second calculation is right and that is what the agents should do in general. (unless they are completely egoistic for their special copies)
This was a simple situation. I’m suggesting a ‘big picture’ idea for the general case.
According to Wei Dei and Nesov above, the anthropic-like puzzles can be re-interpreted as ‘agent co-ordination’ problems (multiple agents trying to coordinate their decision making). And you seemed to have a similiar interpretation. Am I right?
If Dei and Nesov’s interpretation is right, it seems the puzzles could be reinterpreted as being about groups of agents tring to agree in advance about a ‘decision making protocol’.
But now I ask is this not equivalent to trying to find a ‘communication protocol’ which enables them to best coordinate their decision making? And rather than trying to directly calculate the results of every possible protocol (which would be impractical for all but simple problems), I was suggesting trying to use information theory to apply a complexity measure to protocols, in order to rank them.
Indeed I ask whether this is actually the correct way to interpret Occam’s Razor/Complexity Priors? i.e, My suggestion is to re-interpret Occam/Priors as referring to copies of agents trying to co-ordinate their decision making using some communication protocol, such that they seek to minimize the complexity of this protocol.
“Hence if you wake up in a green room and you’re asked to steal from the red rooms and give to the green rooms, you either commit a group of 2 of you to a loss of 52 or commit a group of 18 of you to a gain of 12.”
In the example you care equally about the red room and green room dwellers.
Hence if there are 2 instances of your decision algorithm in Green rooms, there are 2 runs of your decision algorithm, and if they vote to steal there is a loss of 3 from each red and gain 1 for each green, for a total gain of 12-318 = − 52.
If there are 18 instances in Green rooms, there are 18 runs of your decision algorithm, and if they vote to steal there is a loss of 3 from each red and a gain of 1for each green, for a total gain of 118-23 = 12
The “committal of a group of” is noting that there are 2 or 18 runs of your decision algorithm that are logically forced by the decision made this specific instance of the decision algorithm in a green room.