How does this style of reasoning work on something more like the original Pascal’s Wager problem?
Suppose a (to all appearances) perfectly ordinary person goes on TV and says “I am an avatar of the Dark Lords of the Matrix. Please send me $5. When I shut down the simulation in a few months, I will subject those who send me the money to [LARGE NUMBER] years of happiness, and those who do not to [LARGE NUMBER] years of pain”.
Here you can’t solve the problem by pointing out the very large numbers of people involved, because there aren’t very high numbers of people involved. Your probability should depend only on your probability that this is a simulation, your probability that the simulators would make a weird request like this, and your probability that this person’s specific weird request is likely to be it. None of these numbers help you get down to a 1/[LARGE NUMBER] level.
I’ve avoided saying 3^^^3, because maybe there’s some fundamental constraint on computing power that makes it impossible for simulators to simulate 3^^^3 years of happiness in any amount of time they might conceivably be willing to dedicate to the problem. But they might be able to simulate some number of years large enough to outweigh our prior against any given weird request coming from the Dark Lords of the Matrix.
(also, it seems less than 3^^^3-level certain that there’s no clever trick to get effectively infinite computing power or effectively infinite computing time, like the substrateless computation in Permutation City)
When we jump to the version involving causal nodes having Large leverage over other nodes in a graph, there aren’t Large numbers of distinct people involved, but there’s Large numbers of life-centuries involved and those moments of thought and life have to be instantiated by causal nodes.
(also, it seems less than 3^^^3-level certain that there’s no clever trick to get effectively infinite computing power or effectively infinite computing time, like the substrateless computation in Permutation City)
Infinity makes my calculations break down and cry, at least at the moment.
The spaceship has really good life support/recycling
The spaceship is self-repairing and draws power from interstellar hydrogen
I’ve discovered the Universe will last at least another 3^^^3 years
Then they threaten, unless you give them $5, to kidnap you, give you the immortality drug, stick you in the spaceship, launch it at near-light speed, and have you stuck (presumably bound in an uncomfortable position) in the spaceship for the 3^^^3 years the universe will last.
(okay, there are lots of contingent features of the universe that will make this not work, but imagine something better. Pocket dimension, maybe?)
If their claims are true, then their threat seems credible even though it involves a large amount of suffering. Can you explain what you mean by life-centuries being instantiated by causal nodes, and how that makes the madman’s threat less credible?
Are you sure it wouldn’t be rational to pay up? I mean, if the guy looks like he could do that for $5, I’d rather not take chances. If you pay, and it turns out he didn’t have all that equipment for torture, you could just sue him and get that $5 back, since he defrauded you. If he starts making up rules about how you can never ever tell anyone else about this, or later check validity of his claim or he’ll kidnap you, you should, for game-theoretical reasons not abide, since being the kinda agent that accepts those terms makes you valid target for such frauds. Reasons for not abiding being the same as for single-boxing.
If what he says is true, then there will be 3^^^3 years of life in the universe. Then, assuming this anthropic framework is correct, it’s very unlikely to find yourself at the beginning rather than at any other point in time, so this provides 3^^^3-sized evidence against this scenario.
I’m not entirely sure that the doomsday argument also applies to different time slices of the same person, given that Eliezer in 2013 remembers being Eliezer in 2012 but not vice versa.
The spaceship has really good life support/recycling
The spaceship is self-repairing and draws power from interstellar hydrogen
That requires a MTTF of 3^^^3 years, or a per-year probability of failure of roughly 1/3^^^3.
I’ve discovered the Universe will last at least another 3^^^3 years
This implies that physical properties like the cosmological constant and the half-life of protons can be measured to a precision of roughly 1/3^^^3 relative error.
To me it seems like both of those claims have prior probability ~ 1/3^^^3. (How many spaceships would you have to build and how long would you have to test them to get an MTTF estimate as large as 3^^^3? How many measurements do you have to make to get the standard deviation below 1/3^^^3?)
Say the being that suffers for 3^^^3 seconds is morally relevant but not in the same observer moment reference class as humans for some reason. (IIRC putting all possible observers in the same reference class leads to bizarre conclusions...? I can’t immediately re-derive why that would be.) But anyway it really seems that the magical causal juice is the important thing here, not the anthropic/experiential nature or lack thereof of the highly-causal nodes, in which case the anthropic solution isn’t quite hugging the real query.
IIRC putting all possible observers in the same reference class leads to bizarre conclusions...? I can’t immediately re-derive why that would be.
The only reason that I have ever thought of is that our reference class should intuitively consist of only sentient beings, but that nonsentient beings should still be able to reason. Is this what you were thinking of? Whether it applies in a given context may depend on what exactly you mean by a reference class in that context.
If it can reason but isn’t sentient then it maybe doesn’t have “observer” moments, and maybe isn’t itself morally relevant—Eliezer seems to think that way anyway. I’ve been trying something like, maybe messing with the non-sentient observer has a 3^^^3 utilon effect on human utility somehow, but that seems psychologically-architecturally impossible for humans in a way that might end up being fundamental. (Like, you either have to make 3^^^3 humans, which defeats the purpose of the argument, or make a single human have a 3^^^3 times better life without lengthening it, which seems impossible.) Overall I’m having a really surprising amount of difficulty thinking up an example where you have a lot of causal importance but no anthropic counter-evidence.
Anyway, does “anthropic” even really have anything to do with qualia? The way people talk about it it clearly does, but I’m not sure it even shows up in the definition—a non-sentient optimizer could totally make anthropic updates. (That said I guess Hofstadter and other strange loop functionalists would disagree.) Have I just been wrongly assuming that everyone else was including “qualia” as fundamental to anthropics?
Yeah, this whole line of reasoning fails if you can get to 3^^^3 utilons without creating ~3^^^3 sentients to distribute them among.
Overall I’m having a really surprising amount of difficulty thinking up an example where you have a lot of causal importance but no anthropic counter-evidence.
I’m not sure what you mean. If you use an anthropic theory like what Eliezer is using here (e.g. SSA, UDASSA) then an amount of causal importance that is large compared to the rest of your reference class implies few similar members of the reference class, which is anthropic counter-evidence, so of course it would be impossible to think of an example. Even if nonsentients can contribute to utility, if I can create 3^^^3 utilons using nonsentients, than some other people probably can to, so I don’t have a lot of causal importance compared to them.
Anyway, does “anthropic” even really have anything to do with qualia? The way people talk about it it clearly does, but I’m not sure it even shows up in the definition—a non-sentient optimizer could totally make anthropic updates.
This is the contrapositive of the grandparent. I was saying that if we assume that the reference class is sentients, then nonsentients need to reason using different rules i.e. a different reference class. You are saying that if nonsentients should reason using the same rules, then the reference class cannot comprise only sentients. I actually agree with the latter much more strongly, and I only brought up the former because it seemed similar to the argument you were trying to remember.
There are really two separate questions here, that of how to reason anthropically and that of how magic reality-fluid is distributed. Confusing these is common, since the same sort of considerations affect both of them and since they are both badly understood, though I would say that due to UDT/ADT, we now understand the former much better, while acknowledging the possibility of unknown unknowns. (Our current state of knowledge where we confuse these actually feels a lot like people who have never learnt to separate the descriptive and the normative.)
The way Eliezer presented things in the post, it is not entirely clear which of the two he meant to be responsible for the leverage penalty. It seems like he meant for it to be an epistemic consideration due to anthropic reasoning, but this seems obviously wrong given UDT. In the Tegmark IV model that he describes, the leverage penalty is caused by reality-fluid, but it seems like he only intended that as an analogy. It seems a lot more probable to me though, and it is possible that Eliezer would express uncertainty as to whether the leverage penalty is actually caused by reality-fluid, so that it is a bit more than an analogy. There is also a third mathematically equivalent possibility where the leverage penalty is about values, and we just care less about individual people when there are more of them, but Eliezer obviously does not hold that view.
I’m not sure what you mean. If you use an anthropic theory like what Eliezer is using here (e.g. SSA, UDASSA)
A comment: it is not clear to me that Eliezer is intending to use SSA or UDASSA here. The “magic reality fluid” measure looks more like SIA, but with a prior based on Levin complexity rather than Kolmogorov complexity—see my comment here. Or—in an equivalent formulation—he’s using Kolmogorov + SSA but with an extremely broad “reference class” (the class of all causal nodes, most of which aren’t observers in any anthropic sense). This is still not UDASSA.
To get something like UDASSA, we shouldn’t distribute the weight 2^-#p of each program p uniformly among its execution steps. Instead we should consider using another program q to pick out an execution step or a sequence of steps (i.e. a sub-program s) from p, and then give the combination of q,p a weight 2^-(#p+#q). This means each sub-program s will get a total prior weight of Sum {p, q: q(p) = s & s is a sub-program of p} 2^-(#p + #q).
When updating on your evidence E, consider the class S(E) of all sub-programs which correspond to an AI program having that evidence, and normalize. The posterior probability you are in a particular universe p’ then becomes proportional to Sum {q: q(p’) is a sub-program of p’ and a member of S(E)} 2^-(#p’ + #q).
This looks rather different to what I discussed in my other comment, and it maybe handles anthropic problems a bit better. I can’t see there is any shift either towards very big universes (no presumptuous philosopher) or towards dense computronium universes, where we are simulations. There does appear to be a Great Filter or “Doomsday” shift, since it is still a form of SSA, but this is mitigated by the consideration that we may be part of a reference class (program q) which preferentially selects pre-AI biological observers, as opposed to any old observers.
I agree with this; the ‘e.g.’ was meant to point toward the most similar theories that have names, not pin down exactly what Eliezer is doing here. I though that it would be better to refer to the class of similar theories here since there is enough uncertainty that we don’t really have details.
How does this style of reasoning work on something more like the original Pascal’s Wager problem?
Suppose a (to all appearances) perfectly ordinary person goes on TV and says “I am an avatar of the Dark Lords of the Matrix. Please send me $5. When I shut down the simulation in a few months, I will subject those who send me the money to [LARGE NUMBER] years of happiness, and those who do not to [LARGE NUMBER] years of pain”.
Here you can’t solve the problem by pointing out the very large numbers of people involved, because there aren’t very high numbers of people involved. Your probability should depend only on your probability that this is a simulation, your probability that the simulators would make a weird request like this, and your probability that this person’s specific weird request is likely to be it. None of these numbers help you get down to a 1/[LARGE NUMBER] level.
I’ve avoided saying 3^^^3, because maybe there’s some fundamental constraint on computing power that makes it impossible for simulators to simulate 3^^^3 years of happiness in any amount of time they might conceivably be willing to dedicate to the problem. But they might be able to simulate some number of years large enough to outweigh our prior against any given weird request coming from the Dark Lords of the Matrix.
(also, it seems less than 3^^^3-level certain that there’s no clever trick to get effectively infinite computing power or effectively infinite computing time, like the substrateless computation in Permutation City)
When we jump to the version involving causal nodes having Large leverage over other nodes in a graph, there aren’t Large numbers of distinct people involved, but there’s Large numbers of life-centuries involved and those moments of thought and life have to be instantiated by causal nodes.
Infinity makes my calculations break down and cry, at least at the moment.
Imagine someone makes the following claims:
I’ve invented an immortality drug
I’ve invented a near-light-speed spaceship
The spaceship has really good life support/recycling
The spaceship is self-repairing and draws power from interstellar hydrogen
I’ve discovered the Universe will last at least another 3^^^3 years
Then they threaten, unless you give them $5, to kidnap you, give you the immortality drug, stick you in the spaceship, launch it at near-light speed, and have you stuck (presumably bound in an uncomfortable position) in the spaceship for the 3^^^3 years the universe will last.
(okay, there are lots of contingent features of the universe that will make this not work, but imagine something better. Pocket dimension, maybe?)
If their claims are true, then their threat seems credible even though it involves a large amount of suffering. Can you explain what you mean by life-centuries being instantiated by causal nodes, and how that makes the madman’s threat less credible?
Are you sure it wouldn’t be rational to pay up? I mean, if the guy looks like he could do that for $5, I’d rather not take chances. If you pay, and it turns out he didn’t have all that equipment for torture, you could just sue him and get that $5 back, since he defrauded you. If he starts making up rules about how you can never ever tell anyone else about this, or later check validity of his claim or he’ll kidnap you, you should, for game-theoretical reasons not abide, since being the kinda agent that accepts those terms makes you valid target for such frauds. Reasons for not abiding being the same as for single-boxing.
If what he says is true, then there will be 3^^^3 years of life in the universe. Then, assuming this anthropic framework is correct, it’s very unlikely to find yourself at the beginning rather than at any other point in time, so this provides 3^^^3-sized evidence against this scenario.
I’m not entirely sure that the doomsday argument also applies to different time slices of the same person, given that Eliezer in 2013 remembers being Eliezer in 2012 but not vice versa.
That requires a MTTF of 3^^^3 years, or a per-year probability of failure of roughly 1/3^^^3.
This implies that physical properties like the cosmological constant and the half-life of protons can be measured to a precision of roughly 1/3^^^3 relative error.
To me it seems like both of those claims have prior probability ~ 1/3^^^3. (How many spaceships would you have to build and how long would you have to test them to get an MTTF estimate as large as 3^^^3? How many measurements do you have to make to get the standard deviation below 1/3^^^3?)
Say the being that suffers for 3^^^3 seconds is morally relevant but not in the same observer moment reference class as humans for some reason. (IIRC putting all possible observers in the same reference class leads to bizarre conclusions...? I can’t immediately re-derive why that would be.) But anyway it really seems that the magical causal juice is the important thing here, not the anthropic/experiential nature or lack thereof of the highly-causal nodes, in which case the anthropic solution isn’t quite hugging the real query.
The only reason that I have ever thought of is that our reference class should intuitively consist of only sentient beings, but that nonsentient beings should still be able to reason. Is this what you were thinking of? Whether it applies in a given context may depend on what exactly you mean by a reference class in that context.
If it can reason but isn’t sentient then it maybe doesn’t have “observer” moments, and maybe isn’t itself morally relevant—Eliezer seems to think that way anyway. I’ve been trying something like, maybe messing with the non-sentient observer has a 3^^^3 utilon effect on human utility somehow, but that seems psychologically-architecturally impossible for humans in a way that might end up being fundamental. (Like, you either have to make 3^^^3 humans, which defeats the purpose of the argument, or make a single human have a 3^^^3 times better life without lengthening it, which seems impossible.) Overall I’m having a really surprising amount of difficulty thinking up an example where you have a lot of causal importance but no anthropic counter-evidence.
Anyway, does “anthropic” even really have anything to do with qualia? The way people talk about it it clearly does, but I’m not sure it even shows up in the definition—a non-sentient optimizer could totally make anthropic updates. (That said I guess Hofstadter and other strange loop functionalists would disagree.) Have I just been wrongly assuming that everyone else was including “qualia” as fundamental to anthropics?
Yeah, this whole line of reasoning fails if you can get to 3^^^3 utilons without creating ~3^^^3 sentients to distribute them among.
I’m not sure what you mean. If you use an anthropic theory like what Eliezer is using here (e.g. SSA, UDASSA) then an amount of causal importance that is large compared to the rest of your reference class implies few similar members of the reference class, which is anthropic counter-evidence, so of course it would be impossible to think of an example. Even if nonsentients can contribute to utility, if I can create 3^^^3 utilons using nonsentients, than some other people probably can to, so I don’t have a lot of causal importance compared to them.
This is the contrapositive of the grandparent. I was saying that if we assume that the reference class is sentients, then nonsentients need to reason using different rules i.e. a different reference class. You are saying that if nonsentients should reason using the same rules, then the reference class cannot comprise only sentients. I actually agree with the latter much more strongly, and I only brought up the former because it seemed similar to the argument you were trying to remember.
There are really two separate questions here, that of how to reason anthropically and that of how magic reality-fluid is distributed. Confusing these is common, since the same sort of considerations affect both of them and since they are both badly understood, though I would say that due to UDT/ADT, we now understand the former much better, while acknowledging the possibility of unknown unknowns. (Our current state of knowledge where we confuse these actually feels a lot like people who have never learnt to separate the descriptive and the normative.)
The way Eliezer presented things in the post, it is not entirely clear which of the two he meant to be responsible for the leverage penalty. It seems like he meant for it to be an epistemic consideration due to anthropic reasoning, but this seems obviously wrong given UDT. In the Tegmark IV model that he describes, the leverage penalty is caused by reality-fluid, but it seems like he only intended that as an analogy. It seems a lot more probable to me though, and it is possible that Eliezer would express uncertainty as to whether the leverage penalty is actually caused by reality-fluid, so that it is a bit more than an analogy. There is also a third mathematically equivalent possibility where the leverage penalty is about values, and we just care less about individual people when there are more of them, but Eliezer obviously does not hold that view.
A comment: it is not clear to me that Eliezer is intending to use SSA or UDASSA here. The “magic reality fluid” measure looks more like SIA, but with a prior based on Levin complexity rather than Kolmogorov complexity—see my comment here. Or—in an equivalent formulation—he’s using Kolmogorov + SSA but with an extremely broad “reference class” (the class of all causal nodes, most of which aren’t observers in any anthropic sense). This is still not UDASSA.
To get something like UDASSA, we shouldn’t distribute the weight 2^-#p of each program p uniformly among its execution steps. Instead we should consider using another program q to pick out an execution step or a sequence of steps (i.e. a sub-program s) from p, and then give the combination of q,p a weight 2^-(#p+#q). This means each sub-program s will get a total prior weight of Sum {p, q: q(p) = s & s is a sub-program of p} 2^-(#p + #q).
When updating on your evidence E, consider the class S(E) of all sub-programs which correspond to an AI program having that evidence, and normalize. The posterior probability you are in a particular universe p’ then becomes proportional to Sum {q: q(p’) is a sub-program of p’ and a member of S(E)} 2^-(#p’ + #q).
This looks rather different to what I discussed in my other comment, and it maybe handles anthropic problems a bit better. I can’t see there is any shift either towards very big universes (no presumptuous philosopher) or towards dense computronium universes, where we are simulations. There does appear to be a Great Filter or “Doomsday” shift, since it is still a form of SSA, but this is mitigated by the consideration that we may be part of a reference class (program q) which preferentially selects pre-AI biological observers, as opposed to any old observers.
I agree with this; the ‘e.g.’ was meant to point toward the most similar theories that have names, not pin down exactly what Eliezer is doing here. I though that it would be better to refer to the class of similar theories here since there is enough uncertainty that we don’t really have details.