I feel confused because this seems to be a very obvious consideration, so it’s likely that I’m wrong: shouldn’t be the likelihood of 3^^^3 humans being killed decrease proportionally to the huge number that is being casually tossed out?
I mean, even if we go with the hypothesis of a matrix lord out for one of the cruelest prank in history, the likelihood of his simulation being able handle 3^^^3 humans being killed should be proportionally less than the likelihood of his simulation being able to handle 7625597484987 a mere humans being killed, since I think that simulating 3^^^3 humans would wreck any computational agent the laws of physics could allow… even if we consider simulation hypothesis, simulation hypothesis that require a quantum computer larger than the entire universe should be considered less likely and so on.
It feels to me that we believe an AI would give this possibility a likelihood that allows paying for any sum of money because we have no real way of estimating how unbelievably unlikely such an event is.
I guess I am missing something but I’m not sure what it could be.
Should we penalize computations with large space and time requirements? This is a hack that solves the problem, but is it true?
And he points out:
If the probabilities of various scenarios considered did not exactly cancel out, the AI’s action in the case of Pascal’s Mugging would be overwhelmingly dominated by whatever tiny differentials existed in the various tiny probabilities under which 3^^^^3 units of expected utility were actually at stake.
and:
Consider the plight of the first nuclear physicists, trying to calculate whether an atomic bomb could ignite the atmosphere. Yes, they had to do this calculation! Should they have not even bothered, because it would have killed so many people that the prior probability must be very low?The essential problem is that the universe doesn’t care one way or the other and therefore events do not in fact have probabilities that diminish with increasing disutility.
There is also a paper, which I found and lost and found again and lost again, which may just have been a blog post somewhere, to the effect that in a certain setting, all computable unbounded utility functions must necessarily be so dominated by small probabilities of large utilities that no expected utility calculation converges. If someone can remind me of what this paper was I’d appreciate it.
Thank you for your reply. I think I failed to explain some of my thinking, which was to assume that high number of people being killed has a much lower chance to be true when that number is higher than the current human being alive on earth (or other currently inhabited planets in the future).
Like, it wouldn’t make sense for atmospheric ignition to be twice as more likely if it only killed half the people in the world, but it makes sense that, if the number of deaths is so high that the only possible way for it to happen would be to be in a simulation with lots and lots of computing power, then that set of hypothesis is substantially penalised and sunk. And given that there are no constraints (save the theoretical limits of near omnipotent simulators) of how many humans could be killed in such a scenario, I’d expect that any number so unrealistically high to overcome the unlikeliness of “bored simulation overlord with no moral code” would get hit by more unlikeliness caused by the theoretical limits mentioned above.
BUT that was just my uninformed anticipation, and I think also me trying to explain why, as a human, I would decide to not obey Pascal Mugging but would instead take the chance of igniting atmosphere seriously enough.
I feel confused because this seems to be a very obvious consideration, so it’s likely that I’m wrong: shouldn’t be the likelihood of 3^^^3 humans being killed decrease proportionally to the huge number that is being casually tossed out?
I mean, even if we go with the hypothesis of a matrix lord out for one of the cruelest prank in history, the likelihood of his simulation being able handle 3^^^3 humans being killed should be proportionally less than the likelihood of his simulation being able to handle 7625597484987 a mere humans being killed, since I think that simulating 3^^^3 humans would wreck any computational agent the laws of physics could allow… even if we consider simulation hypothesis, simulation hypothesis that require a quantum computer larger than the entire universe should be considered less likely and so on.
It feels to me that we believe an AI would give this possibility a likelihood that allows paying for any sum of money because we have no real way of estimating how unbelievably unlikely such an event is.
I guess I am missing something but I’m not sure what it could be.
Eliezer covers this in the article:
And he points out:
and:
There is also a paper, which I found and lost and found again and lost again, which may just have been a blog post somewhere, to the effect that in a certain setting, all computable unbounded utility functions must necessarily be so dominated by small probabilities of large utilities that no expected utility calculation converges. If someone can remind me of what this paper was I’d appreciate it.
ETA: Found it again, again. “Convergence of expected utilities with algorithmic probability distributions”, by Peter de Blanc.
Thank you for your reply. I think I failed to explain some of my thinking, which was to assume that high number of people being killed has a much lower chance to be true when that number is higher than the current human being alive on earth (or other currently inhabited planets in the future).
Like, it wouldn’t make sense for atmospheric ignition to be twice as more likely if it only killed half the people in the world, but it makes sense that, if the number of deaths is so high that the only possible way for it to happen would be to be in a simulation with lots and lots of computing power, then that set of hypothesis is substantially penalised and sunk. And given that there are no constraints (save the theoretical limits of near omnipotent simulators) of how many humans could be killed in such a scenario, I’d expect that any number so unrealistically high to overcome the unlikeliness of “bored simulation overlord with no moral code” would get hit by more unlikeliness caused by the theoretical limits mentioned above.
BUT that was just my uninformed anticipation, and I think also me trying to explain why, as a human, I would decide to not obey Pascal Mugging but would instead take the chance of igniting atmosphere seriously enough.