Should we penalize computations with large space and time requirements? This is a hack that solves the problem, but is it true?
And he points out:
If the probabilities of various scenarios considered did not exactly cancel out, the AI’s action in the case of Pascal’s Mugging would be overwhelmingly dominated by whatever tiny differentials existed in the various tiny probabilities under which 3^^^^3 units of expected utility were actually at stake.
and:
Consider the plight of the first nuclear physicists, trying to calculate whether an atomic bomb could ignite the atmosphere. Yes, they had to do this calculation! Should they have not even bothered, because it would have killed so many people that the prior probability must be very low?The essential problem is that the universe doesn’t care one way or the other and therefore events do not in fact have probabilities that diminish with increasing disutility.
There is also a paper, which I found and lost and found again and lost again, which may just have been a blog post somewhere, to the effect that in a certain setting, all computable unbounded utility functions must necessarily be so dominated by small probabilities of large utilities that no expected utility calculation converges. If someone can remind me of what this paper was I’d appreciate it.
Thank you for your reply. I think I failed to explain some of my thinking, which was to assume that high number of people being killed has a much lower chance to be true when that number is higher than the current human being alive on earth (or other currently inhabited planets in the future).
Like, it wouldn’t make sense for atmospheric ignition to be twice as more likely if it only killed half the people in the world, but it makes sense that, if the number of deaths is so high that the only possible way for it to happen would be to be in a simulation with lots and lots of computing power, then that set of hypothesis is substantially penalised and sunk. And given that there are no constraints (save the theoretical limits of near omnipotent simulators) of how many humans could be killed in such a scenario, I’d expect that any number so unrealistically high to overcome the unlikeliness of “bored simulation overlord with no moral code” would get hit by more unlikeliness caused by the theoretical limits mentioned above.
BUT that was just my uninformed anticipation, and I think also me trying to explain why, as a human, I would decide to not obey Pascal Mugging but would instead take the chance of igniting atmosphere seriously enough.
Eliezer covers this in the article:
And he points out:
and:
There is also a paper, which I found and lost and found again and lost again, which may just have been a blog post somewhere, to the effect that in a certain setting, all computable unbounded utility functions must necessarily be so dominated by small probabilities of large utilities that no expected utility calculation converges. If someone can remind me of what this paper was I’d appreciate it.
ETA: Found it again, again. “Convergence of expected utilities with algorithmic probability distributions”, by Peter de Blanc.
Thank you for your reply. I think I failed to explain some of my thinking, which was to assume that high number of people being killed has a much lower chance to be true when that number is higher than the current human being alive on earth (or other currently inhabited planets in the future).
Like, it wouldn’t make sense for atmospheric ignition to be twice as more likely if it only killed half the people in the world, but it makes sense that, if the number of deaths is so high that the only possible way for it to happen would be to be in a simulation with lots and lots of computing power, then that set of hypothesis is substantially penalised and sunk. And given that there are no constraints (save the theoretical limits of near omnipotent simulators) of how many humans could be killed in such a scenario, I’d expect that any number so unrealistically high to overcome the unlikeliness of “bored simulation overlord with no moral code” would get hit by more unlikeliness caused by the theoretical limits mentioned above.
BUT that was just my uninformed anticipation, and I think also me trying to explain why, as a human, I would decide to not obey Pascal Mugging but would instead take the chance of igniting atmosphere seriously enough.