But even agents based on Solomonoff Induction, such as AIXI, are not subject to Pascal’s Mugging in reasonable environments. Consider this paper by Hutter. IIUC, the “self-optimizing” property essentially implies robustness against Pascal’s Mugging.
Ah! I’ve got it! I think… At least on the probability side.
The problem is our intuition: the utility of human lives grows linearly with the population of humans, while the message size of the hypothesis needed to describe them grows roughly logarithmically. Since we think we climb down the exponential decline in prior probability slower than utility increases, Pascal’s Bargain sounds favorable.
This is wrong. A real Solomonoff Hypothesis, after all, does not merely say “There are 3^^^3 humans” if there really are 3 ^^^ 3 humans. It describes and predicts each single human, after all, in detail. It’s a hypothesis that aims to predict an entire universe from one small piece of fairy cake.
And when you have to make predictive descriptions of humans, the summed size of those descriptions will grow at least linearly in the number of purported humans. God help you if they start organizing themselves into complicated societies, which are more complex than a mere summed set of single persons. Now your utility from taking the Bargain grows linearly while your negative exponent on the probability of its being real declines linearly.
The question then becomes a simple matter of where your plausability versus utility tradeoff sits for one human life.
Or in other words, if Pascal’s Mugger catches you in a back ally, you should demand that he start describing the people he’s threatening one-by-one.
Or in other words, if Pascal’s Mugger catches you in a back ally, you should demand that he start describing the people he’s threatening one-by-one.
By this reasoning, if someone has their finger on a trigger for a nuclear bomb that will destroy a city of a million people, and says “give me $1 or I will pick a random number and destroy the city at a 1/1000000 chance”, you should refuse. After all, he cannot describe any of the people he is planning to kill, so this is equivalent to killing one person at a 1/1000000 chance, and you could probably exceed that chance of killing someone just by driving a couple of miles to the supermarket.
If it’s a million people possibly dying at a one-in-a-million chance, then in expected-death terms he’s charging me $1 not to kill one person. Since I believe human lives are worth more than $1, I should give him the dollar and make a “profit”.
Of course, the other issue here, and the reason we don’t make analogies between serious military threats and Pascal’s Mugging, is that in the nuclear bomb case, there is actual evidence on which to update my beliefs. For instance, the button he’s got his finger on is either real, or not. If I can see damn well that it’s a plastic toy from the local store, I’ve no reason to give him a dollar.
So in the case of Soviet Russia threatening you, you’ve got real evidence that they might deliberately nuke your cities with a probability much higher than one in a million. In the case of Pascal’s Mugger, you’ve got a 1⁄1,000,000 chance that the mugger is telling the truth at all, and all the other probability mass points at the mugger being a delusion of your badly-coded reasoning algorithms.
If it’s a million people possibly dying at a one-in-a-million chance, and I use the reasoning you used before, because the mugger can’t describe the people he’s threatening to kill, I shouldn’t treat that as any worse than a threat to kill one person at a one-in-a-million chance.
You misconstrue my position. I’m not saying, “Descriptions are magic!”. I’m saying: I prefer evidentialism to pure Bayesianism. Meaning: if the mugger can’t describe anything about the city under threat, that is evidence that he is lying.
Which misses the point of the scenario, since a real Pascal’s Mugging is not about a physical mugger who could ever be lying. It’s about having a flaw in your own reasoning system.
Blurgh. That’s one of the most symbol-dense papers I’ve ever seen.
But from what I can tell, it specifies that AIXI will eventually reason its way out of any finite Pascal’s Mugging. The higher the hypothesized reward in the Mugging, the longer it will take to converge away from the Mugging, but each failure of reality to conform to the Mugging will push down the probability of that environment being true and thus reduce the expected value of acting according to it. Asymptotic convergence is proven.
I’d also bet that Hutter’s formalism might consider large rewards generated for little reason to be not merely complex because of “little reason”, but actually to have greater Kolmogorov Complexity just because large rewards are more complex than small ones. Possibly. So there would be a question of whether the reward of a Mugging grows faster than its probability shrinks, in the limit. Eliezer claims it does, but the question is whether our convenient notations for very, very, very large numbers actually imply some kind of simplicity for those numbers or whether we’re hiding complexity in our brains at that point.
It seems downright obvious that “3” ought be considered vastly more simple than “3 ^^^ 3″. How large a Turing Machine does it take to write down the fullest expansion of the recursive function for that super-exponentiation? Do we have to expand it out? I would think a computational theory of induction ought make a distinction between computations outputting large numbers and actual large numbers, after all.
Blurgh. That’s one of the most symbol-dense papers I’ve ever seen.
I know. It seems that this is Hutter’s usual style.
But from what I can tell, it specifies that AIXI will eventually reason its way out of any finite Pascal’s Mugging. The higher the hypothesized reward in the Mugging, the longer it will take to converge away from the Mugging, but each failure of reality to conform to the Mugging will push down the probability of that environment being true and thus reduce the expected value of acting according to it. Asymptotic convergence is proven.
Well, the Pascal’s Mugging issue essentially boils down to whether the agent decision making is dominated by the bias in its prior. Clearly, an agent that has seen little or no sensory input can’t have possibly learned anything, and is therefore dominated by its bias. What Hutter proved is that, for reasonable classes of environments, the agent eventually overcomes its bias. There is of course the interesting question of convergence speed, which is not addressed in that paper.
I’d also bet that Hutter’s formalism might consider large rewards generated for little reason to be not merely complex because of “little reason”, but actually to have greater Kolmogorov Complexity just because large rewards are more complex than small ones. Possibly. So there would be a question of whether the reward of a Mugging grows faster than its probability shrinks, in the limit. Eliezer claims it does, but the question is whether our convenient notations for very, very, very large numbers actually imply some kind of simplicity for those numbers or whether we’re hiding complexity in our brains at that point.
Note that in Hutter’s formalism rewards are bounded between 0 and some r_max. That’s no accident, since if you allow unbounded rewards, the expectation can diverge. Yudkowsky seems to assume unbounded rewards. I think that if you tried to formalize his argument, you would end up attempting to compare infinities. If rewards are bounded, the bias introduced by the fact that the contributions to the expectations from the tail of the distributions don’t exactly cancel out over different actions is eventually washed away as more evidence accumulates.
It seems downright obvious that “3” ought be considered vastly more simple than “3 ^^^ 3″. How large a Turing Machine does it take to write down the fullest expansion of the recursive function for that super-exponentiation?
It’s not really very large.
The point is that there are computable functions that grow faster than exponential. The Solomonoff prior over natural numbers (or any set of computable numbers with an infimum and and not a supremum) has infinite expectation because of the contribution of these functions. (If the set has neither an infimum nor a supremum, I think that the expectation may be finite, positively infinite or negatively infinite depending on the choice of the universal Turing machine and the number encoding)
I’ve been discussing this whole thing on Reddit, in parallel, and I think this is the point where I would just give up and say: revert to evidentialism when discussing unbounded potential rewards. Any hypothesis with a plausibility (ie: my quantity of belief equals its prior, no evidence accumulated) rather than a probability (ie: priors plus evidence) nulls out to zero and is not allowed to contribute to expected-utility calculations.
(Actually, what does Bayesian reasoning look like if you separate priors from evidence and consider an empty set of evidence to contribute a multiplier of 0.0, thus exactly nulling out all theories that consist of no evidence but their priors?)
But even agents based on Solomonoff Induction, such as AIXI, are not subject to Pascal’s Mugging in reasonable environments.
Consider this paper by Hutter. IIUC, the “self-optimizing” property essentially implies robustness against Pascal’s Mugging.
Ah! I’ve got it! I think… At least on the probability side.
The problem is our intuition: the utility of human lives grows linearly with the population of humans, while the message size of the hypothesis needed to describe them grows roughly logarithmically. Since we think we climb down the exponential decline in prior probability slower than utility increases, Pascal’s Bargain sounds favorable.
This is wrong. A real Solomonoff Hypothesis, after all, does not merely say “There are 3^^^3 humans” if there really are 3 ^^^ 3 humans. It describes and predicts each single human, after all, in detail. It’s a hypothesis that aims to predict an entire universe from one small piece of fairy cake.
And when you have to make predictive descriptions of humans, the summed size of those descriptions will grow at least linearly in the number of purported humans. God help you if they start organizing themselves into complicated societies, which are more complex than a mere summed set of single persons. Now your utility from taking the Bargain grows linearly while your negative exponent on the probability of its being real declines linearly.
The question then becomes a simple matter of where your plausability versus utility tradeoff sits for one human life.
Or in other words, if Pascal’s Mugger catches you in a back ally, you should demand that he start describing the people he’s threatening one-by-one.
By this reasoning, if someone has their finger on a trigger for a nuclear bomb that will destroy a city of a million people, and says “give me $1 or I will pick a random number and destroy the city at a 1/1000000 chance”, you should refuse. After all, he cannot describe any of the people he is planning to kill, so this is equivalent to killing one person at a 1/1000000 chance, and you could probably exceed that chance of killing someone just by driving a couple of miles to the supermarket.
If it’s a million people possibly dying at a one-in-a-million chance, then in expected-death terms he’s charging me $1 not to kill one person. Since I believe human lives are worth more than $1, I should give him the dollar and make a “profit”.
Of course, the other issue here, and the reason we don’t make analogies between serious military threats and Pascal’s Mugging, is that in the nuclear bomb case, there is actual evidence on which to update my beliefs. For instance, the button he’s got his finger on is either real, or not. If I can see damn well that it’s a plastic toy from the local store, I’ve no reason to give him a dollar.
So in the case of Soviet Russia threatening you, you’ve got real evidence that they might deliberately nuke your cities with a probability much higher than one in a million. In the case of Pascal’s Mugger, you’ve got a 1⁄1,000,000 chance that the mugger is telling the truth at all, and all the other probability mass points at the mugger being a delusion of your badly-coded reasoning algorithms.
If it’s a million people possibly dying at a one-in-a-million chance, and I use the reasoning you used before, because the mugger can’t describe the people he’s threatening to kill, I shouldn’t treat that as any worse than a threat to kill one person at a one-in-a-million chance.
You misconstrue my position. I’m not saying, “Descriptions are magic!”. I’m saying: I prefer evidentialism to pure Bayesianism. Meaning: if the mugger can’t describe anything about the city under threat, that is evidence that he is lying.
Which misses the point of the scenario, since a real Pascal’s Mugging is not about a physical mugger who could ever be lying. It’s about having a flaw in your own reasoning system.
Blurgh. That’s one of the most symbol-dense papers I’ve ever seen.
But from what I can tell, it specifies that AIXI will eventually reason its way out of any finite Pascal’s Mugging. The higher the hypothesized reward in the Mugging, the longer it will take to converge away from the Mugging, but each failure of reality to conform to the Mugging will push down the probability of that environment being true and thus reduce the expected value of acting according to it. Asymptotic convergence is proven.
I’d also bet that Hutter’s formalism might consider large rewards generated for little reason to be not merely complex because of “little reason”, but actually to have greater Kolmogorov Complexity just because large rewards are more complex than small ones. Possibly. So there would be a question of whether the reward of a Mugging grows faster than its probability shrinks, in the limit. Eliezer claims it does, but the question is whether our convenient notations for very, very, very large numbers actually imply some kind of simplicity for those numbers or whether we’re hiding complexity in our brains at that point.
It seems downright obvious that “3” ought be considered vastly more simple than “3 ^^^ 3″. How large a Turing Machine does it take to write down the fullest expansion of the recursive function for that super-exponentiation? Do we have to expand it out? I would think a computational theory of induction ought make a distinction between computations outputting large numbers and actual large numbers, after all.
I know. It seems that this is Hutter’s usual style.
Well, the Pascal’s Mugging issue essentially boils down to whether the agent decision making is dominated by the bias in its prior.
Clearly, an agent that has seen little or no sensory input can’t have possibly learned anything, and is therefore dominated by its bias. What Hutter proved is that, for reasonable classes of environments, the agent eventually overcomes its bias. There is of course the interesting question of convergence speed, which is not addressed in that paper.
Note that in Hutter’s formalism rewards are bounded between 0 and some r_max. That’s no accident, since if you allow unbounded rewards, the expectation can diverge.
Yudkowsky seems to assume unbounded rewards. I think that if you tried to formalize his argument, you would end up attempting to compare infinities.
If rewards are bounded, the bias introduced by the fact that the contributions to the expectations from the tail of the distributions don’t exactly cancel out over different actions is eventually washed away as more evidence accumulates.
It’s not really very large.
The point is that there are computable functions that grow faster than exponential. The Solomonoff prior over natural numbers (or any set of computable numbers with an infimum and and not a supremum) has infinite expectation because of the contribution of these functions.
(If the set has neither an infimum nor a supremum, I think that the expectation may be finite, positively infinite or negatively infinite depending on the choice of the universal Turing machine and the number encoding)
I’ve been discussing this whole thing on Reddit, in parallel, and I think this is the point where I would just give up and say: revert to evidentialism when discussing unbounded potential rewards. Any hypothesis with a plausibility (ie: my quantity of belief equals its prior, no evidence accumulated) rather than a probability (ie: priors plus evidence) nulls out to zero and is not allowed to contribute to expected-utility calculations.
(Actually, what does Bayesian reasoning look like if you separate priors from evidence and consider an empty set of evidence to contribute a multiplier of 0.0, thus exactly nulling out all theories that consist of no evidence but their priors?)