I would protest that a program to run our known laws of physics (which only predict that 10^80 atoms exist...so there’s no way 3^^^^3 distinct minds could exist) is smaller by some number of bits on the order of log_2(3^^^^3) than one in which I am seemingly running on the known laws of physics, and my choice whether or not to hand over $5 dollars (to someone acting as if they are running on the known laws of physics...seeming generally human, and trying to gain wealth without doing hard work) is positively correlated with whether or not 3^^^^3 minds running on a non-physical server somewhere (yes, this situation requires the postulation of the supernatural) are extinguished. The mugger is claiming to be God in the flesh. This problem is the steel-manned Pascal’s Wager.
And you should not give the mugger the money, for the same reason you should not become a Christian because of Pascal’s wager. Most of the (vast,vast) improbability lies in the mugger’s claim to be God. The truth of eir claim is about as probable as the truth of an idea that popped into a psychopath’s head that e needs to set eir wallet on fire this instant, in order to please the gods, or else they will torture em for 3^^^^3 years. I claim that Eliezer vastly, vastly underestimated the complexity of this situation...which is already enough to solve the problem. But, suppose, just suppose, this mugger really is God. Then which God is e? For every possible mind that does X in situation A, there is a mind that does not-X in situation A. We don’t interact with gods on a day-to-day basis; we interact with humans. We have no prior information about what this mugger will do in this situation if we give em the money. E could be the “Professor Mugger” that punishes you iff you give em the money, because you acted in a way such that any person off the street could have come up to you and taken your money. E could just not do anything either way. You don’t know. Ignorance prior, .5. I have no effect in this situation; I do $5 better in the (extremely, extremely, more likely) situation where this mugger is a con artist. No money for the mugger. There’s also the slight issue that I’m more likely to be mugged this way if I precommit to losing my money...so I’d be pretty stupid to do that.
The AI will shut up and multiply if it’s programmed properly, and get the right answer, my answer, the answer that also happens to be the one we instinctually lean towards here. If we’re running a human-friendly dynamic, there’s no need to worry about the AI making the wrong choice. Do we seriously think we could do better? If so, then why create the AI in the first place?
is smaller by some number of bits on the order of log_2(3^^^^3)
How do you figure? Since we’re not talking about speed, the program seems to this layman like one a super-intelligence could write while still on Earth (perhaps with difficulty). While the number you just named, and even the range of numbers if I take it that way, looks larger than the number of atoms on Earth. The whole point is that you can describe “3^^^^3” pretty simply by comparison to the actual number.
For every possible mind that does X in situation A, there is a mind that does not-X in situation A.
And some are more likely than others (in any coherent way of representing what we know and don’t know), often by slight amounts that matter when you multiply them by 3^^^3. Anyway, the problem we face is not the original Mugger. The problem is that expected value for any given decision may not converge if we have to think this way!
The whole point is that you can describe 3^^^^3 pretty simply
In retrospect, I think Eliezer should not have focused on that as much as he did. Let’s cut to the core of the issue: How should an AI handle the problem of making choices, which, maybe, just maybe, could have a huge, huge effect?
I think Eliezer overlooked the complexity inherent in a mind...the complexity of the situation isn’t in the number; it’s in what the things being numbered are. To create 3^^^^3 distinct,complex things that would be valued by a posthuman would be an incredibly difficult, time-consuming task. Of course, at this moment, the AI doesn’t care about doing that; it cares whether or not the universe is already running 3^^^^3 of these things. I do think a program to run these computations might be more complex than writing a program to simulate our physics, but stepping back, it would not have to be anywhere near log_2(3^^^^3) bits more complex. Really, really bad case of scope insensitivity on my part.
For every possible mind that does X in situation A, there is a mind that does not-X in situation A.
My first comment was wrong. That argument should have been the primary argument, and the other shouldn’t have been in there, at all...but let’s step back from Eliezer’s exact given situation. This is a general problem which applies to, as far as I can see, pretty much any action an AI could take (see Tom_McCabe2′s “QWERTYUIOP” remark).
Let’s say the AI wants to save a drowning child. However, the universe happens to care about this single moment in time, and iff the AI saves the child, 3^^^^3 people will die instant instantly, and then the AI will be given information to verify that this has occurred with high probability. One of the simplest ways for the universe-program to implement this is:
If (AI saves child), then reset all bits in that constantly evolving 3^^^^3-entry long data structure over there to zero, send proof to AI.
Else, proceed normally.
Note that this is magic. Magic is that which cannot be understood, that which correlates with no data other than itself. The code could just as easily be this:
If (AI saves child), then proceed normally.
Else, reset all bits in that constantly evolving 3^^^^3-entry long data structure over there to zero, send proof to AI.
Those two code segments are equally complicated. The AI shouldn’t weight either higher than the other. For each small increment in complexity to the “malevolent” code you make from there, to have it carry out the same function, I contend that you can make a corresponding increment in the “benevolent” code to do the same thing.
If our universe was optimized to give us hope, and then thwart our values, there’s nothing even an AI can do about that. An AI can only optimize that which it both understands, and is permitted to optimize by the universe’s code. The universe’s code could be such that it gives the AI false beliefs about pretty much everything, and then the AI would be unable to optimize anything.
If the “malevolent” code runs, then the AI would make a HUGE update after that, possibly choosing not to save any drowning children anymore (though that update would be wrong if the code were as above...overfitting). But it can’t update on the possibility that it might update—that would violate conservation of expected evidence. All disease might magically immediately be cured if the AI saves the drowing child. I don’t see how this is any more complex.
some are more likely than others
So, this is what I contest. If one was really that much more likely, the AI would have already known about it (cf. what Eliezer says in “Technical Explanation”: “How would I explain the event of my left arm being replaced by a blue tentacle? The answer is that I wouldn’t. It isn’t going to happen....If I was worried I might someday need a clever excuse for waking up with a tentacle, the reason I was nervous about the possibility would be my explanation.”). An AI is designed to accomplish this task as best as is possible. I noticed my confusion when I recalled this paper referring to AIXI I’d previously taken a short look at. The AI won on Partially Observable Pacman; it did much better than I could ever hope to do (if I were given the data in the form of pure numerical reward signals, written down on paper). It didn’t get stuck wondering whether it would lose 2,000,000 points when the most it had ever lost before was less than 100.
I know almost nothing about AI. I don’t know the right way we should approximate AIXI, and modify it so that it knows it is a part of its environment. I do know enough about rationality from reading Less Wrong to know that we shouldn’t shut it off just because it does something counterintuitive, if we did program it right. (And I hope to one day make both of the first two sentences in this paragraph false.)
Also note that, between Tegmark’s multiverses and the Many-Worlds Hypothesis, there are many, many more than 10^80 atoms in existence. 10^80 atoms are only the number atoms we could ever see, assuming FTL is impossible.
I would protest that a program to run our known laws of physics (which only predict that 10^80 atoms exist...so there’s no way 3^^^^3 distinct minds could exist) is smaller by some number of bits on the order of log_2(3^^^^3) than one in which I am seemingly running on the known laws of physics, and my choice whether or not to hand over $5 dollars (to someone acting as if they are running on the known laws of physics...seeming generally human, and trying to gain wealth without doing hard work) is positively correlated with whether or not 3^^^^3 minds running on a non-physical server somewhere (yes, this situation requires the postulation of the supernatural) are extinguished. The mugger is claiming to be God in the flesh. This problem is the steel-manned Pascal’s Wager.
And you should not give the mugger the money, for the same reason you should not become a Christian because of Pascal’s wager. Most of the (vast,vast) improbability lies in the mugger’s claim to be God. The truth of eir claim is about as probable as the truth of an idea that popped into a psychopath’s head that e needs to set eir wallet on fire this instant, in order to please the gods, or else they will torture em for 3^^^^3 years. I claim that Eliezer vastly, vastly underestimated the complexity of this situation...which is already enough to solve the problem. But, suppose, just suppose, this mugger really is God. Then which God is e? For every possible mind that does X in situation A, there is a mind that does not-X in situation A. We don’t interact with gods on a day-to-day basis; we interact with humans. We have no prior information about what this mugger will do in this situation if we give em the money. E could be the “Professor Mugger” that punishes you iff you give em the money, because you acted in a way such that any person off the street could have come up to you and taken your money. E could just not do anything either way. You don’t know. Ignorance prior, .5. I have no effect in this situation; I do $5 better in the (extremely, extremely, more likely) situation where this mugger is a con artist. No money for the mugger. There’s also the slight issue that I’m more likely to be mugged this way if I precommit to losing my money...so I’d be pretty stupid to do that.
The AI will shut up and multiply if it’s programmed properly, and get the right answer, my answer, the answer that also happens to be the one we instinctually lean towards here. If we’re running a human-friendly dynamic, there’s no need to worry about the AI making the wrong choice. Do we seriously think we could do better? If so, then why create the AI in the first place?
(Edit: A lot of this comment is wrong. See my reply to hairyfigment.)
How do you figure? Since we’re not talking about speed, the program seems to this layman like one a super-intelligence could write while still on Earth (perhaps with difficulty). While the number you just named, and even the range of numbers if I take it that way, looks larger than the number of atoms on Earth. The whole point is that you can describe “3^^^^3” pretty simply by comparison to the actual number.
And some are more likely than others (in any coherent way of representing what we know and don’t know), often by slight amounts that matter when you multiply them by 3^^^3. Anyway, the problem we face is not the original Mugger. The problem is that expected value for any given decision may not converge if we have to think this way!
In retrospect, I think Eliezer should not have focused on that as much as he did. Let’s cut to the core of the issue: How should an AI handle the problem of making choices, which, maybe, just maybe, could have a huge, huge effect?
I think Eliezer overlooked the complexity inherent in a mind...the complexity of the situation isn’t in the number; it’s in what the things being numbered are. To create 3^^^^3 distinct, complex things that would be valued by a posthuman would be an incredibly difficult, time-consuming task. Of course, at this moment, the AI doesn’t care about doing that; it cares whether or not the universe is already running 3^^^^3 of these things. I do think a program to run these computations might be more complex than writing a program to simulate our physics, but stepping back, it would not have to be anywhere near log_2(3^^^^3) bits more complex. Really, really bad case of scope insensitivity on my part.
My first comment was wrong. That argument should have been the primary argument, and the other shouldn’t have been in there, at all...but let’s step back from Eliezer’s exact given situation. This is a general problem which applies to, as far as I can see, pretty much any action an AI could take (see Tom_McCabe2′s “QWERTYUIOP” remark).
Let’s say the AI wants to save a drowning child. However, the universe happens to care about this single moment in time, and iff the AI saves the child, 3^^^^3 people will die instant instantly, and then the AI will be given information to verify that this has occurred with high probability. One of the simplest ways for the universe-program to implement this is:
If (AI saves child), then reset all bits in that constantly evolving 3^^^^3-entry long data structure over there to zero, send proof to AI. Else, proceed normally.
Note that this is magic. Magic is that which cannot be understood, that which correlates with no data other than itself. The code could just as easily be this:
If (AI saves child), then proceed normally. Else, reset all bits in that constantly evolving 3^^^^3-entry long data structure over there to zero, send proof to AI.
Those two code segments are equally complicated. The AI shouldn’t weight either higher than the other. For each small increment in complexity to the “malevolent” code you make from there, to have it carry out the same function, I contend that you can make a corresponding increment in the “benevolent” code to do the same thing.
If our universe was optimized to give us hope, and then thwart our values, there’s nothing even an AI can do about that. An AI can only optimize that which it both understands, and is permitted to optimize by the universe’s code. The universe’s code could be such that it gives the AI false beliefs about pretty much everything, and then the AI would be unable to optimize anything.
If the “malevolent” code runs, then the AI would make a HUGE update after that, possibly choosing not to save any drowning children anymore (though that update would be wrong if the code were as above...overfitting). But it can’t update on the possibility that it might update—that would violate conservation of expected evidence. All disease might magically immediately be cured if the AI saves the drowing child. I don’t see how this is any more complex.
So, this is what I contest. If one was really that much more likely, the AI would have already known about it (cf. what Eliezer says in “Technical Explanation”: “How would I explain the event of my left arm being replaced by a blue tentacle? The answer is that I wouldn’t. It isn’t going to happen....If I was worried I might someday need a clever excuse for waking up with a tentacle, the reason I was nervous about the possibility would be my explanation.”). An AI is designed to accomplish this task as best as is possible. I noticed my confusion when I recalled this paper referring to AIXI I’d previously taken a short look at. The AI won on Partially Observable Pacman; it did much better than I could ever hope to do (if I were given the data in the form of pure numerical reward signals, written down on paper). It didn’t get stuck wondering whether it would lose 2,000,000 points when the most it had ever lost before was less than 100.
I know almost nothing about AI. I don’t know the right way we should approximate AIXI, and modify it so that it knows it is a part of its environment. I do know enough about rationality from reading Less Wrong to know that we shouldn’t shut it off just because it does something counterintuitive, if we did program it right. (And I hope to one day make both of the first two sentences in this paragraph false.)
Also note that, between Tegmark’s multiverses and the Many-Worlds Hypothesis, there are many, many more than 10^80 atoms in existence. 10^80 atoms are only the number atoms we could ever see, assuming FTL is impossible.