is smaller by some number of bits on the order of log_2(3^^^^3)
How do you figure? Since we’re not talking about speed, the program seems to this layman like one a super-intelligence could write while still on Earth (perhaps with difficulty). While the number you just named, and even the range of numbers if I take it that way, looks larger than the number of atoms on Earth. The whole point is that you can describe “3^^^^3” pretty simply by comparison to the actual number.
For every possible mind that does X in situation A, there is a mind that does not-X in situation A.
And some are more likely than others (in any coherent way of representing what we know and don’t know), often by slight amounts that matter when you multiply them by 3^^^3. Anyway, the problem we face is not the original Mugger. The problem is that expected value for any given decision may not converge if we have to think this way!
The whole point is that you can describe 3^^^^3 pretty simply
In retrospect, I think Eliezer should not have focused on that as much as he did. Let’s cut to the core of the issue: How should an AI handle the problem of making choices, which, maybe, just maybe, could have a huge, huge effect?
I think Eliezer overlooked the complexity inherent in a mind...the complexity of the situation isn’t in the number; it’s in what the things being numbered are. To create 3^^^^3 distinct,complex things that would be valued by a posthuman would be an incredibly difficult, time-consuming task. Of course, at this moment, the AI doesn’t care about doing that; it cares whether or not the universe is already running 3^^^^3 of these things. I do think a program to run these computations might be more complex than writing a program to simulate our physics, but stepping back, it would not have to be anywhere near log_2(3^^^^3) bits more complex. Really, really bad case of scope insensitivity on my part.
For every possible mind that does X in situation A, there is a mind that does not-X in situation A.
My first comment was wrong. That argument should have been the primary argument, and the other shouldn’t have been in there, at all...but let’s step back from Eliezer’s exact given situation. This is a general problem which applies to, as far as I can see, pretty much any action an AI could take (see Tom_McCabe2′s “QWERTYUIOP” remark).
Let’s say the AI wants to save a drowning child. However, the universe happens to care about this single moment in time, and iff the AI saves the child, 3^^^^3 people will die instant instantly, and then the AI will be given information to verify that this has occurred with high probability. One of the simplest ways for the universe-program to implement this is:
If (AI saves child), then reset all bits in that constantly evolving 3^^^^3-entry long data structure over there to zero, send proof to AI.
Else, proceed normally.
Note that this is magic. Magic is that which cannot be understood, that which correlates with no data other than itself. The code could just as easily be this:
If (AI saves child), then proceed normally.
Else, reset all bits in that constantly evolving 3^^^^3-entry long data structure over there to zero, send proof to AI.
Those two code segments are equally complicated. The AI shouldn’t weight either higher than the other. For each small increment in complexity to the “malevolent” code you make from there, to have it carry out the same function, I contend that you can make a corresponding increment in the “benevolent” code to do the same thing.
If our universe was optimized to give us hope, and then thwart our values, there’s nothing even an AI can do about that. An AI can only optimize that which it both understands, and is permitted to optimize by the universe’s code. The universe’s code could be such that it gives the AI false beliefs about pretty much everything, and then the AI would be unable to optimize anything.
If the “malevolent” code runs, then the AI would make a HUGE update after that, possibly choosing not to save any drowning children anymore (though that update would be wrong if the code were as above...overfitting). But it can’t update on the possibility that it might update—that would violate conservation of expected evidence. All disease might magically immediately be cured if the AI saves the drowing child. I don’t see how this is any more complex.
some are more likely than others
So, this is what I contest. If one was really that much more likely, the AI would have already known about it (cf. what Eliezer says in “Technical Explanation”: “How would I explain the event of my left arm being replaced by a blue tentacle? The answer is that I wouldn’t. It isn’t going to happen....If I was worried I might someday need a clever excuse for waking up with a tentacle, the reason I was nervous about the possibility would be my explanation.”). An AI is designed to accomplish this task as best as is possible. I noticed my confusion when I recalled this paper referring to AIXI I’d previously taken a short look at. The AI won on Partially Observable Pacman; it did much better than I could ever hope to do (if I were given the data in the form of pure numerical reward signals, written down on paper). It didn’t get stuck wondering whether it would lose 2,000,000 points when the most it had ever lost before was less than 100.
I know almost nothing about AI. I don’t know the right way we should approximate AIXI, and modify it so that it knows it is a part of its environment. I do know enough about rationality from reading Less Wrong to know that we shouldn’t shut it off just because it does something counterintuitive, if we did program it right. (And I hope to one day make both of the first two sentences in this paragraph false.)
How do you figure? Since we’re not talking about speed, the program seems to this layman like one a super-intelligence could write while still on Earth (perhaps with difficulty). While the number you just named, and even the range of numbers if I take it that way, looks larger than the number of atoms on Earth. The whole point is that you can describe “3^^^^3” pretty simply by comparison to the actual number.
And some are more likely than others (in any coherent way of representing what we know and don’t know), often by slight amounts that matter when you multiply them by 3^^^3. Anyway, the problem we face is not the original Mugger. The problem is that expected value for any given decision may not converge if we have to think this way!
In retrospect, I think Eliezer should not have focused on that as much as he did. Let’s cut to the core of the issue: How should an AI handle the problem of making choices, which, maybe, just maybe, could have a huge, huge effect?
I think Eliezer overlooked the complexity inherent in a mind...the complexity of the situation isn’t in the number; it’s in what the things being numbered are. To create 3^^^^3 distinct, complex things that would be valued by a posthuman would be an incredibly difficult, time-consuming task. Of course, at this moment, the AI doesn’t care about doing that; it cares whether or not the universe is already running 3^^^^3 of these things. I do think a program to run these computations might be more complex than writing a program to simulate our physics, but stepping back, it would not have to be anywhere near log_2(3^^^^3) bits more complex. Really, really bad case of scope insensitivity on my part.
My first comment was wrong. That argument should have been the primary argument, and the other shouldn’t have been in there, at all...but let’s step back from Eliezer’s exact given situation. This is a general problem which applies to, as far as I can see, pretty much any action an AI could take (see Tom_McCabe2′s “QWERTYUIOP” remark).
Let’s say the AI wants to save a drowning child. However, the universe happens to care about this single moment in time, and iff the AI saves the child, 3^^^^3 people will die instant instantly, and then the AI will be given information to verify that this has occurred with high probability. One of the simplest ways for the universe-program to implement this is:
If (AI saves child), then reset all bits in that constantly evolving 3^^^^3-entry long data structure over there to zero, send proof to AI. Else, proceed normally.
Note that this is magic. Magic is that which cannot be understood, that which correlates with no data other than itself. The code could just as easily be this:
If (AI saves child), then proceed normally. Else, reset all bits in that constantly evolving 3^^^^3-entry long data structure over there to zero, send proof to AI.
Those two code segments are equally complicated. The AI shouldn’t weight either higher than the other. For each small increment in complexity to the “malevolent” code you make from there, to have it carry out the same function, I contend that you can make a corresponding increment in the “benevolent” code to do the same thing.
If our universe was optimized to give us hope, and then thwart our values, there’s nothing even an AI can do about that. An AI can only optimize that which it both understands, and is permitted to optimize by the universe’s code. The universe’s code could be such that it gives the AI false beliefs about pretty much everything, and then the AI would be unable to optimize anything.
If the “malevolent” code runs, then the AI would make a HUGE update after that, possibly choosing not to save any drowning children anymore (though that update would be wrong if the code were as above...overfitting). But it can’t update on the possibility that it might update—that would violate conservation of expected evidence. All disease might magically immediately be cured if the AI saves the drowing child. I don’t see how this is any more complex.
So, this is what I contest. If one was really that much more likely, the AI would have already known about it (cf. what Eliezer says in “Technical Explanation”: “How would I explain the event of my left arm being replaced by a blue tentacle? The answer is that I wouldn’t. It isn’t going to happen....If I was worried I might someday need a clever excuse for waking up with a tentacle, the reason I was nervous about the possibility would be my explanation.”). An AI is designed to accomplish this task as best as is possible. I noticed my confusion when I recalled this paper referring to AIXI I’d previously taken a short look at. The AI won on Partially Observable Pacman; it did much better than I could ever hope to do (if I were given the data in the form of pure numerical reward signals, written down on paper). It didn’t get stuck wondering whether it would lose 2,000,000 points when the most it had ever lost before was less than 100.
I know almost nothing about AI. I don’t know the right way we should approximate AIXI, and modify it so that it knows it is a part of its environment. I do know enough about rationality from reading Less Wrong to know that we shouldn’t shut it off just because it does something counterintuitive, if we did program it right. (And I hope to one day make both of the first two sentences in this paragraph false.)