I think it is impossible for an AI to estimate the expected utility of the next level above its own
I don’t know what that means. It’s always possible to assign probabilities, even if you don’t have a clue. And assigning utilities seems trivial, too.
Let’s say the AI thinks “Hm. Improving my intelligence will lead to world dominion, if a) vast intelligence improvement doesn’t cost too much ressources, b) doesn’t take too long, c) and if intelligence really is as useful as it seems to be, i.e. is more efficient at discovering “unknown unkowns in design space” than other processes (which seems to me tautological since intelligence is by definition optimization power divided by ressources used; but I could be wrong). Let me assign a 50% probability to each of these claims.” (Or less, but it always can assign a probability and can therefore compute an expected utility. )
And so even if P(Gain World dominion|successful, vast intelligence-improvement) * P(succesful, vast intelligence-improvement) is small (and I think it’s easily larger than 0.05) , the expected utility could be great nonetheless. If the AI is a maximizer, not a satisficer, it will try to take over the world.
The biggest problem that I have with recursive self-improvement is that it’s not at all clear that intelligence is an easily “scalable” process. Some folks seem to think that intelligence is a relatively easy algorithm, that a few mathematical insights it will be possible to “grok” intelligence. But maybe you need many different modules and heuristics for general intelligence (just like the human brain) and there is no “one true and easy path”.
But I’m just guessing.…
I agree that the EU of small improvements is easier to compute and that they are easier to implement. But if intelligence is an “scalable” process you can make those small improvements fairly rapidly and after 100 of those you should be pretty, friggin powerful.
Intelligence is mainly useful to adapt previous discoveries and solve well-defined problems.
Do you think the discovery of General Relativity was a well-defined problem? And what about writing of inspiring novels and creating beautiful art and music? Creativity is a subset of intelligence. There are no creative chimps.
What do you mean by intelligence?
And yes, I believe that people with very high IQ and advanced social skills (another instantiation of high intelligence; chimpanzees just don’t have high social skills) are far more likely to take over the world than people with IQ 110, although it’s still very unlikely.
And as mentioned in the beginning, how dependent are AI’s on the precision with which their goals are defined?
My intuition tells me that they are. :-)
If you give them the goal of creating 10 paperclips and nothing else, they will try everything to achieve this goal.
But Eliezer’s arguments in the AI-Foom debate are far more convincing than mine, so my arguments tell you probably nothing new. The whole discussion is frustrating, because our (subconscious) intuitions seem to differ greatly, and there is little we can do about it. (Just like the debate between Hanson and Yudkowsky was pretty fruitless.) That doesn’t mean that I don’t want to participate in further discussions, but the probability of an agreement seems somewhat thin. I try to do my best :-)
I’m currently rereading the Sequences and I’m trying to summarize the various arguments and counter-arguments for the intelligence explosion, though that will take some time...
I don’t know what that means. It’s always possible to assign probabilities, even if you don’t have a clue. And assigning utilities seems trivial, too. Let’s say the AI thinks “Hm. Improving my intelligence will lead to world dominion, if a) vast intelligence improvement doesn’t cost too much ressources, b) doesn’t take too long, c) and if intelligence really is as useful as it seems to be, i.e. is more efficient at discovering “unknown unkowns in design space” than other processes (which seems to me tautological since intelligence is by definition optimization power divided by ressources used; but I could be wrong). Let me assign a 50% probability to each of these claims.” (Or less, but it always can assign a probability and can therefore compute an expected utility. )
And so even if P(Gain World dominion|successful, vast intelligence-improvement) * P(succesful, vast intelligence-improvement) is small (and I think it’s easily larger than 0.05) , the expected utility could be great nonetheless. If the AI is a maximizer, not a satisficer, it will try to take over the world.
The biggest problem that I have with recursive self-improvement is that it’s not at all clear that intelligence is an easily “scalable” process. Some folks seem to think that intelligence is a relatively easy algorithm, that a few mathematical insights it will be possible to “grok” intelligence. But maybe you need many different modules and heuristics for general intelligence (just like the human brain) and there is no “one true and easy path”. But I’m just guessing.…
I agree that the EU of small improvements is easier to compute and that they are easier to implement. But if intelligence is an “scalable” process you can make those small improvements fairly rapidly and after 100 of those you should be pretty, friggin powerful.
Do you think the discovery of General Relativity was a well-defined problem? And what about writing of inspiring novels and creating beautiful art and music? Creativity is a subset of intelligence. There are no creative chimps.
What do you mean by intelligence?
And yes, I believe that people with very high IQ and advanced social skills (another instantiation of high intelligence; chimpanzees just don’t have high social skills) are far more likely to take over the world than people with IQ 110, although it’s still very unlikely.
My intuition tells me that they are. :-) If you give them the goal of creating 10 paperclips and nothing else, they will try everything to achieve this goal.
But Eliezer’s arguments in the AI-Foom debate are far more convincing than mine, so my arguments tell you probably nothing new. The whole discussion is frustrating, because our (subconscious) intuitions seem to differ greatly, and there is little we can do about it. (Just like the debate between Hanson and Yudkowsky was pretty fruitless.) That doesn’t mean that I don’t want to participate in further discussions, but the probability of an agreement seems somewhat thin. I try to do my best :-)
I’m currently rereading the Sequences and I’m trying to summarize the various arguments and counter-arguments for the intelligence explosion, though that will take some time...