Why should we assume that vastly increased intelligence results in vastly increased power?
A common argument I see for intelligence being powerful stems from two types of examples:
Humans are vastly more powerful than animals because humans are more intelligent than animals. Thus, an AGI vastly more intelligent than humans would also have similarly overwhelming power over humans.
X famous person caused Y massive changes in society because of their superior intelligence. Thus, an AGI with even more intelligence would be able to effect even larger changes.
However, I could easily imagine the following counterarguments being true:
Human advantages over animals stem not from increased intelligence, but from being just intelligent enough to develop complex language. Complex language allows billions of individual humans across space and time to combine their ideas and knowledge. This accumulated knowledge is the true source of human power. I am not more powerful than a chimpanzee because I am smarter, but because I have access to technology like guns and metal cages. And no one invented guns and metal cages from first principles with pure intellect—both were created through many people’s trial and error. It’s possible that every step humans took to develop guns are possible with the intelligence of a chimpanzee, but chimps simply don’t have the language capabilities to pass these developments on. An AGI, while more intelligent than humans, would not have the fundamental advantage of this language-versus-no-language distinction.
Any human whose success is largely attributed to intelligence (say, Elon Musk) actually gained their success mostly from random luck. Looking at these people and assuming that their intelligence gave them power (when in fact, millions of people with similar intelligence but lower levels of success exist) is simply survivorship bias.
If intelligence were a reliable method of achieving power or success in society, we would expect the vast majority of such people to also be highly intelligent. But may powerful people (politicians, celebrities, etc.) don’t seem to be very intelligent, and make obviously poor decisions all the time.
Couldn’t there be a level of intelligence after which any additional gains in intelligence yield diminishing gains in decision-making ability? For instance, a lack of sufficient information could make the outcome of a decision impossible for any level of intelligence to predict, so the AGI’s vastly greater intelligence over a human would merely result in it choosing an action with a 48% chance of success instead of a 45% chance. (I have a suspicion that most actually important decisions work like this.)
It’s possible that there is a ceiling to intelligence gains. It’s also possible that there isn’t. Looking at the available evidence there doesn’t seem to be so—a single ant is a lot less intelligent than a lobster, which is less intelligent than a snake, etc. etc. While it would be nice (in a way) if there was a ceiling, it seems more prudent to assume that there isn’t, and prepare for the worst. Especially as by “superintelligent”, you shouldn’t think of double, or even triple Einstein, rather you should think of a whole other dimension of intelligence, like the difference between you and a hamster.
As to your specific counterarguments:
It’s both, really. But yes—complex language allows humans to keep and build upon previous knowledge. Humans advantage is in the gigantic amounts of know how that can be passed on to future generations. Which is something that computers are eminently good at—you can keep a local copy of Wikipedia in 20GB.
Good point. But it’s not just luck. Yes, luck plays a large part, but it’s also resources (in a very general sense). If you have the basic required talent and a couple of billions of dollars, I’m pretty sure you could become a hollywood star quite quickly. The point is that a superintelligence won’t have a similar level of intelligence to any one else around. Which will allow it to run circles around everyone. Like if a Einstein level intelligence decided to learn to play chess and started playing against 5 year olds—they might win the first couple of games, but after a while you’d probably notice a trend...
Intelligence is an advantage. Quite a big one, generally speaking. But in society most people are generally at the same level if you compare them to e.g. Gila monsters (because we’re talking about superintelligence). So it shouldn’t be all that surprising that other resources are very important. While many powerful people don’t seem to be intelligent, they tend to be either cunning (which is a different kind of intelligence) or have deep pockets (not just money) which offset their relative lack in smarts. Also, while they might not be very clever, very few powerful people are actively stupid.
Why should we assume that vastly increased intelligence results in vastly increased power?
A common argument I see for intelligence being powerful stems from two types of examples:
Humans are vastly more powerful than animals because humans are more intelligent than animals. Thus, an AGI vastly more intelligent than humans would also have similarly overwhelming power over humans.
X famous person caused Y massive changes in society because of their superior intelligence. Thus, an AGI with even more intelligence would be able to effect even larger changes.
However, I could easily imagine the following counterarguments being true:
Human advantages over animals stem not from increased intelligence, but from being just intelligent enough to develop complex language. Complex language allows billions of individual humans across space and time to combine their ideas and knowledge. This accumulated knowledge is the true source of human power. I am not more powerful than a chimpanzee because I am smarter, but because I have access to technology like guns and metal cages. And no one invented guns and metal cages from first principles with pure intellect—both were created through many people’s trial and error. It’s possible that every step humans took to develop guns are possible with the intelligence of a chimpanzee, but chimps simply don’t have the language capabilities to pass these developments on. An AGI, while more intelligent than humans, would not have the fundamental advantage of this language-versus-no-language distinction.
Any human whose success is largely attributed to intelligence (say, Elon Musk) actually gained their success mostly from random luck. Looking at these people and assuming that their intelligence gave them power (when in fact, millions of people with similar intelligence but lower levels of success exist) is simply survivorship bias.
If intelligence were a reliable method of achieving power or success in society, we would expect the vast majority of such people to also be highly intelligent. But may powerful people (politicians, celebrities, etc.) don’t seem to be very intelligent, and make obviously poor decisions all the time.
Couldn’t there be a level of intelligence after which any additional gains in intelligence yield diminishing gains in decision-making ability? For instance, a lack of sufficient information could make the outcome of a decision impossible for any level of intelligence to predict, so the AGI’s vastly greater intelligence over a human would merely result in it choosing an action with a 48% chance of success instead of a 45% chance. (I have a suspicion that most actually important decisions work like this.)
It’s possible that there is a ceiling to intelligence gains. It’s also possible that there isn’t. Looking at the available evidence there doesn’t seem to be so—a single ant is a lot less intelligent than a lobster, which is less intelligent than a snake, etc. etc. While it would be nice (in a way) if there was a ceiling, it seems more prudent to assume that there isn’t, and prepare for the worst. Especially as by “superintelligent”, you shouldn’t think of double, or even triple Einstein, rather you should think of a whole other dimension of intelligence, like the difference between you and a hamster.
As to your specific counterarguments:
It’s both, really. But yes—complex language allows humans to keep and build upon previous knowledge. Humans advantage is in the gigantic amounts of know how that can be passed on to future generations. Which is something that computers are eminently good at—you can keep a local copy of Wikipedia in 20GB.
Good point. But it’s not just luck. Yes, luck plays a large part, but it’s also resources (in a very general sense). If you have the basic required talent and a couple of billions of dollars, I’m pretty sure you could become a hollywood star quite quickly. The point is that a superintelligence won’t have a similar level of intelligence to any one else around. Which will allow it to run circles around everyone. Like if a Einstein level intelligence decided to learn to play chess and started playing against 5 year olds—they might win the first couple of games, but after a while you’d probably notice a trend...
Intelligence is an advantage. Quite a big one, generally speaking. But in society most people are generally at the same level if you compare them to e.g. Gila monsters (because we’re talking about superintelligence). So it shouldn’t be all that surprising that other resources are very important. While many powerful people don’t seem to be intelligent, they tend to be either cunning (which is a different kind of intelligence) or have deep pockets (not just money) which offset their relative lack in smarts. Also, while they might not be very clever, very few powerful people are actively stupid.