I would like to see a better definition. Not necessarily a good definition, but some pseudo-quantitative description better than “super”, or “substantially smarter than you”.
I believe “superintelligence” is a Yudkowsky coinage, and I know that it came up in the context of recursive self-improvement. Almost everybody in certain circles in 1995 was starting from the paradigm of building a “designed” AGI, incrementally smarter than a human, which would then design something incrementally smarter than itself (and faster than humans had built it), and so forth, so that the intelligence would increase, loosely speaking, exponentially. In that world, no particular threshold matters very much, because you’re presumably going to blow through any such threshold pretty fast, and only stop when you hit physical limits.
That model does not obviously apply to ML. Being trained to be smarter than a human doesn’t imply that you can come up with a fundamentally better way to build the “next generation”, and if all that’s available is more and bigger ML systems, you don’t get that exponential growth (at least until/unless you get smart enough to switch to a designed non-ML successor). Your growth may even be sublinear. You don’t have a clear reason to anticipate immediate ballooning to physical limits, so the question of the “threshold of danger” becomes an important one.
We have people running around with IQs up to maybe the 170s, and such individuals are not able to persuade others do just anything, nor can they design full suites of mature nanotechnology “in their heads”, nor any of the other world-threatening scenarios anybody has brought up. It seems very unlikely that having an IQ of 200 suddenly opens up those sorts of options. The required cognitive capacity is pretty obviously vast, even if you think very differently than a human.
So how smart do you actually have to be to open up those options? I think abuse of IQ for qualitative purposes is reasonable for the moment, so do you need an IQ of 1000? 10000? What? And can any known approach actually scale to that, and how long would it take? Teaching yourself to play go is not in the same league as teaching yourself to take over the world with nanonbots.
There is some evidence that complex nanobots could be invented in ones head with a little more IQ and focus because von Neumann designed a mostly functional (but fragile) replicator in a fake simple physics using the brand-new idea of a cellular automata and without a computer and without the idea of DNA. If a slightly smarter von Neumann focused his life on nanobots, could he have produced, for instance, the works of Robert Freitas but in the 1950s, and only on paper?
I do, however, agree it would be helpful to have different words for different styles of AGI but it seems hard to distinguish these AGIs productively when we don’t yet know the order of development and which key dimensions of distinction will be worth using as we move forward. (human-level vs super-? shallow vs deep? passive vs active? autonomy-types? tightness of self-improvement? etc). Which dimensions will pragmatically matter?
“On paper” isn’t “in your head”, though. In the scenario that led to this, the AI doesn’t get any scratch paper. I guess it could be given large working memory pretty easily, but resources in general aren’t givens.
More importantly, even in domains where you have a lot of experience, paper designs rarely work well without some prototyping and iteration. So far as I know, von Neumann’s replicator was never a detailed mechanical design that could actually be built, and certainly never actually was built. I don’t think anything of any complexity that Bob Freitas designed has ever been built, and I also don’t think any of the complex Freitas designs are complete to the point of being buildable. I haven’t paid much attention since the repirocyte days, so I don’t know what he’s done since then, but that wasn’t even a detailed design, and it even the ideas that were “fleshed out” probably wouldn’t have worked in an actual physiological environment.
von Neumann’s design was in full detail, but, iirc, when it was run for the first time (in the 90s) it had a few bugs that needed fixing. I haven’t followed Freitas in a long time either but agree that the designs weren’t fully spelled out and would have needed iteration.
A different measure than IQ might be useful at some point. An IQ of X effectively means you would need a population of Y humans or more to expect to find at least one human with an IQ of X. As IQs get larger, say over 300, the number of humans you would need in a population to expect to find at least one human with such an IQ becomes ridiculous. Since there are intelligence levels that will not be found in human populations of any size, the minimum population size needed to expect to find someone with IQ X tends to infinity as IQ approaches some fixed value (say, 1000). IQ above that point is undefined.
It would be nice to find a new measure of intelligence that could be used to measure differences between humans and other humans, and also differences between humans and AI. But can we design such a measure? I think raw computing power doesn’t work (how do you compare humans to other humans? Humans to an AI with great hardware but terrible software?)
Could you design a questionnaire that you know the correct answers to, that a very intelligent AI (500 IQ?) could not score perfectly on, but an extremely intelligent AI (1000+ IQ) could score perfectly on? If not, how could we design a measure of intelligence that goes beyond our own intelligence?
Maybe we could define an intelligence factor x to be something like: The average x value for humans is zero. If your x value is 1 greater than mine, then you will outwit me and get what you want 90% of the time, if our utility functions are in direct conflict, such that only one of us can get what we want, assuming we have equal capabilities, and the environment is sufficiently complex. With this scale, I suspect humans probably range in x-factors from −2 to 2, or −3 to 3 if we’re being generous. This scale could let us talk about superintelligences as having an x-factor of 5, or an x-factor of 10, or so on. For example, a superintelligence with an x-factor of 5 has some chance of winning against a superintelligence with an x-factor of 6, but is basically outmatched by a superintelligence with an x-factor of 8.
The reason the “sufficiently complex environment” clause exists, is that superintelligences with x-factors of 10 and 20 may both find the physically optimal strategy for success in the real world, and so who wins may simply be down to chance. We can say an environment where there ceases to be a difference in the strategies between intelligences with an x-factor of 5 and and x-factor of 6 has a complexity factor of 5. I would guess the real world has a complexity factor of around 8, but I have no idea.
I would be terrified of any AI with an x-factor of 4-ish, and Yudkowsky seems to be describing an AI with an x-factor of 5 or 6.
X-factor does seem better than IQ, of course with the proviso that anybody who starts trying to do actual math with either one, or indeed to use it for anything other than this kind of basically qualitative talk, is in serious epistemic trouble.
I would suggest that humans run more like −2 to 1 than like −3 to 3. I guess there could be a very, very few 2s.
I get the impression that, except when he’s being especially careful for some specific reason, EY tends to speak as though the X-factor of an AI could and would quickly run up high enough that you couldn’t measure it. More like 20 or 30 than 5 or 6; basically deity-level. Maybe it’s a habit from the 1995 era, or maybe he has some reason to believe that that I don’t understand.
Personally, I have the general impression that you’d be hard pressed to get to 3 with an early ML-based AI, and I think that the “equal capabilities” handicap could realistically be made significant. Maybe 3?
So about this word “superintelligence”.
I would like to see a better definition. Not necessarily a good definition, but some pseudo-quantitative description better than “super”, or “substantially smarter than you”.
I believe “superintelligence” is a Yudkowsky coinage, and I know that it came up in the context of recursive self-improvement. Almost everybody in certain circles in 1995 was starting from the paradigm of building a “designed” AGI, incrementally smarter than a human, which would then design something incrementally smarter than itself (and faster than humans had built it), and so forth, so that the intelligence would increase, loosely speaking, exponentially. In that world, no particular threshold matters very much, because you’re presumably going to blow through any such threshold pretty fast, and only stop when you hit physical limits.
That model does not obviously apply to ML. Being trained to be smarter than a human doesn’t imply that you can come up with a fundamentally better way to build the “next generation”, and if all that’s available is more and bigger ML systems, you don’t get that exponential growth (at least until/unless you get smart enough to switch to a designed non-ML successor). Your growth may even be sublinear. You don’t have a clear reason to anticipate immediate ballooning to physical limits, so the question of the “threshold of danger” becomes an important one.
We have people running around with IQs up to maybe the 170s, and such individuals are not able to persuade others do just anything, nor can they design full suites of mature nanotechnology “in their heads”, nor any of the other world-threatening scenarios anybody has brought up. It seems very unlikely that having an IQ of 200 suddenly opens up those sorts of options. The required cognitive capacity is pretty obviously vast, even if you think very differently than a human.
So how smart do you actually have to be to open up those options? I think abuse of IQ for qualitative purposes is reasonable for the moment, so do you need an IQ of 1000? 10000? What? And can any known approach actually scale to that, and how long would it take? Teaching yourself to play go is not in the same league as teaching yourself to take over the world with nanonbots.
There is some evidence that complex nanobots could be invented in ones head with a little more IQ and focus because von Neumann designed a mostly functional (but fragile) replicator in a fake simple physics using the brand-new idea of a cellular automata and without a computer and without the idea of DNA. If a slightly smarter von Neumann focused his life on nanobots, could he have produced, for instance, the works of Robert Freitas but in the 1950s, and only on paper?
I do, however, agree it would be helpful to have different words for different styles of AGI but it seems hard to distinguish these AGIs productively when we don’t yet know the order of development and which key dimensions of distinction will be worth using as we move forward. (human-level vs super-? shallow vs deep? passive vs active? autonomy-types? tightness of self-improvement? etc). Which dimensions will pragmatically matter?
“On paper” isn’t “in your head”, though. In the scenario that led to this, the AI doesn’t get any scratch paper. I guess it could be given large working memory pretty easily, but resources in general aren’t givens.
More importantly, even in domains where you have a lot of experience, paper designs rarely work well without some prototyping and iteration. So far as I know, von Neumann’s replicator was never a detailed mechanical design that could actually be built, and certainly never actually was built. I don’t think anything of any complexity that Bob Freitas designed has ever been built, and I also don’t think any of the complex Freitas designs are complete to the point of being buildable. I haven’t paid much attention since the repirocyte days, so I don’t know what he’s done since then, but that wasn’t even a detailed design, and it even the ideas that were “fleshed out” probably wouldn’t have worked in an actual physiological environment.
von Neumann’s design was in full detail, but, iirc, when it was run for the first time (in the 90s) it had a few bugs that needed fixing. I haven’t followed Freitas in a long time either but agree that the designs weren’t fully spelled out and would have needed iteration.
I’m very interested in doing this! Please DM me if you think it might be worth collaborating :)
A different measure than IQ might be useful at some point. An IQ of X effectively means you would need a population of Y humans or more to expect to find at least one human with an IQ of X. As IQs get larger, say over 300, the number of humans you would need in a population to expect to find at least one human with such an IQ becomes ridiculous. Since there are intelligence levels that will not be found in human populations of any size, the minimum population size needed to expect to find someone with IQ X tends to infinity as IQ approaches some fixed value (say, 1000). IQ above that point is undefined.
It would be nice to find a new measure of intelligence that could be used to measure differences between humans and other humans, and also differences between humans and AI. But can we design such a measure? I think raw computing power doesn’t work (how do you compare humans to other humans? Humans to an AI with great hardware but terrible software?)
Could you design a questionnaire that you know the correct answers to, that a very intelligent AI (500 IQ?) could not score perfectly on, but an extremely intelligent AI (1000+ IQ) could score perfectly on? If not, how could we design a measure of intelligence that goes beyond our own intelligence?
Maybe we could define an intelligence factor x to be something like: The average x value for humans is zero. If your x value is 1 greater than mine, then you will outwit me and get what you want 90% of the time, if our utility functions are in direct conflict, such that only one of us can get what we want, assuming we have equal capabilities, and the environment is sufficiently complex. With this scale, I suspect humans probably range in x-factors from −2 to 2, or −3 to 3 if we’re being generous. This scale could let us talk about superintelligences as having an x-factor of 5, or an x-factor of 10, or so on. For example, a superintelligence with an x-factor of 5 has some chance of winning against a superintelligence with an x-factor of 6, but is basically outmatched by a superintelligence with an x-factor of 8.
The reason the “sufficiently complex environment” clause exists, is that superintelligences with x-factors of 10 and 20 may both find the physically optimal strategy for success in the real world, and so who wins may simply be down to chance. We can say an environment where there ceases to be a difference in the strategies between intelligences with an x-factor of 5 and and x-factor of 6 has a complexity factor of 5. I would guess the real world has a complexity factor of around 8, but I have no idea.
I would be terrified of any AI with an x-factor of 4-ish, and Yudkowsky seems to be describing an AI with an x-factor of 5 or 6.
X-factor does seem better than IQ, of course with the proviso that anybody who starts trying to do actual math with either one, or indeed to use it for anything other than this kind of basically qualitative talk, is in serious epistemic trouble.
I would suggest that humans run more like −2 to 1 than like −3 to 3. I guess there could be a very, very few 2s.
I get the impression that, except when he’s being especially careful for some specific reason, EY tends to speak as though the X-factor of an AI could and would quickly run up high enough that you couldn’t measure it. More like 20 or 30 than 5 or 6; basically deity-level. Maybe it’s a habit from the 1995 era, or maybe he has some reason to believe that that I don’t understand.
Personally, I have the general impression that you’d be hard pressed to get to 3 with an early ML-based AI, and I think that the “equal capabilities” handicap could realistically be made significant. Maybe 3?