Another thing: We need to distinguish between getting better at designing intelligences vs. getting better at designing intelligences which are in turn better than one’s own. The claim that “the smarter you are, the better you are at designing intelligences” can be interpreted as stating that the function f(x, y) outlined above is decreasing for any fixed y. But the claim that the smarter you are, the easier it is to create an intelligence even smarter is totally different and equivalent to the aforementioned thesis about the shape of f(x, x+1).
I see the two claims conflated shockingly often, e.g., in Bostrom’s article, where he simply states:
Once artificial intelligence reaches human level, there will be a positive feedback loop that will give the development a further boost. AIs would help constructing better AIs, which in turn would help building better AIs, and so forth.
and concludes that superintelligence inevitably follows with no intermediary reasoning on the software level. (Actually, he doesn’t state that outright, but the sentence is at the beginning of the section entitled “Once there is human-level AI there will soon be superintelligence.”) That an IQ 180 AI is (much) better at developing an IQ 190 AI than a human is doesn’t imply that it can develop an IQ 190 AI faster than the human can develop the IQ 180 AI.
Here’s a line of reasoning that seems to suggest the possibility of an interesting region of decreasing f(x, x+1). It focuses on human evolution and evolutionary algorithms.
Human intelligence appeared relatively recently through an evolutionary process. There doesn’t seem to be much reason to believe that if the evolutionary process were allowed to continue (instead of being largely pre-empted by memetic and technological evolution) that future hominids wouldn’t be considerably smarter. Suppose that evolutionary algorithms can be used to design a human-equivalent intelligence with minimal supervision/intervention by truly intelligent-design methods. In that case, we would expect with some substantial probability that carrying the evolution forward would lead to more intelligence. Since the evolutionary experiment is largely driven by brute-force computation, any increase in computing power underlying the evolutionary “playing field” would increase the rate of increase of intelligence of the evolving population.
I’m not an expert on or even practitioner of evolutionary design, so please criticize and correct this line of reasoning.
I agree there’s good reason to imagine that, had further selective pressure on increased intelligence been applied in our evolutionary history, we probably would’ve ended up more intelligent on average. What’s substantially less clear is whether we would’ve ended up much outside the present observed range of intelligence variation had this happened. If current human brain architecture happens to be very close to a local maximum of intelligence, then raising the average IQ by 50 points still may not get us to any IQ 200 individuals. So while there likely is a nearby region of decreasing f(x, x+1), it doesn’t seem so obvious that it’s wide enough to terminate in superintelligence. Given the notorious complexity of biological systems, it’s extremely difficult to extrapolate anything about the theoretical limits of evolutionary optimization.
Another thing: We need to distinguish between getting better at designing intelligences vs. getting better at designing intelligences which are in turn better than one’s own. The claim that “the smarter you are, the better you are at designing intelligences” can be interpreted as stating that the function f(x, y) outlined above is decreasing for any fixed y. But the claim that the smarter you are, the easier it is to create an intelligence even smarter is totally different and equivalent to the aforementioned thesis about the shape of f(x, x+1).
I see the two claims conflated shockingly often, e.g., in Bostrom’s article, where he simply states:
and concludes that superintelligence inevitably follows with no intermediary reasoning on the software level. (Actually, he doesn’t state that outright, but the sentence is at the beginning of the section entitled “Once there is human-level AI there will soon be superintelligence.”) That an IQ 180 AI is (much) better at developing an IQ 190 AI than a human is doesn’t imply that it can develop an IQ 190 AI faster than the human can develop the IQ 180 AI.
Here’s a line of reasoning that seems to suggest the possibility of an interesting region of decreasing f(x, x+1). It focuses on human evolution and evolutionary algorithms.
Human intelligence appeared relatively recently through an evolutionary process. There doesn’t seem to be much reason to believe that if the evolutionary process were allowed to continue (instead of being largely pre-empted by memetic and technological evolution) that future hominids wouldn’t be considerably smarter. Suppose that evolutionary algorithms can be used to design a human-equivalent intelligence with minimal supervision/intervention by truly intelligent-design methods. In that case, we would expect with some substantial probability that carrying the evolution forward would lead to more intelligence. Since the evolutionary experiment is largely driven by brute-force computation, any increase in computing power underlying the evolutionary “playing field” would increase the rate of increase of intelligence of the evolving population.
I’m not an expert on or even practitioner of evolutionary design, so please criticize and correct this line of reasoning.
I agree there’s good reason to imagine that, had further selective pressure on increased intelligence been applied in our evolutionary history, we probably would’ve ended up more intelligent on average. What’s substantially less clear is whether we would’ve ended up much outside the present observed range of intelligence variation had this happened. If current human brain architecture happens to be very close to a local maximum of intelligence, then raising the average IQ by 50 points still may not get us to any IQ 200 individuals. So while there likely is a nearby region of decreasing f(x, x+1), it doesn’t seem so obvious that it’s wide enough to terminate in superintelligence. Given the notorious complexity of biological systems, it’s extremely difficult to extrapolate anything about the theoretical limits of evolutionary optimization.