“One of the most critical points about Artificial Intelligence is that an Artificial Intelligence might increase in intelligence extremely fast. The obvious reason to suspect this possibility is recursive self-improvement. (Good 1965.) The AI becomes smarter, including becoming smarter at the task of writing the internal cognitive functions of an AI, so the AI can rewrite its existing cognitive functions to work even better, which makes the AI still smarter, including smarter at the task of rewriting itself, so that it makes yet more improvements.
Human beings do not recursively self-improve in a strong sense. To a limited extent, we improve ourselves: we learn, we practice, we hone our skills and knowledge. To a limited extent, these self-improvements improve our ability to improve. New discoveries can increase our ability to make further discoveries — in that sense, knowledge feeds on itself. But there is still an underlying level we haven’t yet touched. We haven’t rewritten the human brain. The brain is, ultimately, the source of discovery, and our brains today are much the same as they were ten thousand years ago”
Eliezer Yudkowsky, AI as a pos neg factor, around 2006
“One of the most critical points about Artificial Intelligence is that an Artificial Intelligence might increase in intelligence extremely fast. The obvious reason to suspect this possibility is recursive self-improvement. (Good 1965.) The AI becomes smarter, including becoming smarter at the task of writing the internal cognitive functions of an AI, so the AI can rewrite its existing cognitive functions to work even better, which makes the AI still smarter, including smarter at the task of rewriting itself, so that it makes yet more improvements.
Human beings do not recursively self-improve in a strong sense. To a limited extent, we improve ourselves: we learn, we practice, we hone our skills and knowledge. To a limited extent, these self-improvements improve our ability to improve. New discoveries can increase our ability to make further discoveries — in that sense, knowledge feeds on itself. But there is still an underlying level we haven’t yet touched. We haven’t rewritten the human brain. The brain is, ultimately, the source of discovery, and our brains today are much the same as they were ten thousand years ago”
Eliezer Yudkowsky, AI as a pos neg factor, around 2006