and my best guess about human evolution is pretty similar, it looks like humans are smart enough to foom over a few hundred thousand years [...]
As I see it, humans could indeed foom over a few hundred thousand years, and we’re probably 99.9% of the way through the timeline, but we haven’t actually foomed yet, not in the sense that we mean by AGI fooming. We might, if given another few hundred years without global disaster.
There are really three completely separate stages of process here.
Almost all of the past few million years has probably been evolution improving hominid brains enough to support general intelligence at all. That was probably reached fifty thousand years ago, maybe even somewhat longer. I think AI is still somewhere in this stage. Our computer hardware may or may not already have capacity to support AGI, but the software is definitely not there yet. In biological terms, I think our AI progress has gone through at least a billion years worth of evolution in two decades, not that these are really comparable.
The last fifty thousand years has mostly been building cultural knowledge, made possible by the previous improvements. We have been building toward the potential that our level of AGI opened up, and I don’t think we’re close to an endpoint there yet. AI will get a huge jump-start in this stage by adapting much of our knowledge, and even non-AGI systems can make limited use of it to mimic AGI capabilities. It makes the difference between our ancestors wielding sharp rocks and a world spanning civilization, but in my opinion it isn’t “foom”. If that was all we had to worry about from AGI, it wouldn’t be nearly such an existential risk.
The centuries yet to come (without AGI) could have involved using our increasing proficiency with shaping the world to deliberately do what evolution did blindly: improve the fundamental capabilities of cognition itself, and beyond that to use the increased cognitive potential to increase knowledge and cognitive potential further still. This is how I understand the term “foom”, and it seems very likely that AI will enter this stage long before humans do.
As I see it, humans could indeed foom over a few hundred thousand years, and we’re probably 99.9% of the way through the timeline, but we haven’t actually foomed yet, not in the sense that we mean by AGI fooming. We might, if given another few hundred years without global disaster.
There are really three completely separate stages of process here.
Almost all of the past few million years has probably been evolution improving hominid brains enough to support general intelligence at all. That was probably reached fifty thousand years ago, maybe even somewhat longer. I think AI is still somewhere in this stage. Our computer hardware may or may not already have capacity to support AGI, but the software is definitely not there yet. In biological terms, I think our AI progress has gone through at least a billion years worth of evolution in two decades, not that these are really comparable.
The last fifty thousand years has mostly been building cultural knowledge, made possible by the previous improvements. We have been building toward the potential that our level of AGI opened up, and I don’t think we’re close to an endpoint there yet. AI will get a huge jump-start in this stage by adapting much of our knowledge, and even non-AGI systems can make limited use of it to mimic AGI capabilities. It makes the difference between our ancestors wielding sharp rocks and a world spanning civilization, but in my opinion it isn’t “foom”. If that was all we had to worry about from AGI, it wouldn’t be nearly such an existential risk.
The centuries yet to come (without AGI) could have involved using our increasing proficiency with shaping the world to deliberately do what evolution did blindly: improve the fundamental capabilities of cognition itself, and beyond that to use the increased cognitive potential to increase knowledge and cognitive potential further still. This is how I understand the term “foom”, and it seems very likely that AI will enter this stage long before humans do.