Even if you somehow get AI to human level and stop, you’d still have a whole new species that was capable of duplicating themselves
Also capable of doing AI research themselves. It would be incredibly strange if automating AI research didn’t accelerate AI research.
It’s really unlikely for us to get to human level and then stop, given that that’s [...] not how the evolution of intelligence worked in the biological world
Well, evolution did “stop” at human-level intelligence!
I would add that “human-level intelligence” is just a very specific level to be at. Cf. “bald-eagle-level carrying capacity”; on priors you wouldn’t expect airplanes to exactly hit that narrow target out of the full range of physically possible carrying capacities.
Also capable of doing AI research themselves. It would be incredibly strange if automating AI research didn’t accelerate AI research.
Well, evolution did “stop” at human-level intelligence!
I would add that “human-level intelligence” is just a very specific level to be at. Cf. “bald-eagle-level carrying capacity”; on priors you wouldn’t expect airplanes to exactly hit that narrow target out of the full range of physically possible carrying capacities.