I don’t think either of those are the two most likely.
I see the most likely as related to (2), but nothing to do with the likelihood of AI causing disaster. More likely there will be some other disruption to our society that has nothing to do with AI, but prevents us from making sufficient progress to reach superhuman AGI for the near future. Probably we will recover in the less near future, but that’s out of scope of the question.
Second most likely I see as being some as yet unknown obstacle that makes AGI unexpectedly unlikely with near future technology. The future is, after all, hard to predict. That doesn’t mean that we technologically plateau in general, just that this one problem is much harder than we expect.
A technological plateau is strictly necessary. To give the simplest example; we lucked out on nukes. The next decade alone contains potential for several existential threats—readily made bioweapons, miniaturized drones, AI abuse—that I question our ability to consistently adapt too, particularly one after another.
We might get it, if our tech jumps thanks to exponential progress.
No, it is definitely not a strictly necessary requirement for near-term survival. To be “strictly necessary for near-term survival”, such future technologies would have to be guaranteed to kill all of humanity, and soon. That’s ridiculous hyperbole.
There are risks ahead, even existential risks, from other non-AI technologies but not to nearly that extent.
We’re very good at generating existential risks. Given indefinite technological progression at our current pace, we are likely to get ourselves killed.
Your post—and my comment—are explicitly about necessary requirements for near-term survival. If you want to make another post about indefinite-term existential risks, then we can talk about that.
I don’t think either of those are the two most likely.
I see the most likely as related to (2), but nothing to do with the likelihood of AI causing disaster. More likely there will be some other disruption to our society that has nothing to do with AI, but prevents us from making sufficient progress to reach superhuman AGI for the near future. Probably we will recover in the less near future, but that’s out of scope of the question.
Second most likely I see as being some as yet unknown obstacle that makes AGI unexpectedly unlikely with near future technology. The future is, after all, hard to predict. That doesn’t mean that we technologically plateau in general, just that this one problem is much harder than we expect.
A technological plateau is strictly necessary. To give the simplest example; we lucked out on nukes. The next decade alone contains potential for several existential threats—readily made bioweapons, miniaturized drones, AI abuse—that I question our ability to consistently adapt too, particularly one after another.
We might get it, if our tech jumps thanks to exponential progress.
No, it is definitely not a strictly necessary requirement for near-term survival. To be “strictly necessary for near-term survival”, such future technologies would have to be guaranteed to kill all of humanity, and soon. That’s ridiculous hyperbole.
There are risks ahead, even existential risks, from other non-AI technologies but not to nearly that extent.
We’re very good at generating existential risks. Given indefinite technological progression at our current pace, we are likely to get ourselves killed.
Your post—and my comment—are explicitly about necessary requirements for near-term survival. If you want to make another post about indefinite-term existential risks, then we can talk about that.