Maybe there is some consolation in that if the humanity were to arrive at something approaching AGI, it would rather be better for it to do so using an architecture that’s limited in its ultimate capability, demonstrates as little natural agency as possible, ideally that’s a bit of a dead end in terms of further AI development. It could serve as a sort of vaccine if you will.
Running with the singularity scenario for a moment, I have very serious doubts that a purely theoretical research performed largely in a vacuum will yield any progress on AI safety. The history of science certainly doesn’t imply that we will solve this problem before it becomes a serious threat. So the best case scenario we can hope for is that the first crisis caused by the AGI will not be fatal due to the underlying technology’s limitations and manageable speed of improvement.
Maybe there is some consolation in that if the humanity were to arrive at something approaching AGI, it would rather be better for it to do so using an architecture that’s limited in its ultimate capability, demonstrates as little natural agency as possible, ideally that’s a bit of a dead end in terms of further AI development. It could serve as a sort of vaccine if you will.
Running with the singularity scenario for a moment, I have very serious doubts that a purely theoretical research performed largely in a vacuum will yield any progress on AI safety. The history of science certainly doesn’t imply that we will solve this problem before it becomes a serious threat. So the best case scenario we can hope for is that the first crisis caused by the AGI will not be fatal due to the underlying technology’s limitations and manageable speed of improvement.