It’s not clear whether that will mean the end of humanity in the sense of the systems we’ve created destroying us. It’s not clear if that’s the case, but it’s certainly conceivable. If not, it also just renders humanity a very small phenomenon compared to something else that is far more intelligent and will become incomprehensible to us, as incomprehensible to us as we are to cockroaches.
Q: That’s an interesting thought. [nervous laughter]
Hofstadter: Well, I don’t think it’s interesting. I think it’s terrifying. I hate it. I think about it practically all the time, every single day. [Q: Wow.] And it overwhelms me and depresses me in a way that I haven’t been depressed for a very long time.
I don’t think I’ve ever seen a better description of how I feel about the coming creation of artificial superintelligence. I find myself returning over and over again to that post by benkuhn about “Staring into the abyss as a core life skill” I think that is going to become a necessary core life skill for almost everyone in the coming years.
It has been morbidly gratifying to see more and more people develop the same feelings about AI as I have had for about a year now. Like validation in the worst possible way. I think if people actually understood what was coming there would be a near total call to ban improvements in this technology and only allow advancement under very strict conditions. But almost no one has really thought through the consequences of making a general purpose replacement for human beings.
Yeah. I particularly hate the handwavium which makes this sound like it’s super simple, just make the ASI, have it churn out labour for us, surely human society will adapt just nicely to the new state of things and chill. It’s easy to say this if you think you’re going to be the one in charge of the ASI because you’re a CEO or big shot in some company (you may still be vastly overestimating your chances of controlling the ASI of course but that’s just plain old hubris). But not so easy to believe it if instead you’re more the kind of person who usually gets the short end of the stick. Like, as much as we may celebrate automation, it DOES often have hugely disruptive effects. There’s a lot of pain hidden in that “some jobs are destroyed but others are created so it evens out”. And that’s not even a fraction of the pain that would be possible if ALL jobs were destroyed and never replaced and we would somehow have to find a way to deal with that.
And that’s, again, just the most optimistic scenario. The more pessimistic one is more along the lines of “you’re a megatherium and these strange new hairless apes with sticks have started flooding in from the north”.
I don’t think I’ve ever seen a better description of how I feel about the coming creation of artificial superintelligence. I find myself returning over and over again to that post by benkuhn about “Staring into the abyss as a core life skill” I think that is going to become a necessary core life skill for almost everyone in the coming years.
It has been morbidly gratifying to see more and more people develop the same feelings about AI as I have had for about a year now. Like validation in the worst possible way. I think if people actually understood what was coming there would be a near total call to ban improvements in this technology and only allow advancement under very strict conditions. But almost no one has really thought through the consequences of making a general purpose replacement for human beings.
Yeah. I particularly hate the handwavium which makes this sound like it’s super simple, just make the ASI, have it churn out labour for us, surely human society will adapt just nicely to the new state of things and chill. It’s easy to say this if you think you’re going to be the one in charge of the ASI because you’re a CEO or big shot in some company (you may still be vastly overestimating your chances of controlling the ASI of course but that’s just plain old hubris). But not so easy to believe it if instead you’re more the kind of person who usually gets the short end of the stick. Like, as much as we may celebrate automation, it DOES often have hugely disruptive effects. There’s a lot of pain hidden in that “some jobs are destroyed but others are created so it evens out”. And that’s not even a fraction of the pain that would be possible if ALL jobs were destroyed and never replaced and we would somehow have to find a way to deal with that.
And that’s, again, just the most optimistic scenario. The more pessimistic one is more along the lines of “you’re a megatherium and these strange new hairless apes with sticks have started flooding in from the north”.