Thanks for writing this up, as someone who is currently speedrunning through the AI safety literature, I appreciate the summary. I want to dig deeper into one of the questions posted, because it’s been bugging me lately and the answer addressed a different version of the question than I was thinking about.
Re: Isn’t it fine or maybe good if AIs defeat us? They have rights too.
Given the prevalence of doom on LessWrong lately, this seems worth exploring seriously, and not necessarily from an AI rights perspective. If we conclude that alignment is impossible (not necessarily from an engineering perspective, but from a nonproliferation perspective) and an AI that leads to extinction will likely be developed, well… life is unfair, right?
Even still, we have some choices to make before that happens:
All else being equal, I would much rather humanity’s demise come at the hands of a superintelligence that will continue to do interesting things with the universe, such as creating art and advancing science, rather than falling to Clippy’s cold, methodical optimizer. I’d really like it if someday, an intelligent being could figure out why there exists something rather than nothing, even if it’s not a human being. If our silicon children are gonna eat us no matter what, I’d still rather have a cool kid than a boring one. This seems like a more tractable problem.
Certainly I would prefer that humanity “makes it” in the long run, but it doesn’t strike me as a cosmic tragedy if we don’t. If it looks like there’s an effort available to make us keep that from happening, of course it’s worth fighting to make that effort work out. But if not, eh, maybe it’s fine. Shit happens, people die, we’re used to it. I’m okay with extinction if that turns out to be the only option. There’s still a lot of life left for me to live, and I intend to enjoy it until that happens.
I am not, however, as cavalier about s-risk as I am about extinction risk. Which brings me back around to point 1 -- there’s probably work to do even if we accept doom as an inevitable outcome.
So, although I’m still way way too early in my AI-safety reading to say that doom is certain or near-certain and start thinking about how to live my life conditional upon that knowledge, I think it’s important to consider a gameplan in the eventuality that we do decide that doom is locked-in. There are still clear-eyed choices to be made even if we can’t save ourselves.
Thanks for writing this up, as someone who is currently speedrunning through the AI safety literature, I appreciate the summary. I want to dig deeper into one of the questions posted, because it’s been bugging me lately and the answer addressed a different version of the question than I was thinking about.
Re: Isn’t it fine or maybe good if AIs defeat us? They have rights too.
Given the prevalence of doom on LessWrong lately, this seems worth exploring seriously, and not necessarily from an AI rights perspective. If we conclude that alignment is impossible (not necessarily from an engineering perspective, but from a nonproliferation perspective) and an AI that leads to extinction will likely be developed, well… life is unfair, right?
Even still, we have some choices to make before that happens:
All else being equal, I would much rather humanity’s demise come at the hands of a superintelligence that will continue to do interesting things with the universe, such as creating art and advancing science, rather than falling to Clippy’s cold, methodical optimizer. I’d really like it if someday, an intelligent being could figure out why there exists something rather than nothing, even if it’s not a human being. If our silicon children are gonna eat us no matter what, I’d still rather have a cool kid than a boring one. This seems like a more tractable problem.
Certainly I would prefer that humanity “makes it” in the long run, but it doesn’t strike me as a cosmic tragedy if we don’t. If it looks like there’s an effort available to make us keep that from happening, of course it’s worth fighting to make that effort work out. But if not, eh, maybe it’s fine. Shit happens, people die, we’re used to it. I’m okay with extinction if that turns out to be the only option. There’s still a lot of life left for me to live, and I intend to enjoy it until that happens.
I am not, however, as cavalier about s-risk as I am about extinction risk. Which brings me back around to point 1 -- there’s probably work to do even if we accept doom as an inevitable outcome.
So, although I’m still way way too early in my AI-safety reading to say that doom is certain or near-certain and start thinking about how to live my life conditional upon that knowledge, I think it’s important to consider a gameplan in the eventuality that we do decide that doom is locked-in. There are still clear-eyed choices to be made even if we can’t save ourselves.