Even as a series of worse and worse AGI accidents occur, wih out-of-control AGIs self-replicating around the internet etc., a few people will keep trying to fix the unfixable AGI, seeing this as the only path to get this slow-rolling catastrophe under control (while actually making it worse).
Maybe at this point there would be the political will for a Butlerian Jihad. ;) Or more seriously, a self-imposed ban on AGI similar to the current self-imposed bans on human cloning and biological weapons. I agree this is a long shot given our current experience with climate change, but still, it seems possible. And perhaps the AGI accidents would be more newsworthy and gripping than climate change is, making it easier to rouse the public.
Hmm, interesting. I think human cloning is an imperfect analogy because the only real reason to do it is to impress your friends, so if everyone coordinates on being scornful towards the first person to do human cloning (rather than being impressed), then there’s no more personal benefit to cheating. By contrast, with an AGI, there would be the hope that you’ll actually solve the safety problems, and then get tons of money and power and respect.
Biological weapons is maybe a better example, but not an especially encouraging one: as many as 8 countries may have secret bio-weapons programs, including North Korea. Maybe one could make an argument that there’s a taboo against using bio-weapons, as opposed to merely stockpiling them? Likewise, the taboo against using nuclear weapons was not successfully turned into a taboo against countries starting new nuclear weapons programs. Maybe it’s hard to get riled up against someone doing something that is not purposely aggressive? I don’t know. I can’t think of a great historical analogy.
There’s also the issue that there’s not too many actors who have any reason to start a bio-weapons programs, and the ability to do so without getting shut down. Really just secret military labs. Whereas in the worst case, many orders of magnitude more people would be willing and able to start doing illegal AGI experiments without the authorities realizing it.
I agree that AGI is more omni-use than bioweapons and thus will be harder to get people not to develop and use. I think our prospects look pretty bleak in this scenario, but it’s not completely hopeless.
For human cloning, what I had in mind was a nation cloning its smartest individuals for the purpose of having better science/tech. Think of what the US could have accomplished if they had 10,000 Von Neumanns instead of 1.
Maybe at this point there would be the political will for a Butlerian Jihad. ;) Or more seriously, a self-imposed ban on AGI similar to the current self-imposed bans on human cloning and biological weapons. I agree this is a long shot given our current experience with climate change, but still, it seems possible. And perhaps the AGI accidents would be more newsworthy and gripping than climate change is, making it easier to rouse the public.
Hmm, interesting. I think human cloning is an imperfect analogy because the only real reason to do it is to impress your friends, so if everyone coordinates on being scornful towards the first person to do human cloning (rather than being impressed), then there’s no more personal benefit to cheating. By contrast, with an AGI, there would be the hope that you’ll actually solve the safety problems, and then get tons of money and power and respect.
Biological weapons is maybe a better example, but not an especially encouraging one: as many as 8 countries may have secret bio-weapons programs, including North Korea. Maybe one could make an argument that there’s a taboo against using bio-weapons, as opposed to merely stockpiling them? Likewise, the taboo against using nuclear weapons was not successfully turned into a taboo against countries starting new nuclear weapons programs. Maybe it’s hard to get riled up against someone doing something that is not purposely aggressive? I don’t know. I can’t think of a great historical analogy.
There’s also the issue that there’s not too many actors who have any reason to start a bio-weapons programs, and the ability to do so without getting shut down. Really just secret military labs. Whereas in the worst case, many orders of magnitude more people would be willing and able to start doing illegal AGI experiments without the authorities realizing it.
I agree that AGI is more omni-use than bioweapons and thus will be harder to get people not to develop and use. I think our prospects look pretty bleak in this scenario, but it’s not completely hopeless.
For human cloning, what I had in mind was a nation cloning its smartest individuals for the purpose of having better science/tech. Think of what the US could have accomplished if they had 10,000 Von Neumanns instead of 1.