Imagine it’s 2022 (it is!), and your plan for reducing existential risk is to build or maintain an institution that aims to find a way for you — or someone else you’ll later identify and ally with — to use AGI to forcibly shut down all other AGI projects in the world.
People can start new ones. Therefore people must be destroyed. All die.
What are you imagining, when you imagine “use AGI to forcibly shut down all other AGI projects in the world”? I imagine it might begin by hacking into nuclear weapons systems and virology labs, sending exploding drones to kill everyone who works on AGI except those bearing the Dark Mark of the Death Eaters, and going downhill from there.
More generally, this strategy is an instance of the pattern, “We have THE TRUTH! We must impose it on the whole world lest they fall into ERROR!” This never has good results.
I can’t imagine Coral having anything good to say about this either. It’s even worse than what Mr. Topaz is trying to do in that dialogue. It doesn’t even try to climb the success curve, but runs headlong in the opposite direction.
Therefore, the first group to develop AGI, assuming they manage to align it well enough with their own values that they believe they can safely issue instructions to it
This is the language of dreams, not the language of reality.
People can start new ones. Therefore people must be destroyed. All die.
What are you imagining, when you imagine “use AGI to forcibly shut down all other AGI projects in the world”? I imagine it might begin by hacking into nuclear weapons systems and virology labs, sending exploding drones to kill everyone who works on AGI except those bearing the Dark Mark of the Death Eaters, and going downhill from there.
More generally, this strategy is an instance of the pattern, “We have THE TRUTH! We must impose it on the whole world lest they fall into ERROR!” This never has good results.
I can’t imagine Coral having anything good to say about this either. It’s even worse than what Mr. Topaz is trying to do in that dialogue. It doesn’t even try to climb the success curve, but runs headlong in the opposite direction.
This is the language of dreams, not the language of reality.