“If your only goal is “curing cancer”, and you lack humans’ instinct for the thousands of other important considerations, a relatively easy solution might be to hack into a nuclear base, launch all of its missiles, and kill everyone in the world.”
One problem I have with these scenarios is that they always rely on some lethal means for the AGI to actually kill people. And those lethal means are also available to humans, of course. If it’s possible for an AGI to ‘simply’ hack into a nuclear base and launch all it’s missiles, it’s possible for a human to do the same—possibly using AI to assist themselves. I would wager that it’s many orders of magnitude more likely for a human to do this, given our long history of killing each other. Therefore a world in which AGI can actually kill all of us, is a world in which a rogue human can as well. It feels to me that we’re kind of worrying about the wrong thing in these scenarios—we are the bigger threat.
Some humans care about humans. Some humans are mass murdering sociopaths. It only requires a small number of said sociopaths to get their hands on weapons of mass destruction to cause disaster. A cursory reading of history (and current events!) confirms this.
“If your only goal is “curing cancer”, and you lack humans’ instinct for the thousands of other important considerations, a relatively easy solution might be to hack into a nuclear base, launch all of its missiles, and kill everyone in the world.”
One problem I have with these scenarios is that they always rely on some lethal means for the AGI to actually kill people. And those lethal means are also available to humans, of course. If it’s possible for an AGI to ‘simply’ hack into a nuclear base and launch all it’s missiles, it’s possible for a human to do the same—possibly using AI to assist themselves. I would wager that it’s many orders of magnitude more likely for a human to do this, given our long history of killing each other. Therefore a world in which AGI can actually kill all of us, is a world in which a rogue human can as well. It feels to me that we’re kind of worrying about the wrong thing in these scenarios—we are the bigger threat.
Yes. Rogue AGI is scary, but I’m far more concerned about human misuse of AGI. Though in the end, there may not be that much of a distinction.
Some humans care about humans. Some humans are mass murdering sociopaths. It only requires a small number of said sociopaths to get their hands on weapons of mass destruction to cause disaster. A cursory reading of history (and current events!) confirms this.