Thank you for the reply. I agree we should try and avoid AI taking over the world.
On “doom through normal means”—I just think there are very plausibly limits to what superintelligence can do. “Persuasion, hacking, and warfare” (appreciate this is not a full version of the argument) don’t seem like doom to me. I don’t believe something can persuade generals to go to war in a short period of time, just because it’s very intelligent. Reminds me of this.
On values—I think there’s a conflation between us having ambitious goals, and whatever is actually being optimized by the AI. I am curious to hear what the “galaxy brained reasons” are; my impression was, they are what was outlined (and addressed) in the original post.
a superintelligence will be at least several orders of magnitude more persuasive than character.ai or Stuart Armstrong.
Believing this seems central to believing high P(doom).
But, I think it’s not a coherent enough concept to justify believing it. Yes, some people are far more persuasive than others. But how can you extrapolate that far beyond the distribution we obverse in humans? I do think AI will prove to better than humans at this, and likely muchbetter.
But “much” better isn’t the same as “better enough to be effectively treated as magic”.
Well, even the tail of the human distribution is pretty scary. A single human with a lot of social skills can become the leader of a whole nation, or even a prophet considered literally a divine being. This has already happened several times in history, even in times where you had to be physically close to people to convince them.
On doom through normal means: “Persuasion, hacking, and warfare” aren’t by themselves doom, but they can be used to accumulate lots of power, and then that power can be used to cause doom. Imagine a world in which human are completely economically, militarily, and politically obsolete, thanks to armies of robots directed by superintelligent AIs. Such a world could and would do very nasty things to humans (e.g. let them all starve to death) unless the superintelligent AIs managing everything specifically cared about keeping humans alive and in good living conditions. Because keeping humans alive & in good living conditions would, ex hypothesi, not be instrumentally valuable to the economy, or the military, etc.
How could such a world arise? Well, if we have superintelligent AIs, they can do some hacking, persuasion, and maybe some warfare, and create that world.
How long would this process take? IDK, maybe years? Could be much less. But I wouldn’t be surprised if it takes several years, even maybe five years.
I’m not conflating those things. We have ambitious goals and are trying to get our AIs to have ambitious goals—specifically we are trying to get them to have our ambitious goals. It’s not much of a stretch to imagine this going wrong, and them ending up with ambitious goals that are different from ours in various ways (even if somewhat overlapping).
Remember that persuasion from an ASI doesn’t need to look like “text-based chatting with a human.” It includes all the tools of communication available. Actually-near-flawless forgeries of any and every form of digital data you could ever ask for, as a baseline, all based on the best possible inferences made from all available real data.
How many people today are regularly persuaded of truly ridiculous things by perfectly normal human-scale-intelligent scammers, cults, conspiracy theorists, marketers, politicians, relatives, preachers, and so on? The average human, even the average IQ 120-150 human, just isn’t that resistant to persuasion in favor of untrue claims.
Thank you for the reply. I agree we should try and avoid AI taking over the world.
On “doom through normal means”—I just think there are very plausibly limits to what superintelligence can do. “Persuasion, hacking, and warfare” (appreciate this is not a full version of the argument) don’t seem like doom to me. I don’t believe something can persuade generals to go to war in a short period of time, just because it’s very intelligent. Reminds me of this.
On values—I think there’s a conflation between us having ambitious goals, and whatever is actually being optimized by the AI. I am curious to hear what the “galaxy brained reasons” are; my impression was, they are what was outlined (and addressed) in the original post.
A few things I’ve seen give pretty worrying lower bounds for how persuasive a superintelligence would be:
How it feels to have your mind hacked by an AI
The AI in a box boxes you (content warning: creepy blackmail-y acausal stuff)
Remember that a superintelligence will be at least several orders of magnitude more persuasive than character.ai or Stuart Armstrong.
Believing this seems central to believing high P(doom).
But, I think it’s not a coherent enough concept to justify believing it. Yes, some people are far more persuasive than others. But how can you extrapolate that far beyond the distribution we obverse in humans? I do think AI will prove to better than humans at this, and likely much better.
But “much” better isn’t the same as “better enough to be effectively treated as magic”.
Well, even the tail of the human distribution is pretty scary. A single human with a lot of social skills can become the leader of a whole nation, or even a prophet considered literally a divine being. This has already happened several times in history, even in times where you had to be physically close to people to convince them.
Thanks to you likewise!
On doom through normal means: “Persuasion, hacking, and warfare” aren’t by themselves doom, but they can be used to accumulate lots of power, and then that power can be used to cause doom. Imagine a world in which human are completely economically, militarily, and politically obsolete, thanks to armies of robots directed by superintelligent AIs. Such a world could and would do very nasty things to humans (e.g. let them all starve to death) unless the superintelligent AIs managing everything specifically cared about keeping humans alive and in good living conditions. Because keeping humans alive & in good living conditions would, ex hypothesi, not be instrumentally valuable to the economy, or the military, etc.
How could such a world arise? Well, if we have superintelligent AIs, they can do some hacking, persuasion, and maybe some warfare, and create that world.
How long would this process take? IDK, maybe years? Could be much less. But I wouldn’t be surprised if it takes several years, even maybe five years.
I’m not conflating those things. We have ambitious goals and are trying to get our AIs to have ambitious goals—specifically we are trying to get them to have our ambitious goals. It’s not much of a stretch to imagine this going wrong, and them ending up with ambitious goals that are different from ours in various ways (even if somewhat overlapping).
Remember that persuasion from an ASI doesn’t need to look like “text-based chatting with a human.” It includes all the tools of communication available. Actually-near-flawless forgeries of any and every form of digital data you could ever ask for, as a baseline, all based on the best possible inferences made from all available real data.
How many people today are regularly persuaded of truly ridiculous things by perfectly normal human-scale-intelligent scammers, cults, conspiracy theorists, marketers, politicians, relatives, preachers, and so on? The average human, even the average IQ 120-150 human, just isn’t that resistant to persuasion in favor of untrue claims.