I have gained confidence in my position that all of this happening now is a good thing, both from the perspective of smaller risks like malware attacks, and from the perspective of potential existential threats. Seems worth going over the logic.
What we want to do is avoid what one might call an agent overhang.
One might hope to execute our Plan A of having our AIs not be agents. Alas, even if technically feasible (which is not at all clear) that only can work if we don’t intentionally turn them into agents via wrapping code around them. We’ve checked with actual humans about the possibility of kindly not doing that. Didn’t go great.
This seems like really bad reasoning…
It seems like the evidence that people won’t “kindly not [do] that” is… AutoGPT. So if AutoGPT didn’t exist, you might be able to say: “we asked people to not turn AI systems into agents, and they didn’t. Hooray for plan A!”
Also: I don’t think it’s fair to say “we’ve checked [...] about the possibility”. The AI safety community thought it was sketch for a long time, and has provided some lackluster pushback. Governance folks from the community don’t seem to be calling for a rollback of the plugins, or bans on this kind of behavior, etc.
Your concern is certainly valid—blindly assuming taking action to be beneficial misses the mark. It’s often far better to refrain from embracing disruptive technologies simply to appear progressive. Thinking of ways to ensure people will not promote AI for the sole sake of causing agent overhang is indeed crucial for reducing potential existential threats. Fearlessly rejecting risky technologies is often better than blindly accepting them. With that mindset, encouraging users to explore AutoGPT and other agent-based systems is potentially problematic. Instead, focusing on developing strategies for limiting the potentially dangerous aspects of such creations should take center stage.
This seems like really bad reasoning…
It seems like the evidence that people won’t “kindly not [do] that” is… AutoGPT.
So if AutoGPT didn’t exist, you might be able to say: “we asked people to not turn AI systems into agents, and they didn’t. Hooray for plan A!”
Also: I don’t think it’s fair to say “we’ve checked [...] about the possibility”. The AI safety community thought it was sketch for a long time, and has provided some lackluster pushback. Governance folks from the community don’t seem to be calling for a rollback of the plugins, or bans on this kind of behavior, etc.
Your concern is certainly valid—blindly assuming taking action to be beneficial misses the mark. It’s often far better to refrain from embracing disruptive technologies simply to appear progressive. Thinking of ways to ensure people will not promote AI for the sole sake of causing agent overhang is indeed crucial for reducing potential existential threats. Fearlessly rejecting risky technologies is often better than blindly accepting them. With that mindset, encouraging users to explore AutoGPT and other agent-based systems is potentially problematic. Instead, focusing on developing strategies for limiting the potentially dangerous aspects of such creations should take center stage.