An argument for danger of human-directed misuse doesn’t work as an argument against dangers of AI-directed agentic activity.
I agree. But I was not trying to argue against dangers of AI-directed agentic activity. The thesis is not that “alignment risk” is overblown, nor is the comparison of the risks the point, it’s that those risks accumulate such that the technology is guaranteed to be lethal for the average person. This is significant because the risk of misalignment is typically thought to be accepted because of rewards that will be broadly shared. “You or your children are likely to be killed by this technology, whether it works as designed or not” is a very different story from “there is a chance this will go badly for everyone, but if it doesn’t it will be really great for everyone.”
I agree. But I was not trying to argue against dangers of AI-directed agentic activity. The thesis is not that “alignment risk” is overblown, nor is the comparison of the risks the point, it’s that those risks accumulate such that the technology is guaranteed to be lethal for the average person. This is significant because the risk of misalignment is typically thought to be accepted because of rewards that will be broadly shared. “You or your children are likely to be killed by this technology, whether it works as designed or not” is a very different story from “there is a chance this will go badly for everyone, but if it doesn’t it will be really great for everyone.”
That’s an excellent summary sentence. It seems like that would be a useful statement in advocating for AI slowdown/shutdown.