Yes, sounds right to me. It’s also true that one of the big unproven assumptions here is that we could create an AI strong enough to build such a tool, but too weak to hack humans. I find it plausible, personally, but I don’t yet have an easy-to-communicate argument for it.
I don’t know of a reason we couldn’t do this with a narrow AI. I have no idea how, but it’s possible in principle so far as I know. If anyone can figure out how, they could plausibly execute the pivotal act described above, which would be a very good thing for humanity’s chances of survival.
EDIT: Needless to say, but I’ll say it anyway: Doing this via narrow AI is vastly preferable to using a general AI. It’s both much less risky and means you don’t have to expend an insane amount of effort on checking.
Yes, sounds right to me. It’s also true that one of the big unproven assumptions here is that we could create an AI strong enough to build such a tool, but too weak to hack humans. I find it plausible, personally, but I don’t yet have an easy-to-communicate argument for it.
Why can’t a narrow AI (maybe like Drexler’s proposal) create the tool safely?
I don’t know of a reason we couldn’t do this with a narrow AI. I have no idea how, but it’s possible in principle so far as I know. If anyone can figure out how, they could plausibly execute the pivotal act described above, which would be a very good thing for humanity’s chances of survival.
EDIT: Needless to say, but I’ll say it anyway: Doing this via narrow AI is vastly preferable to using a general AI. It’s both much less risky and means you don’t have to expend an insane amount of effort on checking.