I don’t think the theorem is true. I think there’s some daylight between superintelligent tools and agents—just not much. Not nearly as much as, say, between an Oracle and a sovereign AI.
One thing I am not clear about is whether you are saying that a tool AI spontaneously develops what appears like intentionality or not. It sure seems that that is what you are saying, initially with a human in the feedback loop, until the suggestion to “create an AI with these motivations” is implemented. If so, then why are you saying that “there’s some daylight between superintelligent tools and agents”?
Tool AIs and Agent AIs given the same task will not lead to the same outcomes, nor result in the same routes being taken. However, they are likely dangerous in broadly similar ways, and many warnings and precautions for one of these apply also to the other.
I don’t think the theorem is true. I think there’s some daylight between superintelligent tools and agents—just not much. Not nearly as much as, say, between an Oracle and a sovereign AI.
One thing I am not clear about is whether you are saying that a tool AI spontaneously develops what appears like intentionality or not. It sure seems that that is what you are saying, initially with a human in the feedback loop, until the suggestion to “create an AI with these motivations” is implemented. If so, then why are you saying that “there’s some daylight between superintelligent tools and agents”?
Then formulate a weaker version of the theorem that you think is true.
Tool AIs and Agent AIs given the same task will not lead to the same outcomes, nor result in the same routes being taken. However, they are likely dangerous in broadly similar ways, and many warnings and precautions for one of these apply also to the other.