However, the moment it starts executing a prompt that requires it to exhibit agency or goal directed behavior, it will.
This seems to make a jump from “the prompt requires agency to execute well” to “the AI develops the cognitive capability for agency”?
I read Sarah’s point as being that current AIs are fundamentally incapable of having agency (as she defines it). If that’s the case, it doesn’t matter if the prompt requires the AI to have agency to execute the prompt well: instead, the AI will just fail to execute the prompt well.
This seems to make a jump from “the prompt requires agency to execute well” to “the AI develops the cognitive capability for agency”?
In my scenario the AI already has the cognitive capability for agency. It’s just that the capability is latent until the right prompt causes it to be expressed. We’ve seen early examples of this with ChatGPT, where, if you ask it to plan something or think about adversarial scenarios, it will demonstrate agent-ish behavior.
My point is that while current AIs are probably incapable of having agency, future AIs probably will have that capability. Furthermore, we may not be able to tell the difference between an AI that is capable of building a world-model and engaging in long-term goal directed behavior and the current AI systems that mostly aren’t.
This seems to make a jump from “the prompt requires agency to execute well” to “the AI develops the cognitive capability for agency”?
I read Sarah’s point as being that current AIs are fundamentally incapable of having agency (as she defines it). If that’s the case, it doesn’t matter if the prompt requires the AI to have agency to execute the prompt well: instead, the AI will just fail to execute the prompt well.
In my scenario the AI already has the cognitive capability for agency. It’s just that the capability is latent until the right prompt causes it to be expressed. We’ve seen early examples of this with ChatGPT, where, if you ask it to plan something or think about adversarial scenarios, it will demonstrate agent-ish behavior.
My point is that while current AIs are probably incapable of having agency, future AIs probably will have that capability. Furthermore, we may not be able to tell the difference between an AI that is capable of building a world-model and engaging in long-term goal directed behavior and the current AI systems that mostly aren’t.