I see what you mean, but I meant oracle-like in the sense of my recollection of Nick Bostrom’s usage in Superintelligence. E.g. an AI that only answers questions and does not act. In some sense, it’s how much it’s not an agent.
It does seem to me, that pretrained LLM’s are not very agent-like by default. They are by default currently constrained to question answering. Although it’s changing fast with things like toolformer.
Even the stuff LLMs do, like inner-monologue, which seem to be transparent, are actually just more Bayesian meta-RL agentic behavior, where the inner-monologue is a mish-mash of amortized computation and task location where the model is flexibly using the roleplay as hints rather than what everyone seems to think it does, which is turn into a little Turing machine mindlessly executing instructions (hence eg. the ability to distill inner-monologue into the forward pass, or insert errors into few-shot examples or the monologue and still get correct answers).
It kind of sounds like you are saying that they have a lot of agentic capability, but they are hampered by the lack of memory/planning. If your description here is predictive, then it seems there may be a lot of low hanging agentic behaviour that can be unlocked fairly easily. Like many other things with LLM’s, we just need to “ask it properly”. Perhaps using some standard RL techniques like world models.
Do you see the properties/danger of LLM’s changing once we start using RL to make them into proper agents (not just the few-step chat)?
I see what you mean, but I meant oracle-like in the sense of my recollection of Nick Bostrom’s usage in Superintelligence. E.g. an AI that only answers questions and does not act. In some sense, it’s how much it’s not an agent.
It does seem to me, that pretrained LLM’s are not very agent-like by default. They are by default currently constrained to question answering. Although it’s changing fast with things like toolformer.
It kind of sounds like you are saying that they have a lot of agentic capability, but they are hampered by the lack of memory/planning. If your description here is predictive, then it seems there may be a lot of low hanging agentic behaviour that can be unlocked fairly easily. Like many other things with LLM’s, we just need to “ask it properly”. Perhaps using some standard RL techniques like world models.
Do you see the properties/danger of LLM’s changing once we start using RL to make them into proper agents (not just the few-step chat)?