What if the tasks that your scaffolded LLM is doing are randomly selected pieces of cognitive labor from the full distribution of human cognitive tasks?
It seems to me like your objection is mostly to narrow distributions of tasks and scaffolding which is heavily specialized to that task.
I think narrowness of the task and amount of scaffolding might be correlated in practice, but these attributes don’t have to be related.
(You might think they are correlated because large amounts of scaffolding won’t be very useful for very diverse tasks. I think this is likely false—there exists general purpose software that I find useful for a very broad range of tasks. E.g. neovim. I agree that smart general agents should be able to build their own scaffolding and bootstrap, but its worth noting that the final system might be using a bunch of tools!)
For humans, we can consider eyes to be a type of scaffolding: they help us do various cognitive tasks by adding various affordances but are ultimately just attached.
Nonetheless, I predict that if I didn’t have eyes, I would be notably less efficient at my job.
What if the tasks that your scaffolded LLM is doing are randomly selected pieces of cognitive labor from the full distribution of human cognitive tasks?
It seems to me like your objection is mostly to narrow distributions of tasks and scaffolding which is heavily specialized to that task.
I think narrowness of the task and amount of scaffolding might be correlated in practice, but these attributes don’t have to be related.
(You might think they are correlated because large amounts of scaffolding won’t be very useful for very diverse tasks. I think this is likely false—there exists general purpose software that I find useful for a very broad range of tasks. E.g. neovim. I agree that smart general agents should be able to build their own scaffolding and bootstrap, but its worth noting that the final system might be using a bunch of tools!)
For humans, we can consider eyes to be a type of scaffolding: they help us do various cognitive tasks by adding various affordances but are ultimately just attached.
Nonetheless, I predict that if I didn’t have eyes, I would be notably less efficient at my job.