I haven’t seen these terms consistently defined with respect to LLMs. I’ve been using, and propose standardizing on:
Tooling: affordances for LLMs to make calls, eg ChatGPT plugins.
Scaffolding: an outer process that calls LLMs, where the bulk of the intelligence comes from the called LLMs, eg AutoGPT.
Some smaller details:
If the scaffolding itself becomes as sophisticated as the LLMs it calls (or more so), we should start focusing on the system as a whole rather than just describing it as a scaffolded LLM.
This terminology is relative to a particular LLM. In a complex system (eg a treelike system with code calling LLMs calling code calling LLMs calling...), some particular component can be tooling relative to the LLM above it, and scaffolding relative to the LLM below.
It’s reasonable to think of a system as scaffolded if the outermost layer is a scaffolding layer.
There’s are other possible categories that don’t fit this as neatly, eg LLMs calling each other as peers without a main outer process, but I expect these definitions to cover most real-world cases.
Thanks to @Andy Arditi for helping me nail down the distinction.
There will be some overlap where components are both tools and part of the scaffolding; for instance, a competent language model cognitive architecture would probably have a “tool” of an episodic memory it can write to and read from; and the next LLM call from the scaffolding portion would often be determined by the result of reading from that episodic memory “tool”, making it also part of the scaffolding. Similarly with sensory systems and probably many others; how the scaffolding invokes the LLM will depend on the results of calls to tools.
But it’s useful to have more distinct terminology even when it’s not perfect, so I’ll go ahead and use this.
I might think of some of your specific examples a bit differently, but yeah, I would say that a particular component can be tooling relative to the LLM above it, and scaffolding relative to the LLM below. I’ll add some clarification to the post, thanks!
This seems to be inspired by the library/framework distinction in software engineering:
Inversion of Control is a key part of what makes a framework different to a library. A library is essentially a set of functions that you can call, these days usually organized into classes. Each call does some work and returns control to the client.
A framework embodies some abstract design, with more behavior built in. In order to use it you need to insert your behavior into various places in the framework either by subclassing or by plugging in your own classes. The framework’s code then calls your code at these points. (source)
Your code calls the library; the framework calls your code ≈ The LLM calls the tool; the scaffolding calls the LLM.
Not consciously an inspiration for me, but definitely a similar idea, and applies pretty cleanly to a system with only a couple of layers; I’ll add some clarification to the post re more complex systems where that analogy might not hold quite as well.
Terminology proposal: scaffolding vs tooling.
I haven’t seen these terms consistently defined with respect to LLMs. I’ve been using, and propose standardizing on:
Tooling: affordances for LLMs to make calls, eg ChatGPT plugins.
Scaffolding: an outer process that calls LLMs, where the bulk of the intelligence comes from the called LLMs, eg AutoGPT.
Some smaller details:
If the scaffolding itself becomes as sophisticated as the LLMs it calls (or more so), we should start focusing on the system as a whole rather than just describing it as a scaffolded LLM.
This terminology is relative to a particular LLM. In a complex system (eg a treelike system with code calling LLMs calling code calling LLMs calling...), some particular component can be tooling relative to the LLM above it, and scaffolding relative to the LLM below.
It’s reasonable to think of a system as scaffolded if the outermost layer is a scaffolding layer.
There’s are other possible categories that don’t fit this as neatly, eg LLMs calling each other as peers without a main outer process, but I expect these definitions to cover most real-world cases.
Thanks to @Andy Arditi for helping me nail down the distinction.
Sure, I’ll use this terminology.
There will be some overlap where components are both tools and part of the scaffolding; for instance, a competent language model cognitive architecture would probably have a “tool” of an episodic memory it can write to and read from; and the next LLM call from the scaffolding portion would often be determined by the result of reading from that episodic memory “tool”, making it also part of the scaffolding. Similarly with sensory systems and probably many others; how the scaffolding invokes the LLM will depend on the results of calls to tools.
But it’s useful to have more distinct terminology even when it’s not perfect, so I’ll go ahead and use this.
I might think of some of your specific examples a bit differently, but yeah, I would say that a particular component can be tooling relative to the LLM above it, and scaffolding relative to the LLM below. I’ll add some clarification to the post, thanks!
This seems to be inspired by the library/framework distinction in software engineering:
Your code calls the library; the framework calls your code ≈ The LLM calls the tool; the scaffolding calls the LLM.
Not consciously an inspiration for me, but definitely a similar idea, and applies pretty cleanly to a system with only a couple of layers; I’ll add some clarification to the post re more complex systems where that analogy might not hold quite as well.
I’m writing a page for AIsafety.info on scaffolding, and was struggling to find a principled definition. Thank you for this!
Oh great, I’m glad it helped!