I’m interested in why the transformer architecture has been so successful and the concept of natural abstraction is useful here. I’m not thinking so much about how transformers work, not in any detail, but about the natural environment in which the architecture was originally designed to function, text. What are the natural abstractions over text?
Let’s look at this, a crucial observation:
That convergence isn’t complete yet—I think a lot of the interpretability crowd hasn’t yet fully internalized the framing of “interpretability is primarily about mapping net-internal structures to corresponding high-level interpretable structures in the environment”. In particular I think a lot of interpretability researchers have not yet internalized that mathematically understanding what kinds of high-level interpretable structures appear in the environment is a core part of the problem of interpretability.
For a worm the external environment is mostly dirt, but other worms as well. Birds, a rather different natural environment. Then we have the great apes, still a different natural environment.
Our natural environment is very much like that of the great apes, our close biological relatives. Our fellow conspecifics are a big part of that environment. But we have language as well. Language is part of our natural environment, spoken language, but text as well. The advent of writing has made it possible to immerse ourselves in a purely textual world. Arithmetic, too, is a textual world. Think of it as a very specialized form of language, with a relatively small set of primitive elements and a highly constrained syntax. And then there is program code.
You see where this is going?
For the transformer architecture, text is its natural environment. The attention mechanism allows it to get enough context so that it can isolate the natural abstractions in an environment of pure text. What’s in that environment? Language, whether spoken or written is, after all, a physical thing. Alphanumeric characters, spaces between strings of characters, punctuation, capitalization, paragraphing conventions. Stuff like that. It’s the transformer’s superior capacity to abstract over that environment that has made it so successful.
(I note, as an aside, that speech is a physically complex and requires its own natural abstractions for perception. Phoneticians study the physical features of the soundstream itself while phonologists study those aspects of the signal that are linguistically relevant. They’re thus looking for the natural abstractions over the physical signal itself. When put online, the physical text is extraordinarily simple – strings of ASCII characters – so so requires no sophisticated perceptual mechanisms.)
Note, of course, that text is text. Ultimately textual meanings have to be grounded in the physical world. No matter how much text a transformer consumes, it’s unlikely to substitute for direct access to the natural world. So the architecture can’t do everything, but it can do a lot.
I’ve recently written two posts examining the output of ChatGPT for higher-level discourse structure, mostly the conventions of dialog. It’s simple stuff, obvious in a way. But it has all but convincement that GPT I is picking up a primitive kind of discourse grammar that is separate from and independent from the word-by-word grammar of individual sentences. I can’t see how it would be able to produce such fluent text if it weren’t doing that.
I’m interested in why the transformer architecture has been so successful and the concept of natural abstraction is useful here. I’m not thinking so much about how transformers work, not in any detail, but about the natural environment in which the architecture was originally designed to function, text. What are the natural abstractions over text?
Let’s look at this, a crucial observation:
For a worm the external environment is mostly dirt, but other worms as well. Birds, a rather different natural environment. Then we have the great apes, still a different natural environment.
Our natural environment is very much like that of the great apes, our close biological relatives. Our fellow conspecifics are a big part of that environment. But we have language as well. Language is part of our natural environment, spoken language, but text as well. The advent of writing has made it possible to immerse ourselves in a purely textual world. Arithmetic, too, is a textual world. Think of it as a very specialized form of language, with a relatively small set of primitive elements and a highly constrained syntax. And then there is program code.
You see where this is going?
For the transformer architecture, text is its natural environment. The attention mechanism allows it to get enough context so that it can isolate the natural abstractions in an environment of pure text. What’s in that environment? Language, whether spoken or written is, after all, a physical thing. Alphanumeric characters, spaces between strings of characters, punctuation, capitalization, paragraphing conventions. Stuff like that. It’s the transformer’s superior capacity to abstract over that environment that has made it so successful.
(I note, as an aside, that speech is a physically complex and requires its own natural abstractions for perception. Phoneticians study the physical features of the soundstream itself while phonologists study those aspects of the signal that are linguistically relevant. They’re thus looking for the natural abstractions over the physical signal itself. When put online, the physical text is extraordinarily simple – strings of ASCII characters – so so requires no sophisticated perceptual mechanisms.)
Note, of course, that text is text. Ultimately textual meanings have to be grounded in the physical world. No matter how much text a transformer consumes, it’s unlikely to substitute for direct access to the natural world. So the architecture can’t do everything, but it can do a lot.
I’ve recently written two posts examining the output of ChatGPT for higher-level discourse structure, mostly the conventions of dialog. It’s simple stuff, obvious in a way. But it has all but convincement that GPT I is picking up a primitive kind of discourse grammar that is separate from and independent from the word-by-word grammar of individual sentences. I can’t see how it would be able to produce such fluent text if it weren’t doing that.
The posts:
Of pumpkins, the Falcon Heavy, and Groucho Marx: High-Level discourse structure in ChatGPT
High level discourse structure in ChatGPT: Part 2 [Quasi-symbolic?]