About your opinion on LLMs probably not scaling to general intelligence:
What if the language of thought hypothesis is correct, human intelligence can be represented as rules that manipulate natural language, the context window of LLMs is going to become long enough to match a human’s “context window”, and LLM training is able to find the algorithm?
How does this view fits into your model? What probabilities do you assign to the various steps?
language of thought hypothesis is correct
language of thought close enough to natural language
context window becomes long enough
transformers (or successor alternatives) do have the algorithm in their search space
I do think human thought can be represented as language-manipulation rules, but that’s not a very interesting claim. Natural language is Turing-complete, of course anything can be approximated as rules for manipulating it. The same is true for chains of NAND gates. p→1.
I don’t think it’s close to natural language in the meaningful sense. E. g., you can in fact think using raw abstractions, without an inner monologue, and it’s much faster (= less compute-intensive) in some ways. I expect that’s how we actually think, and the more legible inner monologue is more like a trick we’re using to be able to convey our thoughts to other humans on the fly. A communication tool, not a cognition tool. Trying to use it for actual cognition will be ruinously compute-intensive. p>0.98.
“Is the context window long enough?” seems like the wrong way to think about it. If we’re to draw a human analogue, the context window would mirror working memory, and in this case, I expect it’s already more “roomy” than human working memory (in some sense). The issue is that LLMs can’t update their long-term memory (and no, on-line training ain’t the answer to it). If we’re limited to using the context window, then its length would have to be equivalent to a human’s life… In which case, sure, interesting things may happen in an LLM scaled so far, but this seems obviously computationally intractable. p→1.
Inasmuch as NNs can approximate any continuous function (and chain-of-thought prompting can allow arbitrary-depth recursion) — sure, transformers have general intelligence in their search-space, p→1.
… but the current training schemes, or any obvious tweaks to them, won’t be able to find it. This one I’m actually uncertain about, p≈0.7.
I don’t think it’s close to natural language in the meaningful sense. E. g., you can in fact think using raw abstractions, without an inner monologue, and it’s much faster (= less compute-intensive) in some ways. I expect that’s how we actually think, and the more legible inner monologue is more like a trick we’re using to be able to convey our thoughts to other humans on the fly. A communication tool, not a cognition tool. Trying to use it for actual cognition will be ruinously compute-intensive. p>0.98.
I know very logorrheic people who assert to think mostly verbally. Personally, I do a small amount of verbal thought, but sometimes resort to explicit verbal thinking on purpose to tackle problems I’m confused about. I think it would be sufficient that there exist some people who mostly reason verbally for the thesis to be valid for the purpose of guessing if LLMs are a viable path to intelligence. Do you think that even the most verbally-tuned people are actually doing the heavy lifting of their high-level thinking wordlessly?
“Is the context window long enough?” seems like the wrong way to think about it. If we’re to draw a human analogue, the context window would mirror working memory, and in this case, I expect it’s already more “roomy” than human working memory (in some sense). The issue is that LLMs can’t update their long-term memory (and no, on-line training ain’t the answer to it). If we’re limited to using the context window, then its length would have to be equivalent to a human’s life… In which case, sure, interesting things may happen in an LLM scaled so far, but this seems obviously computationally intractable. p→1.
I expect that “plug-ins” that give a memory to the LLM, as people are already trying to develop, are viable. Do you expect otherwise? (Although they would not allow the LLM to learn new “instincts”.)
Do you think that even the most verbally-tuned people are actually doing the heavy lifting of their high-level thinking wordlessly?
Yes. It’s a distinction similar to whatever computations happen in LLM forward-passes vs. the way Auto-GPT exchanges messages with its subagents. Maybe it’s also a memory aid, such that memorizing the semantic representation of a thought serves as a shortcut to the corresponding mental state; but it’s not the real nuts-and-bolts of cognition. The heavily lifting is done by whatever process figures out what word to put next in the monologue; not by the inner monologue itself.
I expect that “plug-ins” that give a memory to the LLM, as people are already trying to develop, are viable. Do you expect otherwise? (Although they would not allow the LLM to learn new “instincts”.)
I think the instincts are the more crucial part, yes; perhaps I should’ve said “long-term adaptation” rather than “long-term memory”.
I do suspect the current training processes fundamentally shape LLMs’ architecture the wrong way, and not in a way that’s easy to fix with fine-tuning, or conceptually-small architectural adjustments, or plug-ins. But that’s my weakest claim, the one I’m only ~70% confident it. We’ll see, I suppose.
The heavily lifting is done by whatever process figures out what word to put next in the monologue; not by the inner monologue itself.
It seems you use “monologue” in this sentence to refer to the sequence of words only, and then say that of course the monologue is not the cognition. With this I agree, but I don’t think that’s the correct interpretation of the combo “language of thought hypothesis” + “language of thought close to natural language”. Having a “language of thought” means that there is a linear stream of items, and that your abstract cognition works only by applying some algorithm to the stream buffer to append the next item. The tape is not the cognition, but the cognition can be seen as acting (almost) only on the tape. Then “language of thought close to natural language” means that the language of thought has a short encoding in natural language. You can picture this as the language of thought of a verbal thinker being a more abstract version of natural language, similarly to when you feel what to say next but lack the word.
cognition can be seen as acting (almost) only on the tape
… If not for the existence of non-verbal cognition, which works perfectly well even without a “tape”. Suggesting that the tape isn’t a crucial component, that the heavy lifting can be done by the abstract algorithm alone, and therefore that even in supposed verbal thinkers, that algorithm is likely what’s doing the actual heavy lifting.
In my view, there’s an actual stream of abstract cognition, and a “translator” function mapping from that stream to human language. When we’re doing verbal thinking, we’re constantly running the translator on our actual cognition, which has various benefits (e. g., it’s easier to translate our thoughts to other humans); but the items in the natural-language monologue are compressed versions of the items in the abstract monologue, and they’re strictly downstream of the abstract stream.
There’s a “stream” of abstract thought, or “abstract monologue”
The cognition algorithm operates on/produces the abstract stream
Natural language is a compressed stream of the abstract stream
Which seems to me the same thing I said above, unless maybe you are also implying either or both of these additional statements:
a) The abstract cognition algorithm can not be seen as operating mostly autoregressively on its “abstract monologue”;
b) The abstract monologue can not be translated to a longer, but boundedly longer, natural language stream (without claiming that this is what happens typically when someone verbalizes).
Which of (a), (b) do you endorse, eventually with amendments?
Which of (a), (b) do you endorse, eventually with amendments?
I don’t necessarily endorse either. But “boundedly longer” is what does a lot of work there. As I’d mentioned, cognition can also be translated into a finitely long sequence of NAND gates. The real question isn’t “is there a finitely-long translation?”, but how much longer that translation is.
And I’m not aware of any strong evidence suggesting that natural language is close enough to human cognition that the resultant stream would not be much longer. Long enough to be ruinously compute-intensive (effectively as ruinous as translating it into NAND-gate sequences).
Indeed, I’d say there’s plenty of evidence to the contrary, given how central miscommunication is to the human experience.
About your opinion on LLMs probably not scaling to general intelligence:
What if the language of thought hypothesis is correct, human intelligence can be represented as rules that manipulate natural language, the context window of LLMs is going to become long enough to match a human’s “context window”, and LLM training is able to find the algorithm?
How does this view fits into your model? What probabilities do you assign to the various steps?
language of thought hypothesis is correct
language of thought close enough to natural language
context window becomes long enough
transformers (or successor alternatives) do have the algorithm in their search space
training finds the algorithm
I do think human thought can be represented as language-manipulation rules, but that’s not a very interesting claim. Natural language is Turing-complete, of course anything can be approximated as rules for manipulating it. The same is true for chains of NAND gates. p→1.
I don’t think it’s close to natural language in the meaningful sense. E. g., you can in fact think using raw abstractions, without an inner monologue, and it’s much faster (= less compute-intensive) in some ways. I expect that’s how we actually think, and the more legible inner monologue is more like a trick we’re using to be able to convey our thoughts to other humans on the fly. A communication tool, not a cognition tool. Trying to use it for actual cognition will be ruinously compute-intensive. p>0.98.
“Is the context window long enough?” seems like the wrong way to think about it. If we’re to draw a human analogue, the context window would mirror working memory, and in this case, I expect it’s already more “roomy” than human working memory (in some sense). The issue is that LLMs can’t update their long-term memory (and no, on-line training ain’t the answer to it). If we’re limited to using the context window, then its length would have to be equivalent to a human’s life… In which case, sure, interesting things may happen in an LLM scaled so far, but this seems obviously computationally intractable. p→1.
Inasmuch as NNs can approximate any continuous function (and chain-of-thought prompting can allow arbitrary-depth recursion) — sure, transformers have general intelligence in their search-space, p→1.
… but the current training schemes, or any obvious tweaks to them, won’t be able to find it. This one I’m actually uncertain about, p≈0.7.
I know very logorrheic people who assert to think mostly verbally. Personally, I do a small amount of verbal thought, but sometimes resort to explicit verbal thinking on purpose to tackle problems I’m confused about. I think it would be sufficient that there exist some people who mostly reason verbally for the thesis to be valid for the purpose of guessing if LLMs are a viable path to intelligence. Do you think that even the most verbally-tuned people are actually doing the heavy lifting of their high-level thinking wordlessly?
I expect that “plug-ins” that give a memory to the LLM, as people are already trying to develop, are viable. Do you expect otherwise? (Although they would not allow the LLM to learn new “instincts”.)
Yes. It’s a distinction similar to whatever computations happen in LLM forward-passes vs. the way Auto-GPT exchanges messages with its subagents. Maybe it’s also a memory aid, such that memorizing the semantic representation of a thought serves as a shortcut to the corresponding mental state; but it’s not the real nuts-and-bolts of cognition. The heavily lifting is done by whatever process figures out what word to put next in the monologue; not by the inner monologue itself.
I think the instincts are the more crucial part, yes; perhaps I should’ve said “long-term adaptation” rather than “long-term memory”.
I do suspect the current training processes fundamentally shape LLMs’ architecture the wrong way, and not in a way that’s easy to fix with fine-tuning, or conceptually-small architectural adjustments, or plug-ins. But that’s my weakest claim, the one I’m only ~70% confident it. We’ll see, I suppose.
It seems you use “monologue” in this sentence to refer to the sequence of words only, and then say that of course the monologue is not the cognition. With this I agree, but I don’t think that’s the correct interpretation of the combo “language of thought hypothesis” + “language of thought close to natural language”. Having a “language of thought” means that there is a linear stream of items, and that your abstract cognition works only by applying some algorithm to the stream buffer to append the next item. The tape is not the cognition, but the cognition can be seen as acting (almost) only on the tape. Then “language of thought close to natural language” means that the language of thought has a short encoding in natural language. You can picture this as the language of thought of a verbal thinker being a more abstract version of natural language, similarly to when you feel what to say next but lack the word.
… If not for the existence of non-verbal cognition, which works perfectly well even without a “tape”. Suggesting that the tape isn’t a crucial component, that the heavy lifting can be done by the abstract algorithm alone, and therefore that even in supposed verbal thinkers, that algorithm is likely what’s doing the actual heavy lifting.
In my view, there’s an actual stream of abstract cognition, and a “translator” function mapping from that stream to human language. When we’re doing verbal thinking, we’re constantly running the translator on our actual cognition, which has various benefits (e. g., it’s easier to translate our thoughts to other humans); but the items in the natural-language monologue are compressed versions of the items in the abstract monologue, and they’re strictly downstream of the abstract stream.
So you think
There’s a “stream” of abstract thought, or “abstract monologue”
The cognition algorithm operates on/produces the abstract stream
Natural language is a compressed stream of the abstract stream
Which seems to me the same thing I said above, unless maybe you are also implying either or both of these additional statements:
a) The abstract cognition algorithm can not be seen as operating mostly autoregressively on its “abstract monologue”;
b) The abstract monologue can not be translated to a longer, but boundedly longer, natural language stream (without claiming that this is what happens typically when someone verbalizes).
Which of (a), (b) do you endorse, eventually with amendments?
I don’t necessarily endorse either. But “boundedly longer” is what does a lot of work there. As I’d mentioned, cognition can also be translated into a finitely long sequence of NAND gates. The real question isn’t “is there a finitely-long translation?”, but how much longer that translation is.
And I’m not aware of any strong evidence suggesting that natural language is close enough to human cognition that the resultant stream would not be much longer. Long enough to be ruinously compute-intensive (effectively as ruinous as translating it into NAND-gate sequences).
Indeed, I’d say there’s plenty of evidence to the contrary, given how central miscommunication is to the human experience.