That’s the crux, and also in the fact that even with zero practical use, you still instantly develop some ideas about where you can use the new definition. Yes, to properly master it to the level where you apply it instinctively, you need to chisel it into your own heuristics engine, because humans are LLM-like in places, and perhaps even most of the time. But there’s also a higher-level component on the top that actually “groks” unfamiliar terms even if they’ve just been invented right this moment, can put them in the context of the rest of the world-model, and autonomously figure out how to use them.
Its contributions are oftentimes small or subtle, but they’re crucial. It’s why e. g. LLMs only produce stellar results if they’re “steered” by a human — even if the steerer barely intervenes, only making choices between continuations instead of writing-in custom text.
I expect you underestimate how much transfer there is, or how bad “no transfer” actually looks like.
I mean, its not clear to me there’s zero transfer with LLMs either. At least one person (github page linked in case Twitter/X makes this difficult to access) claims to get non-zero transfer with a basic transformer model. Though I haven’t looked super closely at their results or methods.
Added: Perhaps no current LLM has nonzero transfer. In which case, in light of the above results, I’d guess that this fact will go away with scale, mostly at a time uncorrelated with ASI (self-modifications made by the ASI ignored. Obviously the ASI will self-modify to be better at this task if it can. My point here is to say that the requirements needed to implement this are not necessary or sufficient for ASI), which I anticipate would be against your model.
Added2:
I expect you underestimate how much transfer there is, or how bad “no transfer” actually looks like.
I’m happy to give some numbers here, but this likely depends a lot on context, like how much the human knows about the subject, maybe the age of the humans (older probably less likely to invert, ages where plasticity is high probably more likely, if new to language then less likely), and the subject itself. I think when I memorized the Greek letters I had a transfer rate of about 20%, so lets say my 50% confidence interval is like 15-35%. Using a bunch of made up numbers I get
P(0 <= % transfer < 0.1)=0.051732
P(0.1 <= % transfer < 0.2)=0.16171
P(0.2 <= % transfer < 0.3)=0.2697
P(0.3 <= % transfer < 0.4)=0.20471
P(0.4 <= % transfer < 0.5)=0.11631
P(0.5 <= % transfer < 0.6)=0.050871
P(0.6 <= % transfer < 0.7)=0.035978
P(0.7 <= % transfer < 0.8)=0.034968
P(0.8 <= % transfer < 0.9)=0.035175
P(0.9 <= % transfer <= 1)=0.038851
for my probability distribution. Some things here seem unreasonable to me, but its an ok start to giving hard numbers.
I think it’s wrong to say that LLMs fundamentally cannot do that. I think LLMs do do that, they just do it poorly. So poorly compared to humans that it’s tempting to round their ability to do this down to zero. The difference between near-zero and zero is a really important difference though.
I have been working a lot with LLMs over the past couple years doing AI alignment research full-time, and I have the strong impression that LLMs do a worse job of concept generalization and transfer than humans. Worse, but still non-zero. They do some. This is why I believe that current 2023 LLMs aren’t so great at general reasoning, but that they’ve noticeably improved over ~2021 era LLMs. I think further development and scale of LLMs is a very inefficient way to AGI, but nevertheless will get us there if we don’t come up with a more efficient way first. And unfortunately, I suspect that there are specific algorithmic improvements available to be discovered which will greatly improve efficiency at this specific generalization skill.
It wasn’t my intention to respond to your comment specifically, but rather to add to the thread generally. But yes, I suppose since my comment was directed at Thane then it would make sense to place this as a response to his comment so that he receives the notification about it. I’m not too worried about this though, since neither Thane nor you are my intended recipients of my comment, but rather I speak to the general mass of readers who might come across this thread.
That’s the crux, and also in the fact that even with zero practical use, you still instantly develop some ideas about where you can use the new definition. Yes, to properly master it to the level where you apply it instinctively, you need to chisel it into your own heuristics engine, because humans are LLM-like in places, and perhaps even most of the time. But there’s also a higher-level component on the top that actually “groks” unfamiliar terms even if they’ve just been invented right this moment, can put them in the context of the rest of the world-model, and autonomously figure out how to use them.
Its contributions are oftentimes small or subtle, but they’re crucial. It’s why e. g. LLMs only produce stellar results if they’re “steered” by a human — even if the steerer barely intervenes, only making choices between continuations instead of writing-in custom text.
I expect you underestimate how much transfer there is, or how bad “no transfer” actually looks like.
I mean, its not clear to me there’s zero transfer with LLMs either. At least one person (github page linked in case Twitter/X makes this difficult to access) claims to get non-zero transfer with a basic transformer model. Though I haven’t looked super closely at their results or methods.
Added: Perhaps no current LLM has nonzero transfer. In which case, in light of the above results, I’d guess that this fact will go away with scale, mostly at a time uncorrelated with ASI (self-modifications made by the ASI ignored. Obviously the ASI will self-modify to be better at this task if it can. My point here is to say that the requirements needed to implement this are not necessary or sufficient for ASI), which I anticipate would be against your model.
Added2:
I’m happy to give some numbers here, but this likely depends a lot on context, like how much the human knows about the subject, maybe the age of the humans (older probably less likely to invert, ages where plasticity is high probably more likely, if new to language then less likely), and the subject itself. I think when I memorized the Greek letters I had a transfer rate of about 20%, so lets say my 50% confidence interval is like 15-35%. Using a bunch of made up numbers I get
for my probability distribution. Some things here seem unreasonable to me, but its an ok start to giving hard numbers.
I think it’s wrong to say that LLMs fundamentally cannot do that. I think LLMs do do that, they just do it poorly. So poorly compared to humans that it’s tempting to round their ability to do this down to zero. The difference between near-zero and zero is a really important difference though.
I have been working a lot with LLMs over the past couple years doing AI alignment research full-time, and I have the strong impression that LLMs do a worse job of concept generalization and transfer than humans. Worse, but still non-zero. They do some. This is why I believe that current 2023 LLMs aren’t so great at general reasoning, but that they’ve noticeably improved over ~2021 era LLMs. I think further development and scale of LLMs is a very inefficient way to AGI, but nevertheless will get us there if we don’t come up with a more efficient way first. And unfortunately, I suspect that there are specific algorithmic improvements available to be discovered which will greatly improve efficiency at this specific generalization skill.
I think you responded to the wrong comment.
It wasn’t my intention to respond to your comment specifically, but rather to add to the thread generally. But yes, I suppose since my comment was directed at Thane then it would make sense to place this as a response to his comment so that he receives the notification about it. I’m not too worried about this though, since neither Thane nor you are my intended recipients of my comment, but rather I speak to the general mass of readers who might come across this thread.
https://www.lesswrong.com/posts/Wr7N9ji36EvvvrqJK/response-to-quintin-pope-s-evolution-provides-no-evidence?commentId=FLCpkJHWyqoWZG67B