@Daniel Tan raises an interesting possibility here, that LLMs are capable of general reasoning about a problem if and only if they’ve undergone grokking on that problem. In other words, grokking is just what full generalization on a topic is (Daniel please correct me if that’s a misrepresentation).
If that’s the case, my initial guess is that we’ll only see LLMs doing general reasoning on problems that are relatively small, simple (in the sense that the fully general algorithm is simple), and common in the training data, just because grokking requires such an extended amount of training. But I don’t have very clear intuition about the degree to which frontier-scale LLMs are grokking such small common problems during ordinary training.
Alternately it could be that grokking is sufficient but not necessary, so LLMs can reason in a general way about grokked problems but also about other things.
Yup, that’s basically what I think! IMO, grokking = having memorised the “underlying rules” that define the DGP, and these rules are general by definition.”Reasoning” is a loaded term that’s difficult to unpack, but I think a good working definition is “applying a set of rules to arrive at an answer”. In other words, reasoning is learning a “correct algorithm” to solve the problem. Therefore being able to reason correctly 100% of the time is equivalent to models having grokked their problem domain.
See this work, which finds that reasoning only happens through grokking. Separate work has trained models to do tree search, and found that backwards chaining circuits (a correct algorithm) emerge only through grokking. And also the seminal work on modular addition which found that correct algorithms emerge through grokking.
Note that the question of “is reasoning in natural language grokkable?” is a totally separate crux and one which I’m highly uncertain about.
@Daniel Tan raises an interesting possibility here, that LLMs are capable of general reasoning about a problem if and only if they’ve undergone grokking on that problem. In other words, grokking is just what full generalization on a topic is (Daniel please correct me if that’s a misrepresentation).
If that’s the case, my initial guess is that we’ll only see LLMs doing general reasoning on problems that are relatively small, simple (in the sense that the fully general algorithm is simple), and common in the training data, just because grokking requires such an extended amount of training. But I don’t have very clear intuition about the degree to which frontier-scale LLMs are grokking such small common problems during ordinary training.
Alternately it could be that grokking is sufficient but not necessary, so LLMs can reason in a general way about grokked problems but also about other things.
Yup, that’s basically what I think! IMO, grokking = having memorised the “underlying rules” that define the DGP, and these rules are general by definition.”Reasoning” is a loaded term that’s difficult to unpack, but I think a good working definition is “applying a set of rules to arrive at an answer”. In other words, reasoning is learning a “correct algorithm” to solve the problem. Therefore being able to reason correctly 100% of the time is equivalent to models having grokked their problem domain.
See this work, which finds that reasoning only happens through grokking. Separate work has trained models to do tree search, and found that backwards chaining circuits (a correct algorithm) emerge only through grokking. And also the seminal work on modular addition which found that correct algorithms emerge through grokking.
Note that the question of “is reasoning in natural language grokkable?” is a totally separate crux and one which I’m highly uncertain about.