We actually have a resolution for the thread on whether LLMs naturally learn algorithmic reasoning as they scale up with COT vs just reasoning with memorized bags of heuristics, and the answer is that we have both real reasoning, which is indicative of LLMs actually using somewhat clean algorithms, but there are also a lot of heuristic reasoning involved.
So we both got some things wrong, but also got some things right.
The main thing I got wrong was in underestimating how much COT for current models still involves pretty significant memorization/bag of heuristics to get correct answers, which means I have to raise the complexity of human values, given that LLMs didn’t compress as well as I thought, and the thing I got right was that sequential computation like COT does incentive actual noisy reasoning/algorithms to appear, but I was wrong about the strength of the effect, though I was still right to be concerned about the fact that the OthelloGPT network was very wide and skinny, rather than deep and wide, which makes it harder to learn the correct algorithm.
We actually have a resolution for the thread on whether LLMs naturally learn algorithmic reasoning as they scale up with COT vs just reasoning with memorized bags of heuristics, and the answer is that we have both real reasoning, which is indicative of LLMs actually using somewhat clean algorithms, but there are also a lot of heuristic reasoning involved.
So we both got some things wrong, but also got some things right.
The main thing I got wrong was in underestimating how much COT for current models still involves pretty significant memorization/bag of heuristics to get correct answers, which means I have to raise the complexity of human values, given that LLMs didn’t compress as well as I thought, and the thing I got right was that sequential computation like COT does incentive actual noisy reasoning/algorithms to appear, but I was wrong about the strength of the effect, though I was still right to be concerned about the fact that the OthelloGPT network was very wide and skinny, rather than deep and wide, which makes it harder to learn the correct algorithm.
The thread is below:
https://x.com/aksh_555/status/1843326181950828753
I wish someone is willing to do this for the o1 series of models as well.
One other relevant comment is here:
https://www.lesswrong.com/posts/gcpNuEZnxAPayaKBY/othellogpt-learned-a-bag-of-heuristics-1#HqDWs9NHmYivyeBGk