I think that’s one issue; LLMs don’t get the same types of guidance, etc. that humans get; they get a lot of training and RL feedback, but it’s structured very differently.
I think this particular article gets another major factor right, where most analyses overlook it: LLMs by default don’t do metacognitive checks on their thinking. This is a huge factor in humans appearing as smart as we do. We make a variety of mistakes in our first guesses (system 1 thinking) that can be found and corrected with sufficient reflection (system 2 thinking). Adding more of this to LLM agents is likely to be a major source of capabilities improvements. The focus on increasing “9s of reliability” is a very CS approach; humans just make tons of mistakes and then catch many of the important ones; LLMs sort of copy their cognition from humans, so they can benefit from the same approach—but they don’t do much of it by default. Scripting it in to LLM agents is going to at least help, and it may help a lot.
I think that’s one issue; LLMs don’t get the same types of guidance, etc. that humans get; they get a lot of training and RL feedback, but it’s structured very differently.
I think this particular article gets another major factor right, where most analyses overlook it: LLMs by default don’t do metacognitive checks on their thinking. This is a huge factor in humans appearing as smart as we do. We make a variety of mistakes in our first guesses (system 1 thinking) that can be found and corrected with sufficient reflection (system 2 thinking). Adding more of this to LLM agents is likely to be a major source of capabilities improvements. The focus on increasing “9s of reliability” is a very CS approach; humans just make tons of mistakes and then catch many of the important ones; LLMs sort of copy their cognition from humans, so they can benefit from the same approach—but they don’t do much of it by default. Scripting it in to LLM agents is going to at least help, and it may help a lot.