I think (2) does play an important part here, and that the recent work on allowing AIs to notice and correct their mistakes (calibration training, backspace-tokens for error correction) are going to show some dividends once they make their way from the research frontier to actually deployed frontier models.
I think (2) does play an important part here, and that the recent work on allowing AIs to notice and correct their mistakes (calibration training, backspace-tokens for error correction) are going to show some dividends once they make their way from the research frontier to actually deployed frontier models.
Relevant links:
LLMs cannot find reasoning errors, but can correct them!
Physics of LLMs: learning from mistakes
Explanation of Accuracy vs Calibration vs Robustness
A Survey of Calibration Process for Black-Box LLMs