Random thought: maybe (at least pre-reasoning-models) LLMs are RLHF’d to be “competent” in a way that makes them less curious & excitable, which greatly reduces their chance of coming up with (and recognizing) any real breakthroughs. I would expect though that for reasoning models such limitations will necessarily disappear and they’ll be much more likely to produce novel insights. Still, scaffolding and lack of context and agency can be a serious bottleneck.
Random thought: maybe (at least pre-reasoning-models) LLMs are RLHF’d to be “competent” in a way that makes them less curious & excitable, which greatly reduces their chance of coming up with (and recognizing) any real breakthroughs. I would expect though that for reasoning models such limitations will necessarily disappear and they’ll be much more likely to produce novel insights. Still, scaffolding and lack of context and agency can be a serious bottleneck.
I think it’s the latter.