I have yet to read a post here on LW were someone write about a frontier model that have solved a “real” problem were the person really had tried for a long(-ish) time to solve it but failed and then the AI solved it for them, like a research problem, a coding problem, a math problem, a medical problem, a personal problem etc. Has anyone experienced this yet?
I don’t have any first- or second-hand experience with that happening. I see occasional articles about protein and materials research that LLMs have massively accelerated, but my suspicion is the main value currently is NOT cutting-edge significant problem solving.
The “mundane utility” section of Zvi’s writeups have very good examples of what it IS currently good at now. There’s yet a long way to go to handle long-running multi-step creative analysis that the top few percent of humans are engaged in. What is the shape of the curve (both of capabilities and of “intelligence level” metrics) is currently not known.
You mean specifically that an LLM solved it? Otherwise Deepmind’s work will give you many examples. (Although there’ve been surprisingly little breakthroughs in math yet)
Yes, I meant an LLM in the context of a user that fed in a query of his or her problem and got a novel solution back. There is always debatable what a “real” or “hard” problem is, but as a lower bound I mean something that people here at LW would raise an eyebrow or two if the LLM solved. Otherwise there are as you mention plenty of stuff/problems that “custom” AI/machine learning models have solved for a long time.
I have yet to read a post here on LW were someone write about a frontier model that have solved a “real” problem were the person really had tried for a long(-ish) time to solve it but failed and then the AI solved it for them, like a research problem, a coding problem, a math problem, a medical problem, a personal problem etc. Has anyone experienced this yet?
I don’t have any first- or second-hand experience with that happening. I see occasional articles about protein and materials research that LLMs have massively accelerated, but my suspicion is the main value currently is NOT cutting-edge significant problem solving.
The “mundane utility” section of Zvi’s writeups have very good examples of what it IS currently good at now. There’s yet a long way to go to handle long-running multi-step creative analysis that the top few percent of humans are engaged in. What is the shape of the curve (both of capabilities and of “intelligence level” metrics) is currently not known.
Thanks for pointing me to Zvi’s work
You mean specifically that an LLM solved it? Otherwise Deepmind’s work will give you many examples. (Although there’ve been surprisingly little breakthroughs in math yet)
Yes, I meant an LLM in the context of a user that fed in a query of his or her problem and got a novel solution back. There is always debatable what a “real” or “hard” problem is, but as a lower bound I mean something that people here at LW would raise an eyebrow or two if the LLM solved. Otherwise there are as you mention plenty of stuff/problems that “custom” AI/machine learning models have solved for a long time.