We’re definitely on/past the inflection point of the GPT S-curve, but at the same time there’s been work on other AI methods and we know they’re in principle possible. What is needed to break the data wall is some AI that can autonomously collect new data, especially in subjects where it’s weak.
One key question is how much investment AI companies make into this. E.g. the obvious next step to me would be to use a proof assistant to become superhuman at math by self-play. I know there’s been a handful of experiments with this, but AFAIK it hasn’t been done at huge scale or incorporated into current LLMs. Obviously they’re not gonna break into the next S-curve until they try to do so.
We’re definitely on/past the inflection point of the GPT S-curve, but at the same time there’s been work on other AI methods and we know they’re in principle possible. What is needed to break the data wall is some AI that can autonomously collect new data, especially in subjects where it’s weak.
One key question is how much investment AI companies make into this. E.g. the obvious next step to me would be to use a proof assistant to become superhuman at math by self-play. I know there’s been a handful of experiments with this, but AFAIK it hasn’t been done at huge scale or incorporated into current LLMs. Obviously they’re not gonna break into the next S-curve until they try to do so.