I am also very confused. The space of problems has a really surprising structure, permitting algorithms that are incredibly adept at some forms of problem-solving, yet utterly inept at others.
We’re only familiar with human minds, in which there’s a tight coupling between the performances on some problems (e. g., between the performance on chess or sufficiently well-posed math/programming problems, and the general ability to navigate the world). Now we’re generating other minds/proto-minds, and we’re discovering that this coupling isn’t fundamental.
(This is an argument for longer timelines, by the way. Current AIs feel on the very cusp of being AGI, but there in fact might be some vast gulf between their algorithms and human-brain algorithms that we just don’t know how to talk about.)
No current AI system could generate a research paper that would receive anything but the lowest possible score from each reviewer
I don’t think that’s strictly true, the peer-review system often approves utter nonsense. But yes, I don’t think any AI system can generate an actually worthwhile research paper.
Reliability is way more important than people realized. One of the central problems that hasn’t gone away as AI scaled is that their best performance is too unreliable for anything but very easy to verify problems like mathematics and programming, which prevents unreliability from becoming crippling, but otherwise this is the key blocker that standard AI scaling has basically never solved.
It’s possible in practice to disentangle certain capabilities from each other, and in particular math and programming capabilities do not automatically imply other capabilities, even if we somehow had figured out how to make the o-series as good as AlphaZero for math and programming, which is good news for AI control.
The AGI term, and a lot of the foundation built off of it, like timelines to AGI, will become less and less relevant over time, because of both the varying meanings, combined with the fact that as AI progresses, capabilities will be developed in a different order from humans, meaning a lot of confusion is on the way, and we’d need different metrics.
We should expect that AI that automates AI research/the economy to look more like Deep Blue/brute-forcing a problem/having good execution skills than AIs like AlphaZero that use very clean/aesthetically beautiful algorithmic strategies.
Reliability is way more important than people realized
Yes, but whence human reliability? What makes humans so much more reliable than the SotA AIs? What are AIs missing? The gulf in some cases is so vast it’s a quantity-is-a-quality-all-its-own thing.
1 is that the structure of jobs is shaped to accommodate human unreliability by making mistakes less fatal.
2 is that while humans themselves aren’t reliable, their algorithms almost certainly are more powerful at error detection and correction, so the big thing AI needs to achieve is the ability to error-correct or become more reliable.
There’s also the fact that humans are better at sample efficiency than most LLMs, but that’s a more debatable proposition.
the structure of jobs is shaped to accommodate human unreliability by making mistakes less fatal
Mm, so there’s a selection effect on the human end, where the only jobs/pursuits that exist are those which humans happen to be able to reliably do, and there’s a discrepancy between the things humans and AIs are reliable at, so we end up observing AIs being more unreliable, even though this isn’t representative of the average difference between the human vs. AI reliability across all possible tasks?
I don’t know that I buy this. Humans seem pretty decent at becoming reliable at ~anything, and I don’t think we’ve observed AIs being more-reliable-than-humans at anything? (Besides trivial and overly abstract tasks such as “next-token prediction”.)
My claim was more along the lines of if an unaided human can’t do a job safely or reliably, as was almost certainly the case 150-200 years ago, if not more years in the past, we make the jobs safer using tools such that human error is way less of a big deal, and AIs currently haven’t used tools that increased their reliability.
Remember, it took a long time for factories to be made safe, and I’d expect a similar outcome for driving, so while I don’t think 1 is everything, I do think it’s a non-trivial portion of the reliability difference.
I am also very confused. The space of problems has a really surprising structure, permitting algorithms that are incredibly adept at some forms of problem-solving, yet utterly inept at others.
We’re only familiar with human minds, in which there’s a tight coupling between the performances on some problems (e. g., between the performance on chess or sufficiently well-posed math/programming problems, and the general ability to navigate the world). Now we’re generating other minds/proto-minds, and we’re discovering that this coupling isn’t fundamental.
(This is an argument for longer timelines, by the way. Current AIs feel on the very cusp of being AGI, but there in fact might be some vast gulf between their algorithms and human-brain algorithms that we just don’t know how to talk about.)
I don’t think that’s strictly true, the peer-review system often approves utter nonsense. But yes, I don’t think any AI system can generate an actually worthwhile research paper.
I think the main takeaways are the following:
Reliability is way more important than people realized. One of the central problems that hasn’t gone away as AI scaled is that their best performance is too unreliable for anything but very easy to verify problems like mathematics and programming, which prevents unreliability from becoming crippling, but otherwise this is the key blocker that standard AI scaling has basically never solved.
It’s possible in practice to disentangle certain capabilities from each other, and in particular math and programming capabilities do not automatically imply other capabilities, even if we somehow had figured out how to make the o-series as good as AlphaZero for math and programming, which is good news for AI control.
The AGI term, and a lot of the foundation built off of it, like timelines to AGI, will become less and less relevant over time, because of both the varying meanings, combined with the fact that as AI progresses, capabilities will be developed in a different order from humans, meaning a lot of confusion is on the way, and we’d need different metrics.
Tweet below:
https://x.com/ObserverSuns/status/1511883906781356033
We should expect that AI that automates AI research/the economy to look more like Deep Blue/brute-forcing a problem/having good execution skills than AIs like AlphaZero that use very clean/aesthetically beautiful algorithmic strategies.
Yes, but whence human reliability? What makes humans so much more reliable than the SotA AIs? What are AIs missing? The gulf in some cases is so vast it’s a quantity-is-a-quality-all-its-own thing.
I have 2 answers to this.
1 is that the structure of jobs is shaped to accommodate human unreliability by making mistakes less fatal.
2 is that while humans themselves aren’t reliable, their algorithms almost certainly are more powerful at error detection and correction, so the big thing AI needs to achieve is the ability to error-correct or become more reliable.
There’s also the fact that humans are better at sample efficiency than most LLMs, but that’s a more debatable proposition.
Mm, so there’s a selection effect on the human end, where the only jobs/pursuits that exist are those which humans happen to be able to reliably do, and there’s a discrepancy between the things humans and AIs are reliable at, so we end up observing AIs being more unreliable, even though this isn’t representative of the average difference between the human vs. AI reliability across all possible tasks?
I don’t know that I buy this. Humans seem pretty decent at becoming reliable at ~anything, and I don’t think we’ve observed AIs being more-reliable-than-humans at anything? (Besides trivial and overly abstract tasks such as “next-token prediction”.)
(2) seems more plausible to me.
My claim was more along the lines of if an unaided human can’t do a job safely or reliably, as was almost certainly the case 150-200 years ago, if not more years in the past, we make the jobs safer using tools such that human error is way less of a big deal, and AIs currently haven’t used tools that increased their reliability.
Remember, it took a long time for factories to be made safe, and I’d expect a similar outcome for driving, so while I don’t think 1 is everything, I do think it’s a non-trivial portion of the reliability difference.
More here:
https://www.lesswrong.com/posts/DQKgYhEYP86PLW7tZ/how-factories-were-made-safe