b) we continue to improve at agentic scaffolding and other forms of “unhobbling”, and
c) algorithmic efficiency improvement continue at about the same pace, and
d) the willingness of investors to invest exponentially more money in training AI each year continues to scale up at about the same rate, and
e) we don’t hit any new limit like meaningfully running out of training data or power for training clusters, then:
capabilities will look something a lot like or close to Artificial General Intelligence (AGI)/Transformative Artificial Intelligence (TAI). Probably a patchy AGI, with some capabilities well into superhuman, most around expert-human levels (some still perhaps exceeded by rare very-talented-and-skilled individuals), and a few not yet at expert-human levels: depending on which abilities those are, it may be more of less TAI (currently long-term planning/plan execution is an really important major weakness: if that didn’t get mostly-fixed by some combination of scaling, unhobbling, and new training data then it would be a critical lack).
Individually each of those listed preconditions seem pretty likely, but obviously there are five of them. If any of them fail, then we’ll be close to but not quite at AGI, and making slower progress towards it, but we won’t be stalled unless basically all of them fail, which seems like a really unlikely coincidence.
Almost certainly this will not yet by broadly applied across the economy, but given the potential for order-of-magnitude-or-more cost savings, people will be scrambling to apply it rapidly (during which fortunes will be made and lost), and there will be a huge amount of “who moved my cheese?” social upheaval as a result. As AI becomes increasingly AGI-like, the difficulty of applying it effectively to a given economic use case will reduce to somewhere around the difficulty of integrating and bringing-up-to-speed single human new-hire. So a massive and rapid economic upheaval will be going on. As an inevitable result, Luddite views and policies will skyrocket, and AI will become extremely unpopular with a great many people. A significant question here is whether this disruption will, in 2028, be limited to purely-intellectual work, or if advances in robotics will have yet started to have the same effect on jobs that also have a manual work element. I’m not enough of an expert on robotics to have an informed opinion here: my best guess is that robotics will lag, but not by much, since robotics research is mostly intellectual work.
This is of course about the level where the rubber really starts to hit the road on AI safety: we’re no longer talking about naughty stories, cheap phishing, or how-to-guides on making drugs at home, and are looking at systems capable of autonomously or under human direction committing serious criminal or offensive activities at a labor cost at least an order-of-magnitude below current, and an escaped self-replicating malicious agent is feasible and might be able to evade law enforcement and computer security professionals unless they had equivalent AI assistance. If we get any major “warning shots” on AI safety, this is when they’ll happen (personally I expect them to come thick-and fast). It’s teetering on the edge of the existential risk level of Artificial Super-Intelligence (ASI).
Somewhere around that point, we start to hit two conflicting influences: 1) an intelligence feedback explosion from AGI accelerating AI research, vs. 2) to train a super-intelligence you need to synthesize very large amounts of training data displaying superintelligent behavior, rather than just using prexisting data from humans. So we either get a fast takeoff, or a slowdown, or some combination of the two. That’s hard to predict: we’re starting to get close to the singularity, where the usual fact that predictions are hard (especially about the future) is compounded by it being functionally almost impossible to predict the capabilities of something much smarter than us, especially when we’ve never previously seen anything smarter than us.
The big issue, in terms of AI safety, is likely to be misuse, not alignment issues, primarily because I expect these AGIs to exhibit quite a lot less instrumental convergence than humans out of the box, due to being trained on much denser data and rewards than humans, and I think this allows for corrigibility/DWIMAC strategies to alignment to mostly just work.
However, misuse of AIs will become a harder problem to solve, and short term, I expect the solution to be never releasing unrestricted AI to the general public and only allowing unrestricted AIs for internal use like AI research, unless it has robust resistance to fine-tuning attacks, and longer term, I think the solution will have to require more misuse-resistant AIs.
Also, in the world you sketched, with my additions, the political values of who control AIs become very important, for better or worse.
Can you define what AGI means to you in concrete observable terms? Will it concretely be an app that runs on a computer and does white collar jobs, or something else?
I am using Artificial General Intelligence (AGI) to mean a AI that is, broadly, at least as good at most intellectual tasks as the typical person who makes a living from performing that intellectual task. If that applies across most economically-important intellectual tasks at a cost that is lower-than a human, then this is also presumably going to be Transformative Artificial Intelligence (TAI). So the latter means that it would be competitive at most white-collar jobs.
If:
a) the AI scaling curves hold up, and
b) we continue to improve at agentic scaffolding and other forms of “unhobbling”, and
c) algorithmic efficiency improvement continue at about the same pace, and
d) the willingness of investors to invest exponentially more money in training AI each year continues to scale up at about the same rate, and
e) we don’t hit any new limit like meaningfully running out of training data or power for training clusters, then:
capabilities will look something a lot like or close to Artificial General Intelligence (AGI)/Transformative Artificial Intelligence (TAI). Probably a patchy AGI, with some capabilities well into superhuman, most around expert-human levels (some still perhaps exceeded by rare very-talented-and-skilled individuals), and a few not yet at expert-human levels: depending on which abilities those are, it may be more of less TAI (currently long-term planning/plan execution is an really important major weakness: if that didn’t get mostly-fixed by some combination of scaling, unhobbling, and new training data then it would be a critical lack).
Individually each of those listed preconditions seem pretty likely, but obviously there are five of them. If any of them fail, then we’ll be close to but not quite at AGI, and making slower progress towards it, but we won’t be stalled unless basically all of them fail, which seems like a really unlikely coincidence.
Almost certainly this will not yet by broadly applied across the economy, but given the potential for order-of-magnitude-or-more cost savings, people will be scrambling to apply it rapidly (during which fortunes will be made and lost), and there will be a huge amount of “who moved my cheese?” social upheaval as a result. As AI becomes increasingly AGI-like, the difficulty of applying it effectively to a given economic use case will reduce to somewhere around the difficulty of integrating and bringing-up-to-speed single human new-hire. So a massive and rapid economic upheaval will be going on. As an inevitable result, Luddite views and policies will skyrocket, and AI will become extremely unpopular with a great many people. A significant question here is whether this disruption will, in 2028, be limited to purely-intellectual work, or if advances in robotics will have yet started to have the same effect on jobs that also have a manual work element. I’m not enough of an expert on robotics to have an informed opinion here: my best guess is that robotics will lag, but not by much, since robotics research is mostly intellectual work.
This is of course about the level where the rubber really starts to hit the road on AI safety: we’re no longer talking about naughty stories, cheap phishing, or how-to-guides on making drugs at home, and are looking at systems capable of autonomously or under human direction committing serious criminal or offensive activities at a labor cost at least an order-of-magnitude below current, and an escaped self-replicating malicious agent is feasible and might be able to evade law enforcement and computer security professionals unless they had equivalent AI assistance. If we get any major “warning shots” on AI safety, this is when they’ll happen (personally I expect them to come thick-and fast). It’s teetering on the edge of the existential risk level of Artificial Super-Intelligence (ASI).
Somewhere around that point, we start to hit two conflicting influences: 1) an intelligence feedback explosion from AGI accelerating AI research, vs. 2) to train a super-intelligence you need to synthesize very large amounts of training data displaying superintelligent behavior, rather than just using prexisting data from humans. So we either get a fast takeoff, or a slowdown, or some combination of the two. That’s hard to predict: we’re starting to get close to the singularity, where the usual fact that predictions are hard (especially about the future) is compounded by it being functionally almost impossible to predict the capabilities of something much smarter than us, especially when we’ve never previously seen anything smarter than us.
The big issue, in terms of AI safety, is likely to be misuse, not alignment issues, primarily because I expect these AGIs to exhibit quite a lot less instrumental convergence than humans out of the box, due to being trained on much denser data and rewards than humans, and I think this allows for corrigibility/DWIMAC strategies to alignment to mostly just work.
However, misuse of AIs will become a harder problem to solve, and short term, I expect the solution to be never releasing unrestricted AI to the general public and only allowing unrestricted AIs for internal use like AI research, unless it has robust resistance to fine-tuning attacks, and longer term, I think the solution will have to require more misuse-resistant AIs.
Also, in the world you sketched, with my additions, the political values of who control AIs become very important, for better or worse.
Can you define what AGI means to you in concrete observable terms? Will it concretely be an app that runs on a computer and does white collar jobs, or something else?
I am using Artificial General Intelligence (AGI) to mean a AI that is, broadly, at least as good at most intellectual tasks as the typical person who makes a living from performing that intellectual task. If that applies across most economically-important intellectual tasks at a cost that is lower-than a human, then this is also presumably going to be Transformative Artificial Intelligence (TAI). So the latter means that it would be competitive at most white-collar jobs.