Quick sanity check: are we currently able to build human-level AGI, and bottlenecked only by project funding?
VentureBeat claims that GPT-3 cost around $12 million to train, and this is the higher of 2 estimates I found. And GPT-3 used 3640 petaflop/s-days. That’s about ten years worth of petaflop/s level compute. Now, the power of the human brain is often estimated at about 1 exaflop/s, and again, that estimate is on the high side of the range. Ten year old humans are typically generally intelligent. So if we took 3 orders of magnitude more compute than GPT-3, that naively sounds like it should be enough to produce human-level AGI. If costs scale linearly, that’s $12 billion, not chump change, but a fraction of e.g. what the U.S. is currently sending to Ukraine.
Of course, assuming linear scaling is frequently wrong. It’s one thing to pay for compute when it’s not that scarce and you’re only trying to train a 175 billion parameter model for a bit. It’s quite another if you’re trying to use a substantial quantity of the world’s compute, and having to bid against many other projects to do it. The marginal costs should rise substantially; perhaps the total cost would be more on the order of $100 billion.
As for (linear?) scaling of intelligence, that’s why I’m wondering about this: Gato is displaying such effective generalization that some here have called it literal AGI, albeit not human-level yet. While that’s what one would expect from the previous success of scaling laws, it’s still striking to see it confirmed to this degree. A few years ago, the idea that simply throwing more compute at the problem would continuously yield more intelligence was considered absurd. Now, as Gwern put it, we know Scaling Just Works.
Moreover, ten subjective years of training is probably a lot more than human-level AGI would need. An awful lot of human development is limited by the body and brain growing, rather than lack of training data, and quite a lot of experiences growing up do not provide any new information. Perhaps 4 years or so of data would be enough, and if 10 years cost $100 billion, 4 might be more like $40 billion.
Of course, these are extremely rough estimates, but the general point should be clear: even without any further help from Moore’s Law, and even without significant further software breakthroughs, it sounds like humanity is capable of human-level AGI RIGHT NOW. Does this seem plausible?
Quick sanity check: are we currently able to build human-level AGI, and bottlenecked only by project funding?
VentureBeat claims that GPT-3 cost around $12 million to train, and this is the higher of 2 estimates I found. And GPT-3 used 3640 petaflop/s-days. That’s about ten years worth of petaflop/s level compute. Now, the power of the human brain is often estimated at about 1 exaflop/s, and again, that estimate is on the high side of the range. Ten year old humans are typically generally intelligent. So if we took 3 orders of magnitude more compute than GPT-3, that naively sounds like it should be enough to produce human-level AGI. If costs scale linearly, that’s $12 billion, not chump change, but a fraction of e.g. what the U.S. is currently sending to Ukraine.
Of course, assuming linear scaling is frequently wrong. It’s one thing to pay for compute when it’s not that scarce and you’re only trying to train a 175 billion parameter model for a bit. It’s quite another if you’re trying to use a substantial quantity of the world’s compute, and having to bid against many other projects to do it. The marginal costs should rise substantially; perhaps the total cost would be more on the order of $100 billion.
As for (linear?) scaling of intelligence, that’s why I’m wondering about this: Gato is displaying such effective generalization that some here have called it literal AGI, albeit not human-level yet. While that’s what one would expect from the previous success of scaling laws, it’s still striking to see it confirmed to this degree. A few years ago, the idea that simply throwing more compute at the problem would continuously yield more intelligence was considered absurd. Now, as Gwern put it, we know Scaling Just Works.
Moreover, ten subjective years of training is probably a lot more than human-level AGI would need. An awful lot of human development is limited by the body and brain growing, rather than lack of training data, and quite a lot of experiences growing up do not provide any new information. Perhaps 4 years or so of data would be enough, and if 10 years cost $100 billion, 4 might be more like $40 billion.
Of course, these are extremely rough estimates, but the general point should be clear: even without any further help from Moore’s Law, and even without significant further software breakthroughs, it sounds like humanity is capable of human-level AGI RIGHT NOW. Does this seem plausible?