I think the biggest problem with these estimates is that they rely on irrelevant comparisons to the human brain.
What we care about is how much compute is needed to implement the high-level cognitive algorithms that run in the brain; not the amount of compute needed to simulate the low-level operations the brain carries out to perform that cognition. This is a much harder to quantity to estimate, but it’s also the only thing that actually matters.
I think with enough algorithmic improvement, there’s enough hardware lying around already to get to TAI, and once you factor this in, a bunch of other conditional events are actually unnecessary or much more likely. My own estimates:
Event
Forecast
by 2043 or TAGI, conditional on prior steps
We invent algorithms for transformative AGI
90%
We invent a way for AGIs to learn faster than humans
100%
AGI inference costs drop below $25/hr (per human equivalent)
100%
We invent and scale cheap, quality robots
100%
We massively scale production of chips and power
100%
We avoid derailment by human regulation
80%
We avoid derailment by AI-caused delay
95%
We avoid derailment from wars (e.g., China invades Taiwan)
95%
We avoid derailment from pandemics
95%
We avoid derailment from severe depressions
95%
Joint odds
58.6%
Explanations:
Inventing algorithms: this is mostly just a wild guess / gut sense, but it is upper-bounded in difficulty by the fact that evolution managed to discover algorithms for human level cognition through billions of years of pretty dumb and low bandwidth trial-and-error.
“We invent a way for AGIs to learn faster than humans.” This seems like it is mostly implied by the first conditional already and / or already satisfied: current AI systems can already be trained in less wall-clock time than it takes a human to develop during a lifetime, and much less wall clock time than it took evolution to discover the design for human brains. Even the largest AI systems are currently trained on much less data than the amount of raw sensory data that is available to a human over a lifetime. I see no reason to expect these trends to change as AI gets more advanced.
“AGI inference costs drop below $25/hr (per human equivalent)”—Inference costs of running e.g. GPT-4 are already way, way less than the cost of humans on a per-token basis, it’s just that the tokens are way, way lower quality than the best humans can produce. Conditioned on inventing the algorithms to get higher quality output, it seems like there’s plausibly very little left to do to here.
Inventing robots—doesn’t seem necessary for TAI; we already have decent, if expensive, robotics and TAI can help develop more and make them cheaper / more scalable.
Scaling production of chips and power—probably not necessary, see the point in the first part of this comment on relevant comparisons to the brain.
Avoiding derailment—if existing hardware is sufficient to get to TAI, it’s much easier to avoid derailment for any reason. You just need a few top capabilities research labs to continue to be able to push the frontier by experimenting with existing clusters.
I’m curious about your derailment odds. The definition of “transformative AGI” in the paper is restrictive:
AI that can quickly and affordably be trained to perform nearly all economically and strategically valuable tasks at roughly human cost or less.
A narrow superintelligence that can, for example, engineer pandemics or conduct military operations could lead to severe derailment without satisfying this definition. I guess that would qualify as “AI-caused delay”? To follow the paper’s model, we need to estimate these odds in a conditional world where humans are not regulating AI use in ways that significantly delay the path to transformative AGI, which further increases the risk.
engineer pandemics or conduct military operations could lead to severe derailment without satisfying this definition.
I think humans could already do those things pretty well without AI, if they wanted to. Narrow AI might make those things easier, possibly much easier, just like nukes and biotech research have in the past. I agree this increases the chance that things go “off the rails”, but I think once you have an AI that can solve hard engineering problems in the real world like that, there’s just not that much further to go to full-blown superintelligence, whether you call its precursor “narrow” or not.
The probabilities in my OP are mostly just a gut sense wild guess, but they’re based on the intuition that it takes a really big derailment to halt frontier capabilities progress, which mostly happens in well-funded labs that have the resources and will to continue operating through pretty severe “turbulence”—economic depression, war, pandemics, restrictive regulation, etc. Even if new GPU manufacturing stops completely, there are already a lot of H100s and A100s lying around, and I expect that those are sufficient to get pretty far.
Excellent comment—thanks for sticking your neck out to provide your own probabilities.
Given the gulf between our 0.4% and your 58.6%, would you be interested in making a bet (large or small) on TAI by 2043? If yes, happy to discuss how we might operationalize it.
I appreciate the offer to bet! I’m probably going to decline though—I don’t really want or need more skin-in-the-game on this question (many of my personal and professional plans assume short timelines.)
You might be interested in this post (and the bet it is about), for some commentary and issues with operationalizing bets like this.
Also, you might be able to find someone else to bet with you—I think my view is actually closer to the median among EAs / rationalists / alignment researchers than yours. For example, the Open Phil panelists judging this contest say:
Panelist credences on the probability of AGI by 2043 range from ~10% to ~45%.
I’m not convinced about the difficulty of operationalizing Eliezer’s doomer bet. Effectively, loaning money to a doomer who plans to spend it all by 2030 is, in essence, a claim on the doomer’s post-2030 human capital. The doomer thinks it’s worthless, whereas the skeptic thinks it has value. Hence, they transact.
The TAGI case seems trickier than the doomer case. Who knows what a one dollar bill will be worth in a post-TAGI world.
I think the biggest problem with these estimates is that they rely on irrelevant comparisons to the human brain.
What we care about is how much compute is needed to implement the high-level cognitive algorithms that run in the brain; not the amount of compute needed to simulate the low-level operations the brain carries out to perform that cognition. This is a much harder to quantity to estimate, but it’s also the only thing that actually matters.
See Biology-Inspired AGI Timelines: The Trick That Never Works and other extensive prior discussion on this.
I think with enough algorithmic improvement, there’s enough hardware lying around already to get to TAI, and once you factor this in, a bunch of other conditional events are actually unnecessary or much more likely. My own estimates:
Event
Forecast
by 2043 or TAGI,
conditional on
prior steps
We invent a way for AGIs to learn faster than humansAGI inference costs drop below $25/hr (per human equivalent)We invent and scale cheap, quality robotsWe massively scale production of chips and powerExplanations:
Inventing algorithms: this is mostly just a wild guess / gut sense, but it is upper-bounded in difficulty by the fact that evolution managed to discover algorithms for human level cognition through billions of years of pretty dumb and low bandwidth trial-and-error.
“We invent a way for AGIs to learn faster than humans.” This seems like it is mostly implied by the first conditional already and / or already satisfied: current AI systems can already be trained in less wall-clock time than it takes a human to develop during a lifetime, and much less wall clock time than it took evolution to discover the design for human brains. Even the largest AI systems are currently trained on much less data than the amount of raw sensory data that is available to a human over a lifetime. I see no reason to expect these trends to change as AI gets more advanced.
“AGI inference costs drop below $25/hr (per human equivalent)”—Inference costs of running e.g. GPT-4 are already way, way less than the cost of humans on a per-token basis, it’s just that the tokens are way, way lower quality than the best humans can produce. Conditioned on inventing the algorithms to get higher quality output, it seems like there’s plausibly very little left to do to here.
Inventing robots—doesn’t seem necessary for TAI; we already have decent, if expensive, robotics and TAI can help develop more and make them cheaper / more scalable.
Scaling production of chips and power—probably not necessary, see the point in the first part of this comment on relevant comparisons to the brain.
Avoiding derailment—if existing hardware is sufficient to get to TAI, it’s much easier to avoid derailment for any reason. You just need a few top capabilities research labs to continue to be able to push the frontier by experimenting with existing clusters.
I’m curious about your derailment odds. The definition of “transformative AGI” in the paper is restrictive:
A narrow superintelligence that can, for example, engineer pandemics or conduct military operations could lead to severe derailment without satisfying this definition. I guess that would qualify as “AI-caused delay”? To follow the paper’s model, we need to estimate these odds in a conditional world where humans are not regulating AI use in ways that significantly delay the path to transformative AGI, which further increases the risk.
I think humans could already do those things pretty well without AI, if they wanted to. Narrow AI might make those things easier, possibly much easier, just like nukes and biotech research have in the past. I agree this increases the chance that things go “off the rails”, but I think once you have an AI that can solve hard engineering problems in the real world like that, there’s just not that much further to go to full-blown superintelligence, whether you call its precursor “narrow” or not.
The probabilities in my OP are mostly just a gut sense wild guess, but they’re based on the intuition that it takes a really big derailment to halt frontier capabilities progress, which mostly happens in well-funded labs that have the resources and will to continue operating through pretty severe “turbulence”—economic depression, war, pandemics, restrictive regulation, etc. Even if new GPU manufacturing stops completely, there are already a lot of H100s and A100s lying around, and I expect that those are sufficient to get pretty far.
Excellent comment—thanks for sticking your neck out to provide your own probabilities.
Given the gulf between our 0.4% and your 58.6%, would you be interested in making a bet (large or small) on TAI by 2043? If yes, happy to discuss how we might operationalize it.
I appreciate the offer to bet! I’m probably going to decline though—I don’t really want or need more skin-in-the-game on this question (many of my personal and professional plans assume short timelines.)
You might be interested in this post (and the bet it is about), for some commentary and issues with operationalizing bets like this.
Also, you might be able to find someone else to bet with you—I think my view is actually closer to the median among EAs / rationalists / alignment researchers than yours. For example, the Open Phil panelists judging this contest say:
Sounds good. Can also leave money out of it and put you down for 100 pride points. :)
If so, message me your email and I’ll send you a calendar invite for a group reflection in 2043, along with a midpoint check in in 2033.
I’m not convinced about the difficulty of operationalizing Eliezer’s doomer bet. Effectively, loaning money to a doomer who plans to spend it all by 2030 is, in essence, a claim on the doomer’s post-2030 human capital. The doomer thinks it’s worthless, whereas the skeptic thinks it has value. Hence, they transact.
The TAGI case seems trickier than the doomer case. Who knows what a one dollar bill will be worth in a post-TAGI world.