For a given AGI lab, the decision to keep working on the project despite believing at least 10% risk of extinction depends on the character of counterfactuals. Success is not just another draw out of the extinction urn, taking another step on the path to eventual doom, instead it promises that the new equilibrium involves robust safety with no future draws. So it’s all about the alternatives.
One issue for individual labs is that their alternative is likely that the other labs develop AGI instead, they personally have little power to pause AI globally, unless they involve themselves in coordination with all other capable actors. Many arguments stop here, considering such coordination infeasible.
The risk of literal extinction for reasons other than AGI seems vanishingly small for the foreseeable future. There are many global catastrophic risks with moderate probability when added up over decades, some of which might disrupt the course of civilization for millennia, but not literal extinction. The closest risk of actual extinction that doesn’t involve AGI I can imagine is advanced biotechnology of the kind that’s not even on the horizon yet. It’s unclear how long it would take to get there without AI, while dodging civilization-wreaking catastrophes that precede its development, but I would guess a lower bound of many decades before this becomes a near-term possibility. Even then it won’t become a certainty of immediate doom, in a similar way to how large nuclear arsenals still haven’t cashed out in a global nuclear conflict for many decades. So it makes sense to work towards global coordination to pause AI for at least this long, as long as there is vigorous effort to develop AI alignment theory and prepare in all ways that make sense during this time.
For a given AGI lab, the decision to keep working on the project despite believing at least 10% risk of extinction depends on the character of counterfactuals. Success is not just another draw out of the extinction urn, taking another step on the path to eventual doom, instead it promises that the new equilibrium involves robust safety with no future draws. So it’s all about the alternatives.
One issue for individual labs is that their alternative is likely that the other labs develop AGI instead, they personally have little power to pause AI globally, unless they involve themselves in coordination with all other capable actors. Many arguments stop here, considering such coordination infeasible.
The risk of literal extinction for reasons other than AGI seems vanishingly small for the foreseeable future. There are many global catastrophic risks with moderate probability when added up over decades, some of which might disrupt the course of civilization for millennia, but not literal extinction. The closest risk of actual extinction that doesn’t involve AGI I can imagine is advanced biotechnology of the kind that’s not even on the horizon yet. It’s unclear how long it would take to get there without AI, while dodging civilization-wreaking catastrophes that precede its development, but I would guess a lower bound of many decades before this becomes a near-term possibility. Even then it won’t become a certainty of immediate doom, in a similar way to how large nuclear arsenals still haven’t cashed out in a global nuclear conflict for many decades. So it makes sense to work towards global coordination to pause AI for at least this long, as long as there is vigorous effort to develop AI alignment theory and prepare in all ways that make sense during this time.