I’m worried that “pause all AI development” is like the “defund the police” of the alignment community. I’m not convinced it’s net bad because I haven’t been following governance—my current guess is neutral—but I do see these similarities:
It’s incredibly difficult and incentive-incompatible with existing groups in power
There are less costly, more effective steps to reduce the underlying problem, like making the field of alignment 10x larger or passing regulation to require evals
There are some obvious negative effects; potential overhangs or greater incentives to defect in the AI case, and increased crime, including against disadvantaged groups, in the police case
There’s far more discussion than action (I’m not counting the fact that GPT5 isn’t being trained yet; that’s for other reasons)
It’s memetically fit, and much discussion is driven by two factors that don’t advantage good policies over bad policies, and might even do the reverse. This is the toxoplasma of rage.
disagreement with the policy
(speculatively) intragroup signaling; showing your dedication to even an inefficient policy proposal proves you’re part of the ingroup. I’m not 100% this was a large factor in “defund the police” and this seems even less true with the FLI letter, but still worth mentioning.
This seems like a potentially unpopular take, so I’ll list some cruxes. I’d change my mind and endorse the letter if some of the following are true.
The claims above are mistaken/false somehow.
Top labs actually start taking beneficial actions towards the letter’s aims
It’s caused people to start thinking more carefully about AI risk
A 6 month pause now is especially important by setting anti-racing norms, demonstrating how far AI alignment is lagging behind capabilities, or something
A 6 month pause now is worth close to 6 months of alignment research at crunch time (my guess is that research at crunch time is worth 1.5x-3x more depending on whether MIRI is right about everything)
The most important quality to push towards in public discourse is how much we care about safety at all, so I should endorse this proposal even though it’s flawed
The obvious dis-analogy is that if the police had no funding and largely ceased to exist, a string of horrendous things would quickly occur. Murders and thefts and kidnappings and rapes and more would occur throughout every country in which it was occurring, people would revert to tight-knit groups who had weapons to defend themselves, a lot of basic infrastructure would probably break down (e.g. would Amazon be able to pivot to get their drivers armed guards?) and much more chaos would ensue.
And if AI research paused, society would continue to basically function as it has been doing so far.
One of them seems to me like a goal that directly causes catastrophes and a breakdown of society and the other doesn’t.
Fair point. Another difference is that the pause is popular! 66-69% in favor of the pause, and 41% think AI would do more harm than good vs 9% for more good than harm.
There are less costly, more effective steps to reduce the underlying problem, like making the field of alignment 10x larger or passing regulation to require evals
IMO making the field of alignment 10x larger or evals do not solve a big part of the problem, while indefinitely pausing AI development would. I agree it’s much harder, but I think it’s good to at least try, as long as it doesn’t terribly hurt less ambitious efforts (which I think it doesn’t).
There are less costly, more effective steps to reduce the underlying problem, like making the field of alignment 10x larger or passing regulation to require evals
This statement begs for cost-benefit analysis.
Increasing size of alignment field can be efficient, but it won’t be cheap. You need to teach new experts in the field that doesn’t have any polised standardized educational programs and doesn’t have much of teachers. If you want not only increase number of participants in the field, but increase productivity of the field 10x, you need an extraordinary educational effort.
Passing regulation to require evals seems like a meh idea. Nobody knows in enough details how to make such evalutions and every wrong idea that makes its way to law will be here until the end of the world.
I’d be much happier with increasing participants enough to equal 10-20% of the field of ML than a 6 month unconditional pause, and my guess is it’s less costly. It seems like leading labs allowing other labs to catch up by 6 months will reduce their valuations more than 20%, whereas diverting 10-20% of their resources would reduce valuations only 10% or so.
There are currently 300 alignment researchers. If we take additional researchers from the pool of 30k people who attended ICML, you get 3000 researchers, and if they’re equal quality this is 10x participants. I wouldn’t expect alignment to go 10x faster, more like 2x with a decent educational effort. But this is in perpetuity and should speed up alignment by far more than 6 months. There’s the question of getting labs to pay if they’re creating most of the harms, which might be hard though.
I’d be excited about someone doing a real cost-benefit analysis here, or preferably coming up with better ideas. It just seems so unlikely that a 6 month pause is close to the most efficient thing, given it destroys much of the value of a company that has a large lead.
I now think the majority of impact of AI pause advocacy will come from the radical flank effect, and people should study it to decide whether pause advocacy is good or bad.
It’s incredibly difficult and incentive-incompatible with existing groups in power
Why does this have to be true? Can’t governments just compensate existing AGI labs for the expected commercial value of their foregone future advances due to indefinite pause?
This seems good if it could be done. But the original proposal was just a call for labs to individually pause their research, which seems really unlikely to work.
Also, the level of civilizational competence required to compensate labs seems to be higher than for other solutions. I don’t think it’s a common regulatory practice to compensate existing labs like this, and it seems difficult to work out all the details so that labs will feel adequately compensated. Plus there might be labs that irrationally believe they’re undervalued. Regulations similar to the nuclear or aviation industry feel like a more plausible way to get slowdown, and have the benefit that they actually incentivize safety work.
I’m worried that “pause all AI development” is like the “defund the police” of the alignment community. I’m not convinced it’s net bad because I haven’t been following governance—my current guess is neutral—but I do see these similarities:
It’s incredibly difficult and incentive-incompatible with existing groups in power
There are less costly, more effective steps to reduce the underlying problem, like making the field of alignment 10x larger or passing regulation to require evals
There are some obvious negative effects; potential overhangs or greater incentives to defect in the AI case, and increased crime, including against disadvantaged groups, in the police case
There’s far more discussion than action (I’m not counting the fact that GPT5 isn’t being trained yet; that’s for other reasons)
It’s memetically fit, and much discussion is driven by two factors that don’t advantage good policies over bad policies, and might even do the reverse. This is the toxoplasma of rage.
disagreement with the policy
(speculatively) intragroup signaling; showing your dedication to even an inefficient policy proposal proves you’re part of the ingroup. I’m not 100% this was a large factor in “defund the police” and this seems even less true with the FLI letter, but still worth mentioning.
This seems like a potentially unpopular take, so I’ll list some cruxes. I’d change my mind and endorse the letter if some of the following are true.
The claims above are mistaken/false somehow.
Top labs actually start taking beneficial actions towards the letter’s aims
It’s caused people to start thinking more carefully about AI risk
A 6 month pause now is especially important by setting anti-racing norms, demonstrating how far AI alignment is lagging behind capabilities, or something
A 6 month pause now is worth close to 6 months of alignment research at crunch time (my guess is that research at crunch time is worth 1.5x-3x more depending on whether MIRI is right about everything)
The most important quality to push towards in public discourse is how much we care about safety at all, so I should endorse this proposal even though it’s flawed
The obvious dis-analogy is that if the police had no funding and largely ceased to exist, a string of horrendous things would quickly occur. Murders and thefts and kidnappings and rapes and more would occur throughout every country in which it was occurring, people would revert to tight-knit groups who had weapons to defend themselves, a lot of basic infrastructure would probably break down (e.g. would Amazon be able to pivot to get their drivers armed guards?) and much more chaos would ensue.
And if AI research paused, society would continue to basically function as it has been doing so far.
One of them seems to me like a goal that directly causes catastrophes and a breakdown of society and the other doesn’t.
Fair point. Another difference is that the pause is popular! 66-69% in favor of the pause, and 41% think AI would do more harm than good vs 9% for more good than harm.
IMO making the field of alignment 10x larger or evals do not solve a big part of the problem, while indefinitely pausing AI development would. I agree it’s much harder, but I think it’s good to at least try, as long as it doesn’t terribly hurt less ambitious efforts (which I think it doesn’t).
This statement begs for cost-benefit analysis.
Increasing size of alignment field can be efficient, but it won’t be cheap. You need to teach new experts in the field that doesn’t have any polised standardized educational programs and doesn’t have much of teachers. If you want not only increase number of participants in the field, but increase productivity of the field 10x, you need an extraordinary educational effort.
Passing regulation to require evals seems like a meh idea. Nobody knows in enough details how to make such evalutions and every wrong idea that makes its way to law will be here until the end of the world.
I’d be much happier with increasing participants enough to equal 10-20% of the field of ML than a 6 month unconditional pause, and my guess is it’s less costly. It seems like leading labs allowing other labs to catch up by 6 months will reduce their valuations more than 20%, whereas diverting 10-20% of their resources would reduce valuations only 10% or so.
There are currently 300 alignment researchers. If we take additional researchers from the pool of 30k people who attended ICML, you get 3000 researchers, and if they’re equal quality this is 10x participants. I wouldn’t expect alignment to go 10x faster, more like 2x with a decent educational effort. But this is in perpetuity and should speed up alignment by far more than 6 months. There’s the question of getting labs to pay if they’re creating most of the harms, which might be hard though.
I’d be excited about someone doing a real cost-benefit analysis here, or preferably coming up with better ideas. It just seems so unlikely that a 6 month pause is close to the most efficient thing, given it destroys much of the value of a company that has a large lead.
I now think the majority of impact of AI pause advocacy will come from the radical flank effect, and people should study it to decide whether pause advocacy is good or bad.
Why does this have to be true? Can’t governments just compensate existing AGI labs for the expected commercial value of their foregone future advances due to indefinite pause?
This seems good if it could be done. But the original proposal was just a call for labs to individually pause their research, which seems really unlikely to work.
Also, the level of civilizational competence required to compensate labs seems to be higher than for other solutions. I don’t think it’s a common regulatory practice to compensate existing labs like this, and it seems difficult to work out all the details so that labs will feel adequately compensated. Plus there might be labs that irrationally believe they’re undervalued. Regulations similar to the nuclear or aviation industry feel like a more plausible way to get slowdown, and have the benefit that they actually incentivize safety work.