Let’s look at the two horns of the dilemma, as you put it:
Why do many people who want to pause AI not support the organization “PauseAI”?
Why would the organization “PauseAI” not change itself so that people who want to pause AI can support it?
Well, here are some reasons someone who wants pause AI might not want to support the organization PauseAI:
When you visit the website for PauseAI, you might find some very steep proposals for Pausing AI—such as requiring the “Granting [of] approval for new training runs of AI models above a certain size (e.g. 1 billion parameters)” or “Banning the publication of such algorithms” that improve AI performance or prohibiting the training of models that “are expected to exceed a score of 86% on the MMLU benchmark” unless their safety can be guaranteed. Implementing these measures would be really hard—a one-billion parameter model is quite small (I could train one); banning the publication of information on this stuff would be considered by many an infringement on freedom of speech; and there are tons of models now that do better than 86% on the MMLU and have done no harm.
So, if you think the specific measures proposed by them would limit an AI that even many pessimists would think is totally ok and almost risk-free, then you might not want to push for these proposals but for more lenient proposals that, because they are more lenient, might actually get implemented. To stop asking for the sky and actually get something concrete.
If you look at the kind of claims that PauseAI makes in their risks page, you might believe that some of them seem exaggerated, or that PauseAI is simply throwing all the negative things they can find about AI into big list to make it seem bad. If you think that credibility is important to the effort to pause AI, then PauseAI might seem very careless about truth in a way that could backfire.
So, this is why people who want to pause AI might not want to support PauseAI.
And, well, why wouldn’t pause AI want to change?
Well—I’m gonna speak broadly—if you look at the history of PauseAI, they are marked by belief that the measures proposed by others are insufficient for Actually Stopping AI—for instance the kind of policy measures proposed by people working at AI companies isn’t enough; that the kind of measures proposed by people funded by OpenPhil are often not enough; and so on. Similarly, they often believe that people who are talking about these claims are nitpicking, and so on. (Citation needed.)
I don’t think this dynamic is rare. Many movements have “radical wings,” that more moderate organizations in the movement would characterize as having impracticable maximalist policy goals and careless epistemics. And the radical wings would of course criticize back that the “moderate wings” have insufficient or cowardly policy goals and epistemics optimized for respectability and not not truth. And the conflicts between them are intractable because people cannot move away from these prior beliefs about their interlocutors; in this respect the discourse around PauseAI seems unexceptionable and rather predictable.
Well—I’m gonna speak broadly—if you look at the history of PauseAI, they are marked by belief that the measures proposed by others are insufficient for Actually Stopping AI—for instance the kind of policy measures proposed by people working at AI companies isn’t enough; that the kind of measures proposed by people funded by OpenPhil are often not enough; and so on.
They are correct as far as I can tell. Can you identify a policy measure proposed by an AI company or an OpenPhil-funded org that you think would be sufficient to stop unsafe AI development?
I think there is indeed exactly one such policy measure, which is SB 1047, supported by Center for AI Safety which is OpenPhil-funded (IIRC), which most big AI companies lobbied against, and Anthropic opposed the original stronger version and got it reduced to a weaker and probably less-safe version.
When I wrote where I was donating in 2024 I went through a bunch of orgs’ policy proposals and explained why they appear deeply inadequate. Some specific relevant parts: 1, 2, 3, 4
Edit: Adding some color so you don’t have to click through– when I say the proposals I reviewed were inadequate, I mean they said things like (paraphrasing) “safety should be done on a completely voluntary basis with no government regulations” and “companies should have safety officers but those officers should not have final say on anything”, and would simply not address x-risk at all, or would make harmful proposals like “the US Department of Defense should integrate more AI into its weapon systems” or “we need to stop worrying about x-risk because it’s distracting from the real issues”.
“sufficient to stop unsafe AI development? I think there is indeed exactly one such policy measure, which is SB 1047,”
I think it’s obviously untrue that this would stop unsafe AI—it is as close as any measure I’ve seen, and would provide some material reduction in risk in the very near term, but (even if applied universally, and no-one tried to circumvent it,) it would not stop future unsafe AI.
Yeah I actually agree with that, I don’t think it was sufficient, I just think it was pretty good. I wrote the comment too quickly without thinking about my wording.
Disagree that it could stop dangerous work, and doubly disagree given the way things are headed, especially with removing whistleblower protections and the lack of useful metrics for compliance. I don’t think it would even be as good as SB-1047, even in the amended weaker form.
I was previously more hopeful that if the EU COP was a strong enough code, then when things inevitably went poorly anyways we could say “look, doing pretty good isn’t enough, we need to actually regulate specific parts of this dangerous technology,” but I worry that it’s not even going to be strong enough to make that argument.
If you look at the kind of claims that PauseAI makes in their risks page, you might believe that some of them seem exaggerated, or that PauseAI is simply throwing all the negative things they can find about AI into big list to make it see bad. If you think that credibility is important to the effort to pause AI, then PauseAI might seem very careless about truth in a way that could backfire.
A couple notes on this:
AFAICT PauseAI US does not do the thing you describe.
I’ve looked at a good amount of research on protest effectiveness. There are many observational studies showing that nonviolent protests are associated with preferred policy changes / voting patterns, and ~four natural experiments. If protests backfired for fairly minor reasons like “their website makes some hard-to-defend claims” (contrasted with major reasons like “the protesters are setting buildings on fire”), I think that would show up in the literature, and it doesn’t.
I’ve looked at a good amount of research on protest effectiveness. There are many observational studies showing that nonviolent protests are associated with preferred policy changes / voting patterns, and ~four natural experiments. If protests backfired for fairly minor reasons like “their website makes some hard-to-defend claims” (contrasted with major reasons like “the protesters are setting buildings on fire”), I think that would show up in the literature, and it doesn’t.
I’m not trying to get into the object level here. But people could both:
Believe that making such hard-to-defend claims could backfire, disagreeing with those experiments that you point out or
Believe that making such claims violates virtue-ethics-adjacent commitments to truth or
Just not want to be associated, in an instinctive yuck kinda way, with people who make these kinds of dubious-to-them claims.
Of course people could be wrong about the above points. But if you believed these things, then they’d be intelligible reasons not to be associated with someone, and I think a lot of the claims PauseAI makes are such that a large number of people people would have these reactions.
When you visit the website for PauseAI, you might find some very steep proposals for Pausing AI [...] (I could train one)
Their website is probably outdated. I read their proposals as “keep the current level of AI, regulate stronger AI”. Banning current LLaMA models seems silly from an x-risk perspective, in hindsight. I think PauseAI is perfectly fine with pausing “too early”, which I personally don’t object to.
If you look at the kind of claims that PauseAI makes in their risks page
PauseAI is clearly focused on x-risk. The risks page seems like an attempt to guide the general public from naively-realistic “Present dangers” slowly towards introducing (exotic-sounding) x-risk. You can disagree with that approach, of course. I would disagree that mixing AI Safety and AI Ethics is being “very careless about truth”.
Thank you for answering my question! I wanted to know what you people think about PauseAI, so this fits well.
[...] in this respect the discourse around PauseAI seems unexceptionable and rather predictable.
Yes. I hope we can be better at coordination… I would frame PauseAI as “the reasonable [aspiring] mass-movement”. I like that it is easy to support or join PauseAI even without having an ML PhD. StopAI is an organization more radical than them.
Let’s look at the two horns of the dilemma, as you put it:
Why do many people who want to pause AI not support the organization “PauseAI”?
Why would the organization “PauseAI” not change itself so that people who want to pause AI can support it?
Well, here are some reasons someone who wants pause AI might not want to support the organization PauseAI:
When you visit the website for PauseAI, you might find some very steep proposals for Pausing AI—such as requiring the “Granting [of] approval for new training runs of AI models above a certain size (e.g. 1 billion parameters)” or “Banning the publication of such algorithms” that improve AI performance or prohibiting the training of models that “are expected to exceed a score of 86% on the MMLU benchmark” unless their safety can be guaranteed. Implementing these measures would be really hard—a one-billion parameter model is quite small (I could train one); banning the publication of information on this stuff would be considered by many an infringement on freedom of speech; and there are tons of models now that do better than 86% on the MMLU and have done no harm.
So, if you think the specific measures proposed by them would limit an AI that even many pessimists would think is totally ok and almost risk-free, then you might not want to push for these proposals but for more lenient proposals that, because they are more lenient, might actually get implemented. To stop asking for the sky and actually get something concrete.
If you look at the kind of claims that PauseAI makes in their risks page, you might believe that some of them seem exaggerated, or that PauseAI is simply throwing all the negative things they can find about AI into big list to make it seem bad. If you think that credibility is important to the effort to pause AI, then PauseAI might seem very careless about truth in a way that could backfire.
So, this is why people who want to pause AI might not want to support PauseAI.
And, well, why wouldn’t pause AI want to change?
Well—I’m gonna speak broadly—if you look at the history of PauseAI, they are marked by belief that the measures proposed by others are insufficient for Actually Stopping AI—for instance the kind of policy measures proposed by people working at AI companies isn’t enough; that the kind of measures proposed by people funded by OpenPhil are often not enough; and so on. Similarly, they often believe that people who are talking about these claims are nitpicking, and so on. (Citation needed.)
I don’t think this dynamic is rare. Many movements have “radical wings,” that more moderate organizations in the movement would characterize as having impracticable maximalist policy goals and careless epistemics. And the radical wings would of course criticize back that the “moderate wings” have insufficient or cowardly policy goals and epistemics optimized for respectability and not not truth. And the conflicts between them are intractable because people cannot move away from these prior beliefs about their interlocutors; in this respect the discourse around PauseAI seems unexceptionable and rather predictable.
They are correct as far as I can tell. Can you identify a policy measure proposed by an AI company or an OpenPhil-funded org that you think would be sufficient to stop unsafe AI development?
I think there is indeed exactly one such policy measure, which is SB 1047, supported by Center for AI Safety which is OpenPhil-funded (IIRC), which most big AI companies lobbied against, and Anthropic opposed the original stronger version and got it reduced to a weaker and probably less-safe version.When I wrote where I was donating in 2024 I went through a bunch of orgs’ policy proposals and explained why they appear deeply inadequate. Some specific relevant parts: 1, 2, 3, 4
Edit: Adding some color so you don’t have to click through– when I say the proposals I reviewed were inadequate, I mean they said things like (paraphrasing) “safety should be done on a completely voluntary basis with no government regulations” and “companies should have safety officers but those officers should not have final say on anything”, and would simply not address x-risk at all, or would make harmful proposals like “the US Department of Defense should integrate more AI into its weapon systems” or “we need to stop worrying about x-risk because it’s distracting from the real issues”.
“sufficient to stop unsafe AI development? I think there is indeed exactly one such policy measure, which is SB 1047,”
I think it’s obviously untrue that this would stop unsafe AI—it is as close as any measure I’ve seen, and would provide some material reduction in risk in the very near term, but (even if applied universally, and no-one tried to circumvent it,) it would not stop future unsafe AI.
Yeah I actually agree with that, I don’t think it was sufficient, I just think it was pretty good. I wrote the comment too quickly without thinking about my wording.
EU AI Code of Practice is better, a little closer to stopping ai development
Disagree that it could stop dangerous work, and doubly disagree given the way things are headed, especially with removing whistleblower protections and the lack of useful metrics for compliance. I don’t think it would even be as good as SB-1047, even in the amended weaker form.
I was previously more hopeful that if the EU COP was a strong enough code, then when things inevitably went poorly anyways we could say “look, doing pretty good isn’t enough, we need to actually regulate specific parts of this dangerous technology,” but I worry that it’s not even going to be strong enough to make that argument.
A couple notes on this:
AFAICT PauseAI US does not do the thing you describe.
I’ve looked at a good amount of research on protest effectiveness. There are many observational studies showing that nonviolent protests are associated with preferred policy changes / voting patterns, and ~four natural experiments. If protests backfired for fairly minor reasons like “their website makes some hard-to-defend claims” (contrasted with major reasons like “the protesters are setting buildings on fire”), I think that would show up in the literature, and it doesn’t.
I’m not trying to get into the object level here. But people could both:
Believe that making such hard-to-defend claims could backfire, disagreeing with those experiments that you point out or
Believe that making such claims violates virtue-ethics-adjacent commitments to truth or
Just not want to be associated, in an instinctive yuck kinda way, with people who make these kinds of dubious-to-them claims.
Of course people could be wrong about the above points. But if you believed these things, then they’d be intelligible reasons not to be associated with someone, and I think a lot of the claims PauseAI makes are such that a large number of people people would have these reactions.
Their website is probably outdated. I read their proposals as “keep the current level of AI, regulate stronger AI”. Banning current LLaMA models seems silly from an x-risk perspective, in hindsight. I think PauseAI is perfectly fine with pausing “too early”, which I personally don’t object to.
PauseAI is clearly focused on x-risk. The risks page seems like an attempt to guide the general public from naively-realistic “Present dangers” slowly towards introducing (exotic-sounding) x-risk. You can disagree with that approach, of course. I would disagree that mixing AI Safety and AI Ethics is being “very careless about truth”.
Thank you for answering my question! I wanted to know what you people think about PauseAI, so this fits well.
Yes. I hope we can be better at coordination… I would frame PauseAI as “the reasonable [aspiring] mass-movement”. I like that it is easy to support or join PauseAI even without having an ML PhD. StopAI is an organization more radical than them.