I think this is pretty false. There’s no equivalent to Let’s think about slowing down AI, or a tag like Restrain AI Development (both of which are advocating an even stronger claim than just “caution”) -- there’s a few paragraphs in Paul’s post, one short comment by me, and one short post by Kaj. I’d say that hardly any optimization has gone into arguments to AI safety researchers for advancing capabilities. [...] (I agree in the wider world there’s a lot more optimization for arguments in favor of capabilities progress that people in general would find compelling, but I don’t think that matters for “what should alignment researchers think”.)
Thanks for the reply! From what you’re saying here, it seems like we already agree that “in the wider world there’s a lot more optimization for arguments in favor of capabilities progress”.
I’m surprised to hear that you “don’t think that matters for “what should alignment researchers think”.”
Alignment researchers, are part of the wider world too! And conversely, a lot of people in the wider world that don’t work on alignment directly make relevant decisions that will affect alignment and AI, and think about alignment too (likely many more of those exist than “pure” alignment researchers, and this post is addressed to them too!)
I don’t buy this separation with the wider world. Most people involved in this live in social circles connected to AI development, they’re sensitive to status, many work at companies directly developing advanced AI systems, consume information from the broader world and so on. And the vast majority of the real world’s economy has so far been straightforwardly incentivizing reasons to develop new capabilities, faster. Here’s some tweets from Kelsey that illustrate some of this point.
conversely, a lot of people in the wider world that don’t work on alignment directly make relevant decisions that will affect alignment and AI, and think about alignment too (likely many more of those exist than “pure” alignment researchers, and this post is addressed to them too!)
Your post is titled “Don’t accelerate problems you’re trying to solve”. Given that the problem you’re considering is “misalignment”, I would have thought that the people trying to solve the problem are those who work on alignment.
The first sentence of your post is “If one believes that unaligned AGI is a significant problem (>10% chance of leading to catastrophe), speeding up public progress towards AGI is obviously bad.” This is a foundational assumption for the rest of your post. I don’t really know who you have in mind as these other people, but I would guess that they don’t assign >10% chance of catastrophe.
The people you cite as making the arguments you disagree with are full-time alignment researchers.
If you actually want to convey your points to some other audience I’d recommend making another different post that doesn’t give off the strong impression that it is talking to full-time alignment researchers.
I don’t buy this separation with the wider world. Most people involved in this live in social circles connected to AI development, they’re sensitive to status, many work at companies directly developing advanced AI systems, consume information from the broader world and so on.
I agree that status and “what my peers believe” determine what people do to a great extent. If you had said “lots of alignment researchers are embedded in communities where capabilities work is high-status; they should be worried that they’re being biased towards capabilities work as a result”, I wouldn’t have objected.
You also point out that people hear arguments from the broader world, but it seems like arguments from the community are way way way more influential on their beliefs than the ones from the broader world. (For example, they think there’s >10% chance of catastrophe from AI based on argument from this community, despite the rest of the world arguing that this is dumb.)
the vast majority of the real world’s economy has so far been straightforwardly incentivizing reasons to develop new capabilities, faster. Here’s some tweets from Kelsey that illustrate some of this point.
I looked at the linked tweet and a few surrounding it and they seem completely unrelated? E.g. the word “capabilities” doesn’t appear at all (or its synonyms).
I’m guessing you mean Kelsey’s point that EAs go to orgs that think safety is easy because those are the ones that are hiring, but (a) that’s not saying that those EAs then work on capabilities and (b) that’s not talking about optimized arguments, but instead about selection bias in who is hiring.
Thanks for the reply! From what you’re saying here, it seems like we already agree that “in the wider world there’s a lot more optimization for arguments in favor of capabilities progress”.
I’m surprised to hear that you “don’t think that matters for “what should alignment researchers think”.”
Alignment researchers, are part of the wider world too! And conversely, a lot of people in the wider world that don’t work on alignment directly make relevant decisions that will affect alignment and AI, and think about alignment too (likely many more of those exist than “pure” alignment researchers, and this post is addressed to them too!)
I don’t buy this separation with the wider world. Most people involved in this live in social circles connected to AI development, they’re sensitive to status, many work at companies directly developing advanced AI systems, consume information from the broader world and so on. And the vast majority of the real world’s economy has so far been straightforwardly incentivizing reasons to develop new capabilities, faster. Here’s some tweets from Kelsey that illustrate some of this point.
Your post is titled “Don’t accelerate problems you’re trying to solve”. Given that the problem you’re considering is “misalignment”, I would have thought that the people trying to solve the problem are those who work on alignment.
The first sentence of your post is “If one believes that unaligned AGI is a significant problem (>10% chance of leading to catastrophe), speeding up public progress towards AGI is obviously bad.” This is a foundational assumption for the rest of your post. I don’t really know who you have in mind as these other people, but I would guess that they don’t assign >10% chance of catastrophe.
The people you cite as making the arguments you disagree with are full-time alignment researchers.
If you actually want to convey your points to some other audience I’d recommend making another different post that doesn’t give off the strong impression that it is talking to full-time alignment researchers.
I agree that status and “what my peers believe” determine what people do to a great extent. If you had said “lots of alignment researchers are embedded in communities where capabilities work is high-status; they should be worried that they’re being biased towards capabilities work as a result”, I wouldn’t have objected.
You also point out that people hear arguments from the broader world, but it seems like arguments from the community are way way way more influential on their beliefs than the ones from the broader world. (For example, they think there’s >10% chance of catastrophe from AI based on argument from this community, despite the rest of the world arguing that this is dumb.)
I looked at the linked tweet and a few surrounding it and they seem completely unrelated? E.g. the word “capabilities” doesn’t appear at all (or its synonyms).
I’m guessing you mean Kelsey’s point that EAs go to orgs that think safety is easy because those are the ones that are hiring, but (a) that’s not saying that those EAs then work on capabilities and (b) that’s not talking about optimized arguments, but instead about selection bias in who is hiring.