Really happy to see the Anthropic letter. It clearly states their key views on AI risk and the potential benefits of SB 1047. Their concerns seem fair to me: overeager enforcement of the law could be counterproductive. While I endorse the bill on the whole and wish they would too (and I think their lack of support for the bill is likely partially influenced by their conflicts of interest), this seems like a thoughtful and helpful contribution to the discussion.
Agreed, sloppy phrasing on my part. The letter clearly states some of Anthropic’s key views, but doesn’t discuss other important parts of their worldview. Overall this is much better than some of their previous communications and the OpenAI letter, so I think it deserves some praise, but your caveat is also important.
It’s hard for me to reconcile “we take catastrophic risks seriously”, “we believe they could occur within 1-3 years”, and “we don’t believe in pre-harm enforcement or empowering an FMD to give the government more capacity to understand what’s going on.”
It’s also notable that their letter does not mention misalignment risks (and instead only points to dangerous cyber or bio capabilities).
That said, I do like this section a lot:
Catastrophic risks are important to address. AI obviously raises a wide range of issues, but in our assessment catastrophic risks are the most serious and the least likely to be addressed well by the market on its own.As noted earlier in this letter, we believe AI systems are going to develop powerful capabilities in domains like cyber and bio which could be misused– potentially in as little as 1-3 years. In theory, these issues relate to national security and might be best handled at the federal level, but in practice we are concerned that Congressional action simply will not occur in the necessary window of time. It is also possible for California to implement its statutes and regulations in a way that benefits from federal expertise in national security matters: for example the NIST AI Safety Institute will likely develop non-binding guidance on national security risks based on its collaboration with AI companies including Anthropic, which California can then utilize in its own regulations.
I think you’re eliding the difference between “powerful capabilities” being developed, the window of risk, and the best solution.
For example, if Anthropic believes “_we_ will have it internally in 1-3 years, but no small labs will, and we can contain it internally” then they might conclude that the warrant for a state-level FMD is low. Alternatively, you might conclude, “we will have it internally in 1-3 years, other small labs will be close behind, and an American state’s capabilities won’t be sufficient, we need DoD, FBI, and IC authorities to go stompy on this threat”, and thus think a state-level FMD is low-value-add.
Very unsure I agree with either of these hypos to be clear! Just trying to explore the possibility space and point out this is complex.
Really happy to see the Anthropic letter. It clearly states their key views on AI risk and the potential benefits of SB 1047. Their concerns seem fair to me: overeager enforcement of the law could be counterproductive. While I endorse the bill on the whole and wish they would too (and I think their lack of support for the bill is likely partially influenced by their conflicts of interest), this seems like a thoughtful and helpful contribution to the discussion.
Really? The letter just talks about catastrophic misuse risk, which I hope is not representative of Anthropic’s actual priorities.
I think the letter is overall good, but this specific dimension seems like among the weakest parts of the letter.
Agreed, sloppy phrasing on my part. The letter clearly states some of Anthropic’s key views, but doesn’t discuss other important parts of their worldview. Overall this is much better than some of their previous communications and the OpenAI letter, so I think it deserves some praise, but your caveat is also important.
It’s hard for me to reconcile “we take catastrophic risks seriously”, “we believe they could occur within 1-3 years”, and “we don’t believe in pre-harm enforcement or empowering an FMD to give the government more capacity to understand what’s going on.”
It’s also notable that their letter does not mention misalignment risks (and instead only points to dangerous cyber or bio capabilities).
That said, I do like this section a lot:
I think you’re eliding the difference between “powerful capabilities” being developed, the window of risk, and the best solution.
For example, if Anthropic believes “_we_ will have it internally in 1-3 years, but no small labs will, and we can contain it internally” then they might conclude that the warrant for a state-level FMD is low. Alternatively, you might conclude, “we will have it internally in 1-3 years, other small labs will be close behind, and an American state’s capabilities won’t be sufficient, we need DoD, FBI, and IC authorities to go stompy on this threat”, and thus think a state-level FMD is low-value-add.
Very unsure I agree with either of these hypos to be clear! Just trying to explore the possibility space and point out this is complex.