Thank you for providing a nice overview of our Frontier AI Regulation: Managing Emerging Risks to Public Safety that was just released!
I appreciate your feedback, both the positive and critical parts. I’m also glad you think the paper should exist and that it is mostly a good step. And, I think your criticism is fair. Let me also note that I do not speak for the authorship team. We are quite a diverse group from academia, labs, industry, nonprofits, etc. It was no easy task to find common ground across everyone involved.
I think the AI Governance space is difficult in part because different political actors have different goals, even when sharing significant overlap in interests. As I saw it, the goal of this paper was to bring together a wide group of interested individuals and organizations to see if we could come to points of agreement on useful immediate next governance steps. In this way, we weren’t seeking “ambitious” new policy tools, we were seeking for areas of agreement across the diverse stakeholders currently driving change in the AI development space. I think this is a significantly different goal than the Model Evaluation for Extreme Risks paper that you mention, which I agree is another important entry in this space. Additionally, one of the big differences, I think, between our effort and the model evaluation paper, is we are more focused on what governments in particular should consider doing from their available toolkits, where it seems to me that model evaluation paper is more about what companies and labs themselves should do.
A couple of other thoughts:
I don’t think it’s completely accurate that “It doesn’t suggest government oversight of training runs or compute.” As part of the suggestion around licensing we mention that the AI development process may require oversight by an agency. But, in fairness, it’s not a point that we emphasize.
I think the following is a little unfair. You say: “This is overdeterminedly insufficient for safety. “Not complying with mandated standards and ignoring repeated explicit instructions from a regulator” should not be allowed to happen, because it might kill everyone. A single instance of noncompliance should not be allowed to happen, and requires something like oversight of training runs to prevent. Not to mention that denying market access or threatening prosecution are inadequate. Not to mention that naming-and-shaming and fining companies are totally inadequate. This passage totally fails to treat AI as a major risk. I know the authors are pretty worried about x-risk; I notice I’m confused.” Let me explain below.
I’m not sure there’s such a thing as “perfect compliance.” I know of no way to ensure that “a single instance of noncompliance should not be allowed to happen.” And, I don’t think that’s necessary for current models or even very near term future models. I think the idea here is that we setup a standard regulatory process in advance of AI models that might be capable enough to kill everyone and shape the development of the next sets of frontier models. I do think there’s certainly a criticism here that naming and shaming, for example, is not a sufficiently punitive tool, but may have more impact on leading AI labs that one might assume.
I hope this helps clear up some of your confusion here. To recap: I think your criticism that the tools are not ambitious is fair. I don’t think that was our goal. I saw this project as a way of providing tools for which there is broad agreement and that given the current state of AI models we believe would help steer AI development and deployment in a better direction. I do think that another reading of this paper is that it’s quite significant that this group agreed on the recommendations that are made. I consider it progress in the discussion of how to effectively govern increasingly power AI models, but it’s not the last word either. :)
Thanks again for sharing and for providing you feedback on these very important questions of governance.
Thanks for your reply. In brief response to your more specific points:
On government oversight, I think you’re referring to the quote “providing a regulator the power to oversee model development could also promote regulatory visibility, thus allowing regulations to adapt more quickly.” But the paper doesn’t seem to mention the direct benefit of oversight: verifying compliance and enforcing the rules. Good oversight would result in licensing not being a one-time thing but rather that labs could lose their licenses during a training run if they were noncompliant. (In my community ‘oversight of training runs’ means government auditors verifying compliance and the government stopping noncompliant runs; maybe it means something weaker outside my community.)
I agree that “perfect compliance” is hard but stand by my disappointment in the “particularly egregious instances” passage as not aiming high enough,
Thank you for providing a nice overview of our Frontier AI Regulation: Managing Emerging Risks to Public Safety that was just released!
I appreciate your feedback, both the positive and critical parts. I’m also glad you think the paper should exist and that it is mostly a good step. And, I think your criticism is fair. Let me also note that I do not speak for the authorship team. We are quite a diverse group from academia, labs, industry, nonprofits, etc. It was no easy task to find common ground across everyone involved.
I think the AI Governance space is difficult in part because different political actors have different goals, even when sharing significant overlap in interests. As I saw it, the goal of this paper was to bring together a wide group of interested individuals and organizations to see if we could come to points of agreement on useful immediate next governance steps. In this way, we weren’t seeking “ambitious” new policy tools, we were seeking for areas of agreement across the diverse stakeholders currently driving change in the AI development space. I think this is a significantly different goal than the Model Evaluation for Extreme Risks paper that you mention, which I agree is another important entry in this space. Additionally, one of the big differences, I think, between our effort and the model evaluation paper, is we are more focused on what governments in particular should consider doing from their available toolkits, where it seems to me that model evaluation paper is more about what companies and labs themselves should do.
A couple of other thoughts:
I don’t think it’s completely accurate that “It doesn’t suggest government oversight of training runs or compute.” As part of the suggestion around licensing we mention that the AI development process may require oversight by an agency. But, in fairness, it’s not a point that we emphasize.
I think the following is a little unfair. You say: “This is overdeterminedly insufficient for safety. “Not complying with mandated standards and ignoring repeated explicit instructions from a regulator” should not be allowed to happen, because it might kill everyone. A single instance of noncompliance should not be allowed to happen, and requires something like oversight of training runs to prevent. Not to mention that denying market access or threatening prosecution are inadequate. Not to mention that naming-and-shaming and fining companies are totally inadequate. This passage totally fails to treat AI as a major risk. I know the authors are pretty worried about x-risk; I notice I’m confused.” Let me explain below.
I’m not sure there’s such a thing as “perfect compliance.” I know of no way to ensure that “a single instance of noncompliance should not be allowed to happen.” And, I don’t think that’s necessary for current models or even very near term future models. I think the idea here is that we setup a standard regulatory process in advance of AI models that might be capable enough to kill everyone and shape the development of the next sets of frontier models. I do think there’s certainly a criticism here that naming and shaming, for example, is not a sufficiently punitive tool, but may have more impact on leading AI labs that one might assume.
I hope this helps clear up some of your confusion here. To recap: I think your criticism that the tools are not ambitious is fair. I don’t think that was our goal. I saw this project as a way of providing tools for which there is broad agreement and that given the current state of AI models we believe would help steer AI development and deployment in a better direction. I do think that another reading of this paper is that it’s quite significant that this group agreed on the recommendations that are made. I consider it progress in the discussion of how to effectively govern increasingly power AI models, but it’s not the last word either. :)
Thanks again for sharing and for providing you feedback on these very important questions of governance.
Thanks for your reply. In brief response to your more specific points:
On government oversight, I think you’re referring to the quote “providing a regulator the power to oversee model development could also promote regulatory visibility, thus allowing regulations to adapt more quickly.” But the paper doesn’t seem to mention the direct benefit of oversight: verifying compliance and enforcing the rules. Good oversight would result in licensing not being a one-time thing but rather that labs could lose their licenses during a training run if they were noncompliant. (In my community ‘oversight of training runs’ means government auditors verifying compliance and the government stopping noncompliant runs; maybe it means something weaker outside my community.)
I agree that “perfect compliance” is hard but stand by my disappointment in the “particularly egregious instances” passage as not aiming high enough,
Edit: also I get that finding consensus is hard but after reading the consensus-y-but-ambitious Towards Best Practices in AGI Safety and Governance and Model evaluation for extreme risks I was expecting consensus on something stronger.
Thanks for the response! I appreciate the clarification on both point 1 and 2 above. I think they’re fair criticisms. Thanks for pointing them out.