From my vantage point the debate around open-sourcing foundation models became heated as Yann LeCun began advocating for open-sourcing (in particular) Meta’s foundation models. This prompted a knee-jerk reaction in the AI Safety community.
The arguments went something like “of course open-sourcing foundation models is a good idea, just LOOK at all the BENEFITS open-sourcing has given us!” for the “pro” crowd, and something like “of course open-sourcing foundation models is a terrible idea, just THINK about how it increases RISK” for the “anti” crowd.
Given this, I was excited to see the release of GovAI’s new report which, as Elizabeth A. Seger highlights in their brief summary, outlines both the noted benefits and risks of open-sourcing more generally, and how these benefits and risks might be applied in particular to foundation models. In this report titled “Open-Sourcing Highly Capable Foundation Models” Seger, along with her numerous co-authors walk us through these benefits and risks and also explore alternative policies that arguably provide similar benefits as open-sourcing while mitigating the risks of open-sourcing foundation models.
After reading that report, I have a few summarizing thoughts:
To open source or not to open source foundation models is a false dichotomy. Instead, there is a gradient of options to consider.
This is something that should be more obvious but does seem to be lost in the current debate. The gradient runs from fully closed to fully open and includes additional categories including gradual/staged release, hosted access, cloud-based/API access, and downloadable. (“Box 1: Further research is needed to define open-source gradients” from the paper illustrates this well.) It’s worth noting that even Meta’s Llama2 is not fully open.
Structured access seems to be a particularly useful option. It provides many of the benefits of fully open-sourcing while protecting against some of the risks of both fully closed models and fully open-sourced models.
“Structured access involves constructing, through technical and often bureaucratic means, a controlled interaction between an AI system and its user. The interaction is structured to both (a) prevent the user from using the system in a harmful way, whether intentional or unintentional, and (b) prevent the user from circumventing those restrictions by modifying or reproducing the system.”
The GovAI report echoes these benefits, and I find them compelling.
A rigorous and healthy ecosystem for auditing foundation models could alleviate substantial risks of open sourcing.
“outline the components of an initial internal audit frame- work, which can be framed as encompassing five distinct stages— Scoping, Mapping, Artifact Collection, Testing and Reflection (SMACTR)— all of which have their own set of documentation requirements and account for a different level of the analysis of a system.”
The report makes modest, cautious, and reasonable recommendations for governance.
The recommendations are as follows:
1. Developers and governments should recognise that some highly capable models will be too risky to open-source, at least initially. These models may become safe to open-source in the future as societal resilience to AI risk increases and improved safety mechanisms are developed.
2. Decisions about open-sourcing highly capable foundation models should be informed by rigorous risk assessments. In addition to evaluating models for dangerous capabilities and immediate misuse applications, risk assessments must consider how a model might be fine-tuned or otherwise amended to facilitate misuse.
3. Developers should consider alternatives to open-source release that capture some of the same [distributive, democratic, and societal] benefits, without creating as much risk. Some promising alternatives include gradual or “staged” model release, model access for researchers and auditors, and democratic oversight of AI development and governance decisions.
4. Developers, standards setting bodies, and open-source communities should engage in collaborative and multi-stakeholder efforts to define fine-grained standards for when model components should be released. These standards should be based on an understanding of the risks posed by releasing (different combinations of) model components.
5. Governments should exercise oversight of open source AI models and enforce safety measures when stakes are sufficiently high. AI developers may not voluntarily adopt risk assessment and model sharing standards. Governments will need to enforce such measures through options such as liability law and regulation (e.g. via licensing requirements, fines, or penalties). Governments will also need to build the capacity to enforce such oversight mechanisms effectively.
Here, I’d also like to point to another post titled “Navigating the Open-Source AI Landscape: Data, Funding, and Safety” from April of this year that also points at company-centered recommendations that somewhat overlap with the GovAI report recommendations. These recommendations focus more specifically on what developers and companies can do (rather than governments), but I think the list is a good one for developers to be considering as well. Their 10 recommendations are:
Prioritize safety over speed in publication. Focus on AI models’ alignment before release to minimize risks.
Regularly audit and update models to reduce risks and maintain AI alignment.
Seek external feedback through red teams, beta-testers, and third parties to identify potential issues before release.
Favor structured access to ensure that only authorized users with proper understanding can deploy the technology.
Avoid developing agentic systems to reduce the risks of autonomous decision-making misaligned with human values.
Communicate model’s limitations, helping users make informed decisions and reducing unintended consequences.
Omit harmful datasets during training, preventing undesirable content outputs (e.g. gpt-4chan was trained on 4chan’s politically-incorrect board.)
Enhance open-source content moderation datasets, such as OIG-moderation, to develop more robust and safer AI systems.
Develop an infohazard policy, like this one by Conjecture, to prevent the leakage of information that could accelerate AGI.
I really appreciate and endorse the conclusion of the report.
The authors of the report conclude with the following:
“Overall, openness, transparency, accessibility, and wider community input are key to facilitating a future for beneficial AI. The goal of this paper is therefore not to argue that foundation model development should be kept behind closed doors. Model sharing, including open-sourcing, remains a valuable practice in most cases. Rather, we submit that decisions to open-source increasingly capable models must be considered with great care. Comprehensive risk assessments and careful consideration of alternative methods for pursuing open-source objectives are minimum first steps.”
I enjoyed reading this article that at least attempts to place the benefits and risks of OS side-by-side and discuss how they might be applied to the context of foundation models rather than the oversimplifications that have been dominating this discussion.
In addition to my thoughts above, I’m wondering whether concerned communities could provide insight on the following two questions:
How large are the risks from fully open-sourced foundation models? More specifically, how significant are these risks compared to the overall risks inherent in the development and deployment of foundation models?
It seems to me that many of the risks identified in the GovAI report are not that distinguishable from the inherent risks of the development and widespread deployment of foundation models. Does keeping foundation models more closed actually help prevent some of the more serious risks presented by the development and deployment of more capable foundation models?
Is there any reasonable way to prevent leaks in a world with stricter regulation of fully OS foundation models?
I don’t have a good sense of how easy it would be for a company or even a rogue employee to leak weights and architecture, other than it has already been done at least once.
Thanks again to the authors of this paper for providing a detailed and nuanced treatment of this topic, and many thanks to GovAI for sponsoring this important and interesting work! I would be very interested in any additional thoughts on these two questions.
To open-source or to not open-source, that is (an oversimplification of) the question.
*** This an edited and expanded version of a post I made on X in response to GovAI’s new report“Open-Sourcing Highly Capable Foundation Models” I think the report points in the right direction, but also leaves me with some additional questions. Also, thanks for significant feedback from @David_Kristoffersson, @Elliot_Mckernon, @Corin Katzke, and @cwdicarlo ***
From my vantage point the debate around open-sourcing foundation models became heated as Yann LeCun began advocating for open-sourcing (in particular) Meta’s foundation models. This prompted a knee-jerk reaction in the AI Safety community.
The arguments went something like “of course open-sourcing foundation models is a good idea, just LOOK at all the BENEFITS open-sourcing has given us!” for the “pro” crowd, and something like “of course open-sourcing foundation models is a terrible idea, just THINK about how it increases RISK” for the “anti” crowd.
Given this, I was excited to see the release of GovAI’s new report which, as Elizabeth A. Seger highlights in their brief summary, outlines both the noted benefits and risks of open-sourcing more generally, and how these benefits and risks might be applied in particular to foundation models. In this report titled “Open-Sourcing Highly Capable Foundation Models” Seger, along with her numerous co-authors walk us through these benefits and risks and also explore alternative policies that arguably provide similar benefits as open-sourcing while mitigating the risks of open-sourcing foundation models.
After reading that report, I have a few summarizing thoughts:
To open source or not to open source foundation models is a false dichotomy. Instead, there is a gradient of options to consider.
This is something that should be more obvious but does seem to be lost in the current debate. The gradient runs from fully closed to fully open and includes additional categories including gradual/staged release, hosted access, cloud-based/API access, and downloadable. (“Box 1: Further research is needed to define open-source gradients” from the paper illustrates this well.) It’s worth noting that even Meta’s Llama2 is not fully open.
Structured access seems to be a particularly useful option. It provides many of the benefits of fully open-sourcing while protecting against some of the risks of both fully closed models and fully open-sourced models.
The report cites work from Toby Shevlane, including his paper “Structured access: an emerging paradigm for safe AI deployment” which is also a chapter in The Oxford Handbook of AI Governance. Shevlane describes the idea of structured access in the following way:
The GovAI report echoes these benefits, and I find them compelling.
A rigorous and healthy ecosystem for auditing foundation models could alleviate substantial risks of open sourcing.
The report references work by Deb Raji and colleagues titled “Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing” The authors
The report makes modest, cautious, and reasonable recommendations for governance.
The recommendations are as follows:
Here, I’d also like to point to another post titled “Navigating the Open-Source AI Landscape: Data, Funding, and Safety” from April of this year that also points at company-centered recommendations that somewhat overlap with the GovAI report recommendations. These recommendations focus more specifically on what developers and companies can do (rather than governments), but I think the list is a good one for developers to be considering as well. Their 10 recommendations are:
I really appreciate and endorse the conclusion of the report.
The authors of the report conclude with the following:
I enjoyed reading this article that at least attempts to place the benefits and risks of OS side-by-side and discuss how they might be applied to the context of foundation models rather than the oversimplifications that have been dominating this discussion.
In addition to my thoughts above, I’m wondering whether concerned communities could provide insight on the following two questions:
How large are the risks from fully open-sourced foundation models? More specifically, how significant are these risks compared to the overall risks inherent in the development and deployment of foundation models?
It seems to me that many of the risks identified in the GovAI report are not that distinguishable from the inherent risks of the development and widespread deployment of foundation models. Does keeping foundation models more closed actually help prevent some of the more serious risks presented by the development and deployment of more capable foundation models?
Is there any reasonable way to prevent leaks in a world with stricter regulation of fully OS foundation models?
I don’t have a good sense of how easy it would be for a company or even a rogue employee to leak weights and architecture, other than it has already been done at least once.
Thanks again to the authors of this paper for providing a detailed and nuanced treatment of this topic, and many thanks to GovAI for sponsoring this important and interesting work! I would be very interested in any additional thoughts on these two questions.