Yudkowksy writes in his AGI Ruin post: “We can’t just “decide not to build AGI” because GPUs are everywhere...”
Is anyone thinking seriously about how we might bring it about such that we coordinate globally to not build AGI (at least until we’re confident we can do so safely)? If so, who? If not, why not? It seems like something we should at least try to do, especially if the situation is as dire as Yudkowsky thinks. The sort of thing I’m thinking of is (and this touches on points others have made in their questions):
international governance/regulation
start a protest movement against building AI
do lots of research and thinking about rhetoric and communication and diplomacy, find some extremely charming and charismatic people to work on this, and send them to persuade all actors capable of building AGI to not do it (and to do everything they can to prevent others from doing it)
as someone suggested in another question, translate good materials on why people are concerned about AI safety into Mandarin and other languages
more popularising of AI concerns in English
To be clear, I’m not claiming that this will be easy—this is not a “why don’t we just-” point. I agree with the things Yudkowsky says in that paragraph about why it would be difficult. I’m just saying that it’s not obvious to me that this is fundamentally intractable or harder than solving the technical alignment problem. Reasons for relative optimism:
we seem to achieved some international cooperation around nuclear weapons—isn’t it theoretically possible to do so around AGI?
there are lots of actors who could build AGIs, but it’s still a limited number. Larger groups of actors do cooperate.
through negotiation and diplomacy, people successfully persuade other people to do stuff that’s not even in their interest.AI safety should be a much easier sell because if developing AGI is really dangerous, it’s in everyone’s interest to stop developing it. There are coordination problems to be sure, but the fact remains that the AI safety ‘message’ is fundamentally ‘if you stop doing this we won’t all die’
Nuclear weapons seem like a relatively easy case, in that they require a massive investment to build, are basically of interest only to nation-states, and ultimately don’t provide any direct economic benefit. Regulating AI development looks more similar to something like restricting climate emissions: many different actors could create it, all nations could benefit (economically and otherwise) from continuing to develop it, and the risks of it seem speculative and unproven to many people.
And while there have been significant efforts to restrict climate emissions, there’s still significant resistance to that as well—with it having taken decades for us to get to the current restriction treaties, which many people still consider insufficient.
Given the obvious long-term risks associated with AGI development, is it feasible that governments might enact legislation intended to stop AI from being developed? Surely government regulatory bodies would slow down the progress of AGI development in order to enable measured development of accompanying ethical tools, practices, and understandings? This however seems unlikely, for the following reasons.
Let us consider two cases separately. First, there is the case of banning AGI research and development after an “AGI Sputnik” moment has occurred. We define an AGI Sputnik moment as a technological achievement that makes the short- to medium-term possibility of highly functional and useful human-level AGI broadly evident to the public and policy makers, bringing it out of the realm of science fiction to reality. Second, we might choose to ban it before such a moment has happened.
After an AGI Sputnik moment, even if some nations chose to ban AI technology due to the perceived risks, others would probably proceed eagerly with AGI development because of the wide-ranging perceived benefits. International agreements are difficult to reach and enforce, even for extremely obvious threats like nuclear weapons and pollution, so it’s hard to envision that such agreements would come rapidly in the case of AGI. In a scenario where some nations ban AGI while others do not, it seems the slow speed of international negotiations would contrast with the rapid speed of development of a technology in the midst of revolutionary breakthrough. While worried politicians sought to negotiate agreements, AGI development would continue, and nations would gain increasing competitive advantage from their differential participation in it.
The only way it seems feasible for such an international ban to come into play, would be if the AGI Sputnik moment turned out to be largely illusory because the path from the moment to full human-level AGI turned out to be susceptible to severe technical bottlenecks. If AGI development somehow slowed after the AGI Sputnik moment, then there might be time for the international community to set up a system of international treaties similar to what we now have to control nuclear weapons research. However, we note that the nuclear weapons research ban is not entirely successful – and that nuclear weapons development and testing tend to have large physical impacts that are remotely observable by foreign nations. On the other hand, if a nation decides not to cooperate with an international AGI ban, this would be much more difficult for competing nations to discover.
An unsuccessful attempt to ban AGI research and development could end up being far riskier than no ban. An international R&D ban that was systematically violated in the manner of current international nuclear weapons bans would shift AGI development from cooperating developed nations to “rogue nations,” thus slowing down AGI development somewhat, but also perhaps decreasing the odds of the first AGI being developed in a manner that is concerned with ethics and Friendly AI.
Thus, subsequent to an AGI Sputnik moment, the overall value of AGI will be too obvious for AGI to be effectively banned, and monitoring AGI development would be next to impossible.
The second option is an AGI R&D ban earlier than the AGI Sputnik moment – before it’s too late. This also seems infeasible, for the following reasons:
• Early stage AGI technology will supply humanity with dramatic economic and quality of life improvements, as narrow AI does now. Distinguishing narrow AI from AGI from a government policy perspective would also be prohibitively difficult.
• If one nation chose to enforce such a slowdown as a matter of policy, the odds seem very high that other nations would explicitly seek to accelerate their own progress on AI/AGI, so as to reap the ensuing differential economic benefits.
To make the point more directly, the prospect of any modern government seeking to put a damper on current real-world narrow-AI technology seems remote and absurd. It’s hard to imagine the US government forcing a roll-back from modern search engines like Google and Bing to more simplistic search engines like 1997 AltaVista, on the basis that the former embody natural language processing technology that represents a step along the path to powerful AGI.
Wall Street firms (that currently have powerful economic influence on the US government) will not wish to give up their AI-based trading systems, at least not while their counterparts in other countries are using such systems to compete with them on the international currency futures market. Assuming the government did somehow ban AI-based trading systems, how would this be enforced? Would a programmer at a hedge fund be stopped from inserting some more-effective machine learning code in place of the government-sanctioned linear regression code? The US military will not give up their AI-based planning and scheduling systems, as otherwise they would be unable to utilize their military resources effectively. The idea of the government placing an IQ limit on the AI characters in video games, out of fear that these characters might one day become too smart, also seems absurd. Even if the government did so, hackers worldwide would still be drawn to release “mods” for their own smart AIs inserted illicitly into games; and one might see a subculture of pirate games with illegally smart AI.
“Okay, but all these examples are narrow AI, not AGI!” you may argue. “Banning AI that occurs embedded inside practical products is one thing; banning autonomous AGI systems with their own motivations and self-autonomy and the ability to take over the world and kill all humans is quite another!” Note though that the professional AI community does not yet draw a clear border between narrow AI and AGI. While we do believe there is a clear qualitative conceptual distinction, we would find it hard to embody this distinction in a rigorous test for distinguishing narrow AI systems from “proto-AGI systems” representing dramatic partial progress toward human-level AGI. At precisely what level of intelligence would you propose to ban a conversational natural language search interface, an automated call center chatbot, or a house-cleaning robot? How would you distinguish rigorously, across all areas of application, a competent non-threatening narrow-AI system from something with sufficient general intelligence to count as part of the path to dangerous AGI?
A recent workshop of a dozen AGI experts, oriented largely toward originating such tests, failed to come to any definitive conclusions (Adams et al. 2010), recommending instead that a looser mode of evaluation be adopted, involving qualitative synthesis of multiple rigorous evaluations obtained in multiple distinct scenarios. A previous workshop with a similar theme, funded by the US Naval Research Office, came to even less distinct conclusions (Laird et al. 2009). The OpenCog system is explicitly focused on AGI rather than narrow AI, but its various learning modules are also applicable as narrow AI systems, and some of them have largely been developed in this context. In short, there’s no rule for distinguishing narrow AI work from proto-AGI work that is sufficiently clear to be enshrined in government policy, and the banning of narrow AI work seems infeasible as the latter is economically and humanistically valuable, tightly interwoven with nearly all aspects of the economy, and nearly always non-threatening in nature. Even in the military context, the biggest use of AI is in relatively harmless-sounding contexts such as back-end logistics systems, not in frightening applications like killer robots.
Surveying history, one struggles to find good examples of advanced, developed economies slowing down development of any technology with a nebulous definition, obvious wide-ranging short to medium term economic benefits, and rich penetration into multiple industry sectors, due to reasons of speculative perceived long-term risks. Nuclear power research is an example where government policy has slowed things down, but here the perceived economic benefit is relatively modest, the technology is restricted to one sector, the definition of what’s being banned is very clear, and the risks are immediate rather than speculative. More worryingly, nuclear weapons research and development continued unabated for years, despite the clear threat it posed.
In summary, we submit that, due to various aspects of the particular nature of AGI and its relation to other technologies and social institutions, it is very unlikely to be explicitly banned, either before or after an AGI Sputnik moment.
My comment-box got glitchy but just to add: this category of intervention might be a good thing to do for people who care about AI safety and don’t have ML/programming skills, but do have people skills/comms skills/political skills/etc.
Maybe lots of people are indeed working on this sort of thing, I’ve just heard much less discussion of this kind of solution relative to technical solutions.
Asking people not to build AI is like asking them to give up a money machine, almost
We need everyone to agree to stop
There is no clear line. With an atom bomb, it is pretty well defined if you sent it or not. It’s much more vague with “did you do AI research?”
It’s pretty easy to notice if someone sent an atom bomb. Not so easy to notice if they researched AI
AI research is getting cheaper. Today only a few actors can do it, but notice, there are already open source versions of gpt-like models. How long could we hold it back?
Still, people are trying to do things in this direction, and I’m pretty sure that the situation is “try any direction that seems at all plausible”
Yudkowksy writes in his AGI Ruin post:
“We can’t just “decide not to build AGI” because GPUs are everywhere...”
Is anyone thinking seriously about how we might bring it about such that we coordinate globally to not build AGI (at least until we’re confident we can do so safely)? If so, who? If not, why not? It seems like something we should at least try to do, especially if the situation is as dire as Yudkowsky thinks. The sort of thing I’m thinking of is (and this touches on points others have made in their questions):
international governance/regulation
start a protest movement against building AI
do lots of research and thinking about rhetoric and communication and diplomacy, find some extremely charming and charismatic people to work on this, and send them to persuade all actors capable of building AGI to not do it (and to do everything they can to prevent others from doing it)
as someone suggested in another question, translate good materials on why people are concerned about AI safety into Mandarin and other languages
more popularising of AI concerns in English
To be clear, I’m not claiming that this will be easy—this is not a “why don’t we just-” point. I agree with the things Yudkowsky says in that paragraph about why it would be difficult. I’m just saying that it’s not obvious to me that this is fundamentally intractable or harder than solving the technical alignment problem. Reasons for relative optimism:
we seem to achieved some international cooperation around nuclear weapons—isn’t it theoretically possible to do so around AGI?
there are lots of actors who could build AGIs, but it’s still a limited number. Larger groups of actors do cooperate.
through negotiation and diplomacy, people successfully persuade other people to do stuff that’s not even in their interest. AI safety should be a much easier sell because if developing AGI is really dangerous, it’s in everyone’s interest to stop developing it. There are coordination problems to be sure, but the fact remains that the AI safety ‘message’ is fundamentally ‘if you stop doing this we won’t all die’
Nuclear weapons seem like a relatively easy case, in that they require a massive investment to build, are basically of interest only to nation-states, and ultimately don’t provide any direct economic benefit. Regulating AI development looks more similar to something like restricting climate emissions: many different actors could create it, all nations could benefit (economically and otherwise) from continuing to develop it, and the risks of it seem speculative and unproven to many people.
And while there have been significant efforts to restrict climate emissions, there’s still significant resistance to that as well—with it having taken decades for us to get to the current restriction treaties, which many people still consider insufficient.
Goertzel & Pitt (2012) talk about the difficulties of regulating AI:
Thanks! This is interesting.
My comment-box got glitchy but just to add: this category of intervention might be a good thing to do for people who care about AI safety and don’t have ML/programming skills, but do have people skills/comms skills/political skills/etc.
Maybe lots of people are indeed working on this sort of thing, I’ve just heard much less discussion of this kind of solution relative to technical solutions.
Meta: There’s an AI Governance tag and a Regulation and AI Risk tag
My own (very limited) understanding is:
Asking people not to build AI is like asking them to give up a money machine, almost
We need everyone to agree to stop
There is no clear line. With an atom bomb, it is pretty well defined if you sent it or not. It’s much more vague with “did you do AI research?”
It’s pretty easy to notice if someone sent an atom bomb. Not so easy to notice if they researched AI
AI research is getting cheaper. Today only a few actors can do it, but notice, there are already open source versions of gpt-like models. How long could we hold it back?
Still, people are trying to do things in this direction, and I’m pretty sure that the situation is “try any direction that seems at all plausible”
Thanks, this is helpful!