At least as far as I can tell from the public discourse, there seems to be a growing consensus that humans should always and forever be in the loop of AGIs.
Maybe; but there also seems to be a general consensus that humans should be kept in the loop when doing any important decisions in general; yet there are also powerful incentives pushing various actors to automate their modern-day autonomous systems. In particular, there are cases where not having a human in the loop is an advantage by itself, because it e.g. buys you a faster reaction time (see high-frequency trading).
The historical trend has been to automate everything that can be automated, both to reduce costs and because machines can do things better than humans can. Any kind of a business could potentially run better if it were run by a mind that had been custom-built for running the business—up to and including the replacement of all the workers with one or more with such minds. An AI can think faster and smarter, deal with more information at once, and work for a unified purpose rather than have its efficiency weakened by the kinds of office politics that plague any large organization. Some estimates already suggest that half of the tasks that people are paid to do are susceptible to being automated using techniques from modern-day machine learning and robotics, even without postulating AIs with general intelligence (Frey & Osborne 2013, Manyika et al. 2017).
The trend toward automation has been going on throughout history, doesn’t show any signs of stopping, and inherently involves giving the AI systems whatever agency they need in order to run the company better. There is a risk that AI systems that were initially simple and of limited intelligence would gradually gain increasing power and responsibilities as they learned and were upgraded, until large parts of society were under AI control. [...]
[Deploying autonomous AI systems] can happen in two forms: either by expanding the amount of control that already-existing systems have, or alternatively by upgrading existing systems or adding new ones with previously-unseen capabilities. These two forms can blend into each other. If humans previously carried out some functions which are then given over to an upgraded AI which has become recently capable of doing them, this can increase the AI’s autonomy both by making it more powerful and by reducing the amount of humans that were previously in the loop.
As a partial example, the U.S. military is seeking to eventually transition to a state where the human operators of robot weapons are “on the loop” rather than “in the loop” (Wallach & Allen 2013). In other words, whereas a human was previously required to explicitly give the order before a robot was allowed to initiate possibly lethal activity, in the future humans are meant to merely supervise the robot’s actions and interfere if something goes wrong. While this would allow the system to react faster, it would also limit the window that the human operators have for overriding any mistakes that the system makes. For a number of military systems, such as automatic weapons defense systems designed to shoot down incoming missiles and rockets, the extent of human oversight is already limited to accepting or overriding a computer’s plan of actions in a matter of seconds, which may be too little to make a meaningful decision in practice (Human Rights Watch 2012).
Sparrow (2016) reviews three major reasons which incentivize major governments to move toward autonomous weapon systems and reduce human control:
1. Currently existing remotely piloted military “drones,” such as the U.S. Predator and Reaper, require a high amount of communications bandwidth. This limits the amount of drones that can be fielded at once, and makes them dependent on communications satellites which not every nation has, and which can be jammed or targeted by enemies. A need to be in constant communication with remote operators also makes it impossible to create drone submarines, which need to maintain a communications blackout before and during combat. Making the drones autonomous and capable of acting without human supervision would avoid all of these problems.
2. Particularly in air-to-air combat, victory may depend on making very quick decisions. Current air combat is already pushing against the limits of what the human nervous system can handle: further progress may be dependent on removing humans from the loop entirely.
3. Much of the routine operation of drones is very monotonous and boring, which is a major contributor to accidents. The training expenses, salaries, and other benefits of the drone operators are also major expenses for the militaries employing them.
Sparrow’s arguments are specific to the military domain, but they demonstrate the argument that “any broad domain involving high stakes, adversarial decision making, and a need to act rapidly is likely to become increasingly dominated by autonomous systems” (Sotala & Yampolskiy 2015, p. 18). Similar arguments can be made in the business domain: eliminating human employees to reduce costs from mistakes and salaries is something that companies would also be incentivized to do, and making a profit in the field of high-frequency trading already depends on outperforming other traders by fractions of a second. While the currently existing AI systems are not powerful enough to cause global catastrophe, incentives such as these might drive an upgrading of their capabilities that eventually brought them to that point.
In the absence of sufficient regulation, there could be a “race to the bottom of human control” where state or business actors competed to reduce human control and increased the autonomy of their AI systems to obtain an edge over their competitors (see also Armstrong et al. 2016 for a simplified “race to the precipice” scenario). This would be analogous to the “race to the bottom” in current politics, where government actors compete to deregulate or to lower taxes in order to retain or attract businesses.
Suppose that you have a powerful government or corporate actor which has been spending a long time upgrading its AI systems to be increasingly powerful, and achieved better and better gains that way. Now someone shows up and says that they shouldn’t make [some set of additional upgrades], because that would push it to the level of a general intelligence, and having autonomous AGIs is bad. I would expect them to do everything in power to argue that no, actually this is still narrow AI, doing these upgrades and keeping the system in control of their operations are fine—especially if they know that failing to do so is likely to confer an advantage to one of their competitors.
The problem is related to one discussed by Goertzel & Pitt (2012): it seems unlikely that governments would ban narrow AI or restrict its development, but there’s no clear dividing line between narrow AI and AGI, meaning that if you don’t restrict narrow AI then you can’t restrict AGI either.
To make the point more directly, the prospect of any modern government seeking to put a damper on current real-world narrow-AI technology seems remote and absurd. It’s hard to imagine the US government forcing a roll-back from modern search engines like Google and Bing to more simplistic search engines like 1997 AltaVista, on the basis that the former embody natural language processing technology that represents a step along the path to powerful AGI.
Wall Street firms (that currently have powerful economic influence on the US government) will not wish to give up their AI-based trading systems, at least not while their counterparts in other countries are using such systems to compete with them on the international currency futures market. Assuming the government did somehow ban AI-based trading systems, how would this be enforced? Would a programmer at a hedge fund be stopped from inserting some more-effective machine learning code in place of the government-sanctioned linear regression code? The US military will not give up their AI-based planning and scheduling systems, as otherwise they would be unable to utilize their military resources effectively. The idea of the government placing an IQ limit on the AI characters in video games, out of fear that these characters might one day become too smart, also seems absurd. Even if the government did so, hackers worldwide would still be drawn to release “mods” for their own smart AIs inserted illicitly into games; and one might see a subculture of pirate games with illegally smart AI.
“Okay, but all these examples are narrow AI, not AGI!” you may argue. “Banning AI that occurs embedded inside practical products is one thing; banning autonomous AGI systems with their own motivations and self-autonomy and the ability to take over the world and kill all humans is quite another!” Note though that the professional AI community does not yet draw a clear border between narrow AI and AGI. While we do believe there is a clear qualitative conceptual distinction, we would find it hard to embody this distinction in a rigorous test for distinguishing narrow AI systems from “proto-AGI systems” representing dramatic partial progress toward human-level AGI. At precisely what level of intelligence would you propose to ban a conversational natural language search interface, an automated call center chatbot, or a house-cleaning robot? How would you distinguish rigorously, across all areas of application, a competent non-threatening narrow-AI system from something with sufficient general intelligence to count as part of the path to dangerous AGI?
A recent workshop of a dozen AGI experts, oriented largely toward originating such tests, failed to come to any definitive conclusions (Adams et al. 2010), recommending instead that a looser mode of evaluation be adopted, involving qualitative synthesis of multiple rigorous evaluations obtained in multiple distinct scenarios. A previous workshop with a similar theme, funded by the US Naval Research Office, came to even less distinct conclusions (Laird et al. 2009). The OpenCog system is explicitly focused on AGI rather than narrow AI, but its various learning modules are also applicable as narrow AI systems, and some of them have largely been developed in this context. In short, there’s no rule for distinguishing narrow AI work from proto-AGI work that is sufficiently clear to be enshrined in government policy, and the banning of narrow AI work seems infeasible as the latter is economically and humanistically valuable, tightly interwoven with nearly all aspects of the economy, and nearly always non-threatening in nature. Even in the military context, the biggest use of AI is in relatively harmless-sounding contexts such as back-end logistics systems, not in frightening applications like killer robots.
Surveying history, one struggles to find good examples of advanced, developed economies slowing down development of any technology with a nebulous definition, obvious wide-ranging short to medium term economic benefits, and rich penetration into multiple industry sectors, due to reasons of speculative perceived long-term risks. Nuclear power research is an example where government policy has slowed things down, but here the perceived economic benefit is relatively modest, the technology is restricted to one sector, the definition of what’s being banned is very clear, and the risks are immediate rather than speculative. More worryingly, nuclear weapons research and development continued unabated for years, despite the clear threat it posed.
Thank you, those are very interesting references, and very important points! I was arguing that solving a certain coordination problem is even harder than solving a different coordination problem, but I’ll agree that this argument is moot if (as you seem to be arguing) it’s utterly impossible to solve either!
Since you’ve clearly thought a lot about this, have you written up anything about very-long-term scenarios where you see things going well? Are you in the camp of “we should make a benevolent dictator AI implementing CEV”, or “we can make task-limited-AGI-agents and coordinate to never make long-term-planning-AGI-agents”, or something else?
Are you in the camp of “we should make a benevolent dictator AI implementing CEV”, or “we can make task-limited-AGI-agents and coordinate to never make long-term-planning-AGI-agents”, or something else?
No idea. :-)
My general feeling is that having an opinion on the best course of approach would require knowing what AGI and the state of the world will be like when it is developed, but we currently don’t know either.
Lots of historical predictions about coming problems have been rendered completely irrelevant because something totally unexpected happened. And the other way around; it would have been hard for people to predict the issue of computer viruses before electricity had been invented, and harder yet to think about how to prepare for it. That might be a bit of exaggeration—our state of understanding about AGI is probably better than the understanding that pre-electric people would have had of computer viruses—but it still feels impossible to effectively reason about at the moment.
My preferred approach is to just have people pursue many different kinds of basic research on AI safety, understanding human values, etc., while also engaging with near-term AI issues so that they get influence in the kinds of organizations which will eventually make decisions about AI. And then hope that we figure out something once the picture becomes clearer.
It does seem that regulation of AI, should it become necessary, basically has to take the form of regulating access to computer chips. Supercomputers (and server farms) are relatively expensive. You can’t make your own in your basement. Production is centralized at a few locations and so it would not be terribly difficult to track who they’re sold to. They also use lots of electricity, making it easier to track down people who have acquired lots of them illicitly.
I think it’s likely that the computing power required for dangerous AGI will remain at a level well above what most people or non-AI businesses will need for their normal activities, at least up until transformative AI has become widespread. So putting strict limits on chip access would allow goverments to severely cripple AI research, without rolling back the narrow-AI tech we’ve already developed and without looking over every programmer’s shoulder to make sure they don’t code up a neural net.
(A plan like this could also backfire by creating a large hardware overhang and contributing to a fast takeoff.)
Maybe; but there also seems to be a general consensus that humans should be kept in the loop when doing any important decisions in general; yet there are also powerful incentives pushing various actors to automate their modern-day autonomous systems. In particular, there are cases where not having a human in the loop is an advantage by itself, because it e.g. buys you a faster reaction time (see high-frequency trading).
From “Disjunctive Scenarios of Catastrophic AI Risk”:
Suppose that you have a powerful government or corporate actor which has been spending a long time upgrading its AI systems to be increasingly powerful, and achieved better and better gains that way. Now someone shows up and says that they shouldn’t make [some set of additional upgrades], because that would push it to the level of a general intelligence, and having autonomous AGIs is bad. I would expect them to do everything in power to argue that no, actually this is still narrow AI, doing these upgrades and keeping the system in control of their operations are fine—especially if they know that failing to do so is likely to confer an advantage to one of their competitors.
The problem is related to one discussed by Goertzel & Pitt (2012): it seems unlikely that governments would ban narrow AI or restrict its development, but there’s no clear dividing line between narrow AI and AGI, meaning that if you don’t restrict narrow AI then you can’t restrict AGI either.
Thank you, those are very interesting references, and very important points! I was arguing that solving a certain coordination problem is even harder than solving a different coordination problem, but I’ll agree that this argument is moot if (as you seem to be arguing) it’s utterly impossible to solve either!
Since you’ve clearly thought a lot about this, have you written up anything about very-long-term scenarios where you see things going well? Are you in the camp of “we should make a benevolent dictator AI implementing CEV”, or “we can make task-limited-AGI-agents and coordinate to never make long-term-planning-AGI-agents”, or something else?
No idea. :-)
My general feeling is that having an opinion on the best course of approach would require knowing what AGI and the state of the world will be like when it is developed, but we currently don’t know either.
Lots of historical predictions about coming problems have been rendered completely irrelevant because something totally unexpected happened. And the other way around; it would have been hard for people to predict the issue of computer viruses before electricity had been invented, and harder yet to think about how to prepare for it. That might be a bit of exaggeration—our state of understanding about AGI is probably better than the understanding that pre-electric people would have had of computer viruses—but it still feels impossible to effectively reason about at the moment.
My preferred approach is to just have people pursue many different kinds of basic research on AI safety, understanding human values, etc., while also engaging with near-term AI issues so that they get influence in the kinds of organizations which will eventually make decisions about AI. And then hope that we figure out something once the picture becomes clearer.
It does seem that regulation of AI, should it become necessary, basically has to take the form of regulating access to computer chips. Supercomputers (and server farms) are relatively expensive. You can’t make your own in your basement. Production is centralized at a few locations and so it would not be terribly difficult to track who they’re sold to. They also use lots of electricity, making it easier to track down people who have acquired lots of them illicitly.
I think it’s likely that the computing power required for dangerous AGI will remain at a level well above what most people or non-AI businesses will need for their normal activities, at least up until transformative AI has become widespread. So putting strict limits on chip access would allow goverments to severely cripple AI research, without rolling back the narrow-AI tech we’ve already developed and without looking over every programmer’s shoulder to make sure they don’t code up a neural net.
(A plan like this could also backfire by creating a large hardware overhang and contributing to a fast takeoff.)