Indeed, the truth of the matter is that I would be interested in contributing to SIAI, but at the moment I am still not convinced that it would be a good use of my resources. My other objections still haven’t been satisfied, but here’s another argument. As usual, I don’t personally commit to what I claim, since I don’t have enough knowledge to discuss anything in this area with certainty.
The main thing this community seems to lack when discussing Singularity is a lack of political savvy. The primary forces that shape history are, and quite likely, will always be economic and political motives, rather than technology. Technology and innovation are expensive, and innovators require financial and social motivation to create. This applies superlinearly for projects that are so large as to require collaboration.
General AI is exactly that sort of project. There is no magic mathematical insight that will enable us to write a program in a hundred lines of code that will allow it to improve itself in any reasonable amount of time. I’m sure Eliezer is aware of the literature on optimization processes, but the no free lunch principle and the practical randomness of innovation mean that an AI seeking to self-improve can only do so with an (optimized) random search. Humans essentially do the same thing, except we have knowledge and certain built-in processes to help us constrain the search space (but this also makes us miss certain obvious innovations.) To make GAI a real threat, you have to give it enough knowledge so that it can understand the basics of human behavior, or enough knowledge to learn more on its own from human-created resources. This is highly specific information which would take a fully general learning agent a lot of cycles to infer unless it were fed the information, in a machine-friendly form.
Now we will discuss the political and economic aspects of GAI. Support of general artificial intelligence is a political impossibility, because general AI, by definition, is a threat to the jobs of voters. By the time GAI becomes remotely viable, a candidate supporting a ban of GAI will have nearly universal support. It is impossible even to defend GAI on the grounds that the research it produces could save lives, because no medical researcher will welcome a technology that does their job for them. The same applies to any professional. There is a worry on this site that people underestimate GAI, but far more likely is that GAI or anything remotely like it is vastly overestimated as a threat.
The economic aspects are similar. GAI is vastly more costly to develop (for reasons I’ve outlined), and doesn’t provide many advantages over expert systems. Besides, no company is going to produce a self-improving tool in the first place, because nobody, in theory, would ever have to buy an upgraded version.
These political and economic forces are a powerful retardant against the possibility a General AI catastrophe, and have more heft than any focused organization like SIAI could ever have. Yet much like Nader spoiling Al Gore’s vote, the minor influence of SIAI might actually weaken rather than reinforce these protective forces. By claiming to have the tools in place to implement the strategically named ‘friendly AI’, SIAI might in fact assuage public worries about AI. Even if the organization itself does not take actions to do so, GAI advocates will be able to exaggerate the safety of friendly AI and point out that ‘experts have already developed Friendly AI guidelines’ in press releases. And by developing the framework to teach machines about human behavior, SIAI lowers the cost for any enterprise that for some reason, is interested in developing GAI.
At this point, I conclude my hypothetical argument. But I have realized that it is now my true position that SIAI should make it a clear position that: if tenable, NO general AI is preferable to friendly AI. (Back to no-accountability mode: it may be that general AI will eventually come, but by the point it will have become an eventuality, the human race will be vastly more prepared than it is now to deal with such an agent on an equal footing.)
By the time GAI becomes remotely viable, a candidate supporting a ban of GAI will have nearly universal support.
It is already “remotely viable” in the sense that when I thought hard about assigning probabilities to AGI timelines, I had to put a few percent on it happening in the next decade.
Your ideas about the interaction of contemporary political processes and AGI seem wrong to me. You might want to go back to basics and think about how politics, public opinion and the media operate, for example that they had little opinion on the hugely important probabilistic revolution in AI over the last 15 years, but spilled loads of ink over stem cells.
“You might want to go back to basics and think about how politics, public opinion and the media operate, for example that they had little opinion on the hugely important probabilistic revolution in AI over the last 15 years, but spilled loads of ink over stem cells.”
That’s one possible reason. Another possible reason is that AI is not a threat worth caring about, yet. AI may not induce a gut reaction, but what explains the lack of concern about AI among mainstream scientists?
But stem cell research is much more prominent in that it is producing notable direct applications or very close to it. It also isn’t just a yuck factor (although that’s certainly one part), in many different moral systems, stem cells research produced serious moral qualms. AI may very well trigger some similar issues if it becomes more viable.
Probabilistic AI has more apps than stem cells do right now. For example, google. But the point I am making is that an application of a technology is a logical factor, whereas people actually respond to emotional factors, like whether it breaks taboos that go back to the stone age. For example, anything that involves sex, flesh, blood, overtones of bestiality, overtones of harm to children, trading a sacred good for an unsacred one etc.
The ideal technology for people to want to ban would involve harvesting a foetus that was purchased from a hooker, then hybridizing it with a pig foetus, then injecting the resultant cells into the gonads of little kids. That technology would get nuked by the public.
The ideal dangerous technology for people to not give a shit about banning would involve a theoretical threat which is hard to understand, has never happened before, involves only nonphysical harards like information, and has nothing to do with flesh, sex or anything disgusting or with fire, sharp objects or other natural disasters.
“The ideal dangerous technology for people to not give a shit about banning would involve a theoretical threat which is hard to understand”
I don’t think The Terminator was hard to understand. The second you get some credible people saying that AI is a threat, the media reaction is going to be overexcessive, as it always is.
The second you get some credible people saying that AI is a threat
It’s already happened—didn’t you see the media about Stephen Hawking saying AI could be dangerous? And Bill Joy?
The general point I am trying to make is that the general public are not rational in terms of collective epistemology. They don’t respond to complex logical and quantitative analyses. Yes, Joy and Hawking did say that AI is a risk, but there are many risks, including the risk that vaccinations cause autism and the risk that foreign workers will take all our jobs. The public does not understand the difference between these risks.
Thanks; I was mistaken. Would you say, then, that mainstream scientists are similarly irrational? (The main comparison I have in mind throughout this section, by the way, is global warming.)
I would say that poor social epistemology and, poor social axiology and mediocre individual rationality are the big culprits that prevent many scientists from taking AI risk seriously.
By “social axiology” I mean that our society is just not consequentialist enough. We don’t solve problems that way, and even the debate about global warming is not really dealing well with the problem of how to quantify risks under uncertainty. We don’t try to improve the world in a systematic, rational way; rather it is done piecemeal.
There may be an issue here about what we define as AI. For example, I would not see what Google does as AI but rather as harvesting human intelligence. The lines here may be blurry are hard to define.
Could someone explain why this comment got modded down? I don’t see any errors in reasoning or other issues. (Was the content level too low for the desired signal/noise ratio?)
Google uses exactly the techniques from the probabilistic revolution, namely machine learning, which is the relevant fact. Whether you call it AI is not relevant to the point at issue as far as I can see.
Do you have a citation for Google using machine learning in any substantial scale? The most basic of the Google algorithms is PageRank which isn’t a machine learning algorithm by most definitions of that term.
The ideal dangerous technology for people to not give a shit about banning would involve a theoretical threat which is hard to understand, has never happened before, involves only nonphysical harards like information, and has nothing to do with flesh, sex or anything disgusting or with fire, sharp objects or other natural disasters.
Yes, but these are precisely the dangers humans should certainly not worry about to begin with.
The main thing this community seems to lack when discussing Singularity is a lack of political savvy. The primary forces that shape history are, and quite likely, will always be economic and political motives, rather than technology.
I think a simple examination of the history of the last couple centuries really fails to support this.
Support of general artificial intelligence is a political impossibility, because general AI, by definition, is a threat to the jobs of voters.
Expert AI systems are already used in hospitals, and will surely be used more and more as the technology progresses. There isn’t a single point where AI is suddenly better than humans at all aspects of a field. Current AIs are already better than doctors in some areas, but worse in many others. As the range of AI expertise increases doctors will shift more towards managerial roles, understanding the strengths and weakness of the myriad expert systems, refereeing between them and knowing when to overrule them.
By the time true AGI arrives narrow AI will probably be pervasive enough that the line between the two will be too fuzzy to allow for a naive ban on AGI. Moreover, I highly doubt people are going to vote to save jobs (especially jobs of the affluent) at the expense of human life.
EDIT: I’ve realized that some misinterpretation of my arguments has been due to disagreements in terminology. I define “expert systems” as systems designed to address a specific class of well-defined problems, capable of logical reasoning and probabilistic inference given a set of “axiom-like” rules, and updating their knowledge database with specific kinds of information.
AGI I define specifically as AI which has human or extra-human level capabilities, or the potential to reach those capabilities.
Now my response to the above:
“Expert AI systems are already used in hospitals, and will surely be used more and more as the technology progresses. There isn’t a single point where AI is suddenly better than humans at all aspects of a field. Current AIs are already better than doctors in some areas, but worse in many others. As the range of AI expertise increases doctors will shift more towards managerial roles, understanding the strengths and weakness of the myriad expert systems, refereeing between them and knowing when to overrule them.”
I agree with all of these.
“By the time true AGI arrives narrow AI will probably be pervasive enough that the line between the two will be too fuzzy to allow for a naive ban on AGI.”
To me it seems the greatest enabler of AI catastrophe is ignorance. But by the time narrow AI becomes pervasive, it’s also likely that people will possess much more of the technical understanding needed to comprehend the threat that AGI possesses.
“Moreover, I highly doubt people are going to vote to save jobs (especially jobs of the affluent) at the expense of human life.”
Ban all self-modifying code and you should be in the clear.
So instead of modifying its own source code, the AI programs a new, more powerful AI from scratch, that has the same values as the old AI, and has no prohibition against modifying its source code.
Yes, you can forbid that too, but you didn’t think to, and you only get one shot. And then it can decide to arrange a bunch of transistors into a pattern that it predicts will produce a state of the universe it prefers.
The problem here is that you are trying to use ad hoc constraints on a creative intelligence that is motivated to get around the constraints.
I know that the FAI argument is that the only way to prevent disaster is to make the agent “want” to not modify itself. But I’m arguing that for an agent to even be dangerous, it has to “want” to modify itself. There is no plausible scenario where an agent solving a specific problem decides that the most efficient path to the solution involves upgrading its own capabilities. It’s certainly not going to stumble upon a self-improvement randomly.
You don’t think that a sufficiently powerful seed AI would, if self-modification were clearly the most efficient way to reach its goal, discover the idea of self-modification? Humans have independently discovered self-improvement many times.
EDIT: Sorry, I’m specifically not talking about seed AI’s. I’m talking about the (non-) possibility of commercial programs designed for specific applications “going rogue”
To adopt self-modification as a strategy, it would have to have knowledge of itself. And then, it order to pursue the strategy, it would have to decide that the costs of discovering self-improvements were an efficient use of its resources, if it could even estimate the amount of time it took to discover an actual improvement on its system.
Intelligence can’t just instantly come up with the right answer by applying heuristics. Intelligence has to go through a heuristic (narrowing the search space)/random search/TEST (or PROVE) cycle.
Self-improvement is very costly in terms of these cycles. To even confirm that a modification is a self-improvement, a system has to simulate its modified performance on a variety of test problems. If a system is designed to solve problems that take X amount of time, it would take at least X that amount of time to get an empirical sample to answer whether or not a proposed modification would be worth it (and likely more time for proof). And with no prior knowledge, most proposed modifications would not be improvements.
AI ethics is not necessary to constrain such systems. Just a non-lenient pruning process, (which would be required anyways for efficiency on ordinary problems.)
You are talking about an AI that was designed to self-examine and optimize itself. Otherwise it will never ever be a full AGI. We are not smart enough to build one from scratch. The trick, if possible, is to get it to not modify the fundamental Friendliness goal during its self-modifications.
There are algoritms in narrow AI that do learning and modify algorithm specifics or chose among algorithms or combinations of algorithms. There are algorithms that search for better algorithms. In some languages (LISP family) there is little/no difference in code and data so code modifying code is a common working methodology for human Lisp programmers. A cross from code/data space to hardware space is sufficient to have such an AI redesign the hardware it runs on as well. Such goals can be either hardwired or arise under the general goal of improvement plus an adequate knowledge of hardware or the ability to acquire it.
We ourselves are general purpose machines that happen to be biological and seek to some degree to understand ourselves enough to self-modify to become better.
I am talking about AIs designed for solving specific bounded problems. In this case the goal of the AI—which is to solve the problem efficiently—is as much of a constraint as its technical capabilities. Even if the AI has fundamental-self-modification routines at its disposal, I can hardly envisage a scenario in which the AI decides that the use of these routines would constitute an efficient use of its time for solving its specific problem.
“So instead of modifying its own source code, the AI programs a new, more powerful AI from scratch, that has the same values as the old AI, and has no prohibition against modifying its source code.”
But by the time narrow AI becomes pervasive, it’s also likely that people will possess much more of the technical understanding needed to comprehend the threat that AGI possesses.
Or perhaps it’s the contrary: pervasive narrow AI fosters an undue sense of security. People become comfortable via familiarity, whether it’s justified or not. This morning I was peering down a 50 foot cliff, half way up, suspended by nothing but a half inch wide rope. No fear, no hesitation, perfect familiarity. Luckily, due to knowledge of numerous deaths of past climbers I can maintain a conscious alertness to safety and stave off complacency. But in the case of AI, what overt catastrophes will similarly stave off complacency toward existential risk short of an existential catastrophe itself?
Our current conception of AGI is based on a biased comparison of hypothetical AGI capabilities with our relatively unehanced capabilities. By the time AGI is viable, a typical professional with expert systems will be able to vastly outperform current professionals with our current tools.
What about the speed bottleneck from human decision making, compounded by human working memory bottleneck, if lots of relevant data is involved? Algorithmic trading already has automated systems doing stock trades since they can make decisions so much faster than a human expert.
I imagine being very fast would be a great help in quite a few creative tasks. Off the top of my head, being able to develop new features in software in seconds instead of days would be a significant competitive advantage.
You make some good points about economic and political realities. However, I’m deeply puzzled by some of your other remarks. For example, you make the claim that general AI wouldn’t provide any benefits above expert systems. I’m deeply puzzled by this claim since expert systems are by nature highly limited. Expert systems cannot construct new ideas nor can they handle anything that’s even vaguely cross-disciplinary. No number of expert systems will be able to engage in the same degree of scientific productivity as a single bright scientists.
You also claim that no general AI is better than friendly AI. This is deeply puzzling. This makes sense only if one is fantastically paranoid about the loss of jobs. But new technologies are often economically disruptive. There are all sorts of jobs that don’t exist now that were around a hundred years ago, or even fifty years ago. And yes, people lost jobs. But overall, they are better for it. You would need to make a much stronger case if you are trying to establish that no general AI is somehow better than general AI.
Why do you think expert systems cannot handle anything cross-disciplinary? I even say that expert systems can generate new ideas, by more or less the same process that humans do. An expert system only needs an understanding of manufacturing, physics, and chemistry to design better computer chips, for instance. If you’re talking about revolutionary, paradigm shifting ideas—we are probably already saturated with such ideas. The main bottleneck inhibiting paradigm shifts is not the ideas but the infrastructure and economic need for the paradigm shift. A company that can produce a 10% better product can already take over the market, a 200% better product is overkill, and especially unnecessary if there are substantial costs in overhauling the production line.
The reason why NO general AI is better than friendly (general) AI is very simple. IF general AI is an existential threat, than no organization claiming to put humans first could justify being pro-AGI (friendly or not), since no possible benefit* can justify the risk of destroying humanity.
*save for mitigating an even larger risk of annihilation, of course
Why do you think expert systems cannot handle anything cross-disciplinary? I even
say that expert systems can generate new ideas, by more or less the same process > that humans do. An expert system only needs an understanding of manufacturing,
physics, and chemistry to design better computer chips, for instance.
Expert systems generally need very narrow problem domains to function. I’m not sure how you would expect an expert system to have an understanding of three very broad topics. Moreover, I don’t know exactly how humans come up with new ideas (sometimes when people ask me, I tell them that I bang my head against the wall. That’s not quite true but it does reflect that I only understand at a very gross level how I construct new ideas. I’m bright but not very bright, and I can see that much smarter people have the same trouble). So how you are convinced that expert systems could construct new ideas is not at all clear to me.
To be sure, there have been some limited work with computer systems coming up with new, interesting ideas. There’s been some limited success with computers in my own field. See for example Simon Colton’s work. There’s also been similar work in geometry and group theory. But none of these systems were expert systems as that term is normally used. Moreover, none of the ideas they’ve come up with have that impressive. The only exception I’m aware of that is the proof of the Robbins conjecture. So even in narrow areas we’ve had very little success using specialized AIs. Are you using a more general definition of expert system than is standard?
The reason why NO general AI is better than friendly (general) AI is very simple. IF
general AI is an existential threat, than no organization claiming to put humans
first could justify being pro-AGI (friendly or not), since no possible benefit* can
justify the risk of destroying humanity
Multiple problems with that claim. First, the existential threat may be low. There’s some tiny risk for example that the LHC will destroy the Earth in some very fun way. There’s also some risk that work with genetic engineering might give fanatics the skill to make a humanity destroying pathogen. And there’s a chance that nanotech might turn everything into purple with green stripes goo (this is much more likely than gray goo of course). There’s even some risk that proving the wrong theorem might summon Lovecraftian horrors. All events have some degree of risk. Moreover, general AI might actually help mitigate some serious threats, such as making it easier to track and deal with rogue asteroids or other catastrophic threats.
Also, even if one accepted the general outline of your argument, one would conclude that that’s a reason why organizations shouldn’t try to make general friendly AI. It isn’t a reason that actually having no AI is better than having no friendly AI.
“First, the existential threat [of AGI] may be low.”
Let me trace back the argument tree for a second. I originally asked for a defense of the claim that “SIAI is tackling the world’s most important task.” Michael Porter responded, “The real question is, do you even believe that unfriendly AI is a threat to the human race, and if so, is there anyone else tackling the problem in even a semi-competent way?” So NOW in this argument tree, we’re assuming that unfriendly AI IS an existential threat, enough that preventing it is the “world’s most important task.”
Now in this branch of the argument, I assumed (but did not state) the following: If unfriendly AI is an existential threat, friendly AI is an existential threat, as long as there is some chance of it being modified into unfriendly AI. Furthermore, I assert that it’s a naive notion that any organization could protect friendly AI from being subverted.
AIs, including ones with Friendly goals, are apt to work to protect their goal systems from modification, as this will prevent their efforts from being directed towards things other than their (current) aims. There might be a window while the AI is mid-FOOM where it’s vulnerable, but not a wide one.
Let me posit that FAI may be much less capable than unfriendly AI. The power of unfriendly AI is that it can increase its growth rate by taking resources by force. An FAI would be limited to what resources it could ethically obtain. Therefore, a low-grade FAI might be quite vulnerable to human antagonists, while its unrestricted version could be magnitudes of order more dangerous. In short, FAI could be low-reward high-risk.
There are plenty of resources that an FAI could ethically obtain, and with a lead of time of less than 1 day, it could grow enough to be vastly more powerful than an unfriendly seed AI.
Really, asking which AI wins going head to head is the wrong question. The goal is to get an FAI running before unfriendly AGI is implemented.
The power of unfriendly AI is that it can increase its growth rate by taking resources by force. An FAI would be limited to what resources it could ethically obtain.
Wrong. FAI will make whatever unethical steps it must, as long as it’s on the net the best path it can see, taking into account both the (ethically harmful) instrumental actions and their expected outcome. There is no such general disadvantage coming with AI being Friendly. Not that I expect any need for such drastic measures (in an apparent way), especially considering the likely fist-mover advantage it’ll have.
An expert system only needs an understanding of manufacturing, physics, and chemistry to design better computer chips, for instance.
If a program can take an understanding of those subjects and design a better computer chip, I don’t think it’s just an “expert system” anymore. I would think it would take an AI to do that. That’s an AI complete problem.
If you’re talking about revolutionary, paradigm shifting ideas—we are probably already saturated with such ideas. The main bottleneck inhibiting paradigm shifts is not the ideas but the infrastructure and economic need for the paradigm shift.
Are you serious? I would think the exact opposite would be true: we have an infrastructure starving for paradigm shifting ideas. I’d love to hear some of these revolutionary ideas that we’re saturated with. I think we have some insights, but these insights need to be fleshed out and implemented, and figuring out how to do that is the paradigm shift that needs to occur
no organization claiming to put humans first could justify being pro-AGI (friendly or not), since no possible benefit* can justify the risk of destroying humanity.
Wait a minute. If I could press a button now with a 10% chance of destroying humanity and a 90% chance of solving the world’s problems, I’d do it. Everything we do has some risks. Even the LHC had an (extremely miniscule) risk of destroying the universe, but doing a cost-benefit analysis should reveal that some things are worth minor chances of destroying humanity.
“If a program can take an understanding of those subjects and design a better computer chip, I don’t think it’s just an “expert system” anymore. I would think it would take an AI to do that. That’s an AI complete problem.”
What I had in mind was some sort of combinatorial approach to designing chips, i.e. take these materials and randomly generate a design, test it, and then start altering the search space based on the results. I didn’t mean “understanding” in the human sense of the word, sorry.
“I’d love to hear some of these revolutionary ideas that we’re saturated with. I think we have some insights, but these insights need to be fleshed out and implemented, and figuring out how to do that is the paradigm shift that needs to occur”
Example: many aspects of the legal and political systems could be reformed, and it’s not difficult to come up with ideas on how they could be reformed. The benefit is simply insufficient to justify spending much of the limited resources we have on solving those problems.
“Wait a minute. If I could press a button now with a 10% chance of destroying humanity and a 90% chance of solving the world’s problems, I’d do it. ”
So you think there’s a >10% chance that the world’s problems are going to destroy humanity in the near future?
What I had in mind was some sort of combinatorial approach to designing chips, i.e. > take these materials and randomly generate a design, test it, and then start altering
the search space based on the results. I didn’t mean “understanding” in the human
sense of the word, sorry.
Given the very large number of possibilities and the difficulty with making prototypes, this seems like an extremely inefficient process without more thought going into to it.
What I had in mind was some sort of combinatorial approach to designing chips
Oh, okay, fair enough, though I’m still not sure I would call that an “expert system” (this time for the opposite reason that it seems too stupid).
many aspects of the legal and political systems could be reformed, and it’s not difficult to come up with ideas on how they could be reformed. The benefit is simply insufficient to justify spending much of the limited resources we have on solving those problems.
Ah. I was thinking of designing an AI, probably because I was primed by your expert system comment. Well, in those cases, I think the issue is that our legal and political systems were purposely set up to be difficult to change: change requires overturning precedents, obtaining majority or 3⁄5 or 2⁄3 votes in various legislative bodies, passing constitutional amendments, and so forth. And I can guarantee you that for any of these reforms, there are powerful interests who would be harmed by the reforms, and many people who don’t want reform: this is more of a persuasion problem than an infrastructure problem. But yes, you’re right that there are plenty of revolutionary ideas about how to reform, say, the education system: they’re just not widely accepted enough to happen.
So you think there’s a >10% chance that the world’s problems are going to destroy humanity in the near future?
I’m confused by this sentence. I’m not sure if I think that, but what does it have to do with the hypothetical button that has a 10% chance of destroying humanity? My point was that it’s worth taking a small risk of destroying humanity if the benefits are great enough.
Bear in mind that the people who used steam engines to make money didn’t make it by selling the engines: rather, the engines were useful in producing other goods. I don’t think that the creators of a cheap substitute for human labor (GAI could be one such example) would be looking to sell it necessarily. They could simply want to develop such a tool in order to produce a wide array of goods at low cost.
I may think that I’m clever enough, for example, to keep it in a box and ask it for stock market predictions now and again. :)
As for the “no free lunch” business, while its true that any real-world GAI could not efficiently solve every induction problem, it wouldn’t need to either for it to be quite fearsome. Indeed being able to efficiently solve at least the same set of induction problems that humans solve (particularly if its in silicon and the hardware is relatively cheap) is sufficient to pose a big threat (and be potentially quite useful economically).
Also, there is a non-zero possibility that there already exists a GAI and its creators, decided the safest, most lucrative, and beneficial thing to do is set the GAI on designing drugs: thereby avoiding giving the GAI too much information about the world. The creators could have then set up a biotech company that just so happens to produce a few good drugs now and again. Its kind of like how automated trading came from computer scientists and not the currently employed traders. I do think its unlikely that somebody working in medical research is going to develop GAI least of all because of the job threat. The creators of a GAI are probably going to be full time professionals who are are working on the project.
I’m surprised that nobody so far has pointed out a rather obvious counter to my argument that “AGI will be politically unjustifiable.” I don’t post flawed arguments on purpose, but I usually realize counteraguments shortly after I post them. In any case, even if the popular sentiment in democracies is to block AGI, this doesn’t mean that other governments couldn’t support AGI. I wonder what the SIAI plans to do for the possibility of a hostile government funding unfriendly AI for military purposes.
Indeed, the truth of the matter is that I would be interested in contributing to SIAI, but at the moment I am still not convinced that it would be a good use of my resources. My other objections still haven’t been satisfied, but here’s another argument. As usual, I don’t personally commit to what I claim, since I don’t have enough knowledge to discuss anything in this area with certainty.
The main thing this community seems to lack when discussing Singularity is a lack of political savvy. The primary forces that shape history are, and quite likely, will always be economic and political motives, rather than technology. Technology and innovation are expensive, and innovators require financial and social motivation to create. This applies superlinearly for projects that are so large as to require collaboration.
General AI is exactly that sort of project. There is no magic mathematical insight that will enable us to write a program in a hundred lines of code that will allow it to improve itself in any reasonable amount of time. I’m sure Eliezer is aware of the literature on optimization processes, but the no free lunch principle and the practical randomness of innovation mean that an AI seeking to self-improve can only do so with an (optimized) random search. Humans essentially do the same thing, except we have knowledge and certain built-in processes to help us constrain the search space (but this also makes us miss certain obvious innovations.) To make GAI a real threat, you have to give it enough knowledge so that it can understand the basics of human behavior, or enough knowledge to learn more on its own from human-created resources. This is highly specific information which would take a fully general learning agent a lot of cycles to infer unless it were fed the information, in a machine-friendly form.
Now we will discuss the political and economic aspects of GAI. Support of general artificial intelligence is a political impossibility, because general AI, by definition, is a threat to the jobs of voters. By the time GAI becomes remotely viable, a candidate supporting a ban of GAI will have nearly universal support. It is impossible even to defend GAI on the grounds that the research it produces could save lives, because no medical researcher will welcome a technology that does their job for them. The same applies to any professional. There is a worry on this site that people underestimate GAI, but far more likely is that GAI or anything remotely like it is vastly overestimated as a threat.
The economic aspects are similar. GAI is vastly more costly to develop (for reasons I’ve outlined), and doesn’t provide many advantages over expert systems. Besides, no company is going to produce a self-improving tool in the first place, because nobody, in theory, would ever have to buy an upgraded version.
These political and economic forces are a powerful retardant against the possibility a General AI catastrophe, and have more heft than any focused organization like SIAI could ever have. Yet much like Nader spoiling Al Gore’s vote, the minor influence of SIAI might actually weaken rather than reinforce these protective forces. By claiming to have the tools in place to implement the strategically named ‘friendly AI’, SIAI might in fact assuage public worries about AI. Even if the organization itself does not take actions to do so, GAI advocates will be able to exaggerate the safety of friendly AI and point out that ‘experts have already developed Friendly AI guidelines’ in press releases. And by developing the framework to teach machines about human behavior, SIAI lowers the cost for any enterprise that for some reason, is interested in developing GAI.
At this point, I conclude my hypothetical argument. But I have realized that it is now my true position that SIAI should make it a clear position that: if tenable, NO general AI is preferable to friendly AI. (Back to no-accountability mode: it may be that general AI will eventually come, but by the point it will have become an eventuality, the human race will be vastly more prepared than it is now to deal with such an agent on an equal footing.)
It is already “remotely viable” in the sense that when I thought hard about assigning probabilities to AGI timelines, I had to put a few percent on it happening in the next decade.
Your ideas about the interaction of contemporary political processes and AGI seem wrong to me. You might want to go back to basics and think about how politics, public opinion and the media operate, for example that they had little opinion on the hugely important probabilistic revolution in AI over the last 15 years, but spilled loads of ink over stem cells.
“You might want to go back to basics and think about how politics, public opinion and the media operate, for example that they had little opinion on the hugely important probabilistic revolution in AI over the last 15 years, but spilled loads of ink over stem cells.”
And why is that?
Yuck factor for stem cells but not for probabilistic AI.
That’s one possible reason. Another possible reason is that AI is not a threat worth caring about, yet. AI may not induce a gut reaction, but what explains the lack of concern about AI among mainstream scientists?
But stem cell research is much more prominent in that it is producing notable direct applications or very close to it. It also isn’t just a yuck factor (although that’s certainly one part), in many different moral systems, stem cells research produced serious moral qualms. AI may very well trigger some similar issues if it becomes more viable.
Probabilistic AI has more apps than stem cells do right now. For example, google. But the point I am making is that an application of a technology is a logical factor, whereas people actually respond to emotional factors, like whether it breaks taboos that go back to the stone age. For example, anything that involves sex, flesh, blood, overtones of bestiality, overtones of harm to children, trading a sacred good for an unsacred one etc.
The ideal technology for people to want to ban would involve harvesting a foetus that was purchased from a hooker, then hybridizing it with a pig foetus, then injecting the resultant cells into the gonads of little kids. That technology would get nuked by the public.
The ideal dangerous technology for people to not give a shit about banning would involve a theoretical threat which is hard to understand, has never happened before, involves only nonphysical harards like information, and has nothing to do with flesh, sex or anything disgusting or with fire, sharp objects or other natural disasters.
“The ideal dangerous technology for people to not give a shit about banning would involve a theoretical threat which is hard to understand”
I don’t think The Terminator was hard to understand. The second you get some credible people saying that AI is a threat, the media reaction is going to be overexcessive, as it always is.
It’s already happened—didn’t you see the media about Stephen Hawking saying AI could be dangerous? And Bill Joy?
The general point I am trying to make is that the general public are not rational in terms of collective epistemology. They don’t respond to complex logical and quantitative analyses. Yes, Joy and Hawking did say that AI is a risk, but there are many risks, including the risk that vaccinations cause autism and the risk that foreign workers will take all our jobs. The public does not understand the difference between these risks.
Thanks; I was mistaken. Would you say, then, that mainstream scientists are similarly irrational? (The main comparison I have in mind throughout this section, by the way, is global warming.)
I would say that poor social epistemology and, poor social axiology and mediocre individual rationality are the big culprits that prevent many scientists from taking AI risk seriously.
By “social axiology” I mean that our society is just not consequentialist enough. We don’t solve problems that way, and even the debate about global warming is not really dealing well with the problem of how to quantify risks under uncertainty. We don’t try to improve the world in a systematic, rational way; rather it is done piecemeal.
There may be an issue here about what we define as AI. For example, I would not see what Google does as AI but rather as harvesting human intelligence. The lines here may be blurry are hard to define.
You make a good point about older taboos.
Could someone explain why this comment got modded down? I don’t see any errors in reasoning or other issues. (Was the content level too low for the desired signal/noise ratio?)
Google uses exactly the techniques from the probabilistic revolution, namely machine learning, which is the relevant fact. Whether you call it AI is not relevant to the point at issue as far as I can see.
Do you have a citation for Google using machine learning in any substantial scale? The most basic of the Google algorithms is PageRank which isn’t a machine learning algorithm by most definitions of that term.
Adwords uses more core ML techniques
Yes, but these are precisely the dangers humans should certainly not worry about to begin with.
I think a simple examination of the history of the last couple centuries really fails to support this.
Expert AI systems are already used in hospitals, and will surely be used more and more as the technology progresses. There isn’t a single point where AI is suddenly better than humans at all aspects of a field. Current AIs are already better than doctors in some areas, but worse in many others. As the range of AI expertise increases doctors will shift more towards managerial roles, understanding the strengths and weakness of the myriad expert systems, refereeing between them and knowing when to overrule them.
By the time true AGI arrives narrow AI will probably be pervasive enough that the line between the two will be too fuzzy to allow for a naive ban on AGI. Moreover, I highly doubt people are going to vote to save jobs (especially jobs of the affluent) at the expense of human life.
EDIT: I’ve realized that some misinterpretation of my arguments has been due to disagreements in terminology. I define “expert systems” as systems designed to address a specific class of well-defined problems, capable of logical reasoning and probabilistic inference given a set of “axiom-like” rules, and updating their knowledge database with specific kinds of information.
AGI I define specifically as AI which has human or extra-human level capabilities, or the potential to reach those capabilities.
Now my response to the above:
“Expert AI systems are already used in hospitals, and will surely be used more and more as the technology progresses. There isn’t a single point where AI is suddenly better than humans at all aspects of a field. Current AIs are already better than doctors in some areas, but worse in many others. As the range of AI expertise increases doctors will shift more towards managerial roles, understanding the strengths and weakness of the myriad expert systems, refereeing between them and knowing when to overrule them.”
I agree with all of these.
“By the time true AGI arrives narrow AI will probably be pervasive enough that the line between the two will be too fuzzy to allow for a naive ban on AGI.”
To me it seems the greatest enabler of AI catastrophe is ignorance. But by the time narrow AI becomes pervasive, it’s also likely that people will possess much more of the technical understanding needed to comprehend the threat that AGI possesses.
“Moreover, I highly doubt people are going to vote to save jobs (especially jobs of the affluent) at the expense of human life.”
You are being too idealistic here.
So instead of modifying its own source code, the AI programs a new, more powerful AI from scratch, that has the same values as the old AI, and has no prohibition against modifying its source code.
Yes, you can forbid that too, but you didn’t think to, and you only get one shot. And then it can decide to arrange a bunch of transistors into a pattern that it predicts will produce a state of the universe it prefers.
The problem here is that you are trying to use ad hoc constraints on a creative intelligence that is motivated to get around the constraints.
I know that the FAI argument is that the only way to prevent disaster is to make the agent “want” to not modify itself. But I’m arguing that for an agent to even be dangerous, it has to “want” to modify itself. There is no plausible scenario where an agent solving a specific problem decides that the most efficient path to the solution involves upgrading its own capabilities. It’s certainly not going to stumble upon a self-improvement randomly.
You don’t think that a sufficiently powerful seed AI would, if self-modification were clearly the most efficient way to reach its goal, discover the idea of self-modification? Humans have independently discovered self-improvement many times.
EDIT: Sorry, I’m specifically not talking about seed AI’s. I’m talking about the (non-) possibility of commercial programs designed for specific applications “going rogue”
To adopt self-modification as a strategy, it would have to have knowledge of itself. And then, it order to pursue the strategy, it would have to decide that the costs of discovering self-improvements were an efficient use of its resources, if it could even estimate the amount of time it took to discover an actual improvement on its system.
Intelligence can’t just instantly come up with the right answer by applying heuristics. Intelligence has to go through a heuristic (narrowing the search space)/random search/TEST (or PROVE) cycle.
Self-improvement is very costly in terms of these cycles. To even confirm that a modification is a self-improvement, a system has to simulate its modified performance on a variety of test problems. If a system is designed to solve problems that take X amount of time, it would take at least X that amount of time to get an empirical sample to answer whether or not a proposed modification would be worth it (and likely more time for proof). And with no prior knowledge, most proposed modifications would not be improvements.
AI ethics is not necessary to constrain such systems. Just a non-lenient pruning process, (which would be required anyways for efficiency on ordinary problems.)
You are talking about an AI that was designed to self-examine and optimize itself. Otherwise it will never ever be a full AGI. We are not smart enough to build one from scratch. The trick, if possible, is to get it to not modify the fundamental Friendliness goal during its self-modifications.
There are algoritms in narrow AI that do learning and modify algorithm specifics or chose among algorithms or combinations of algorithms. There are algorithms that search for better algorithms. In some languages (LISP family) there is little/no difference in code and data so code modifying code is a common working methodology for human Lisp programmers. A cross from code/data space to hardware space is sufficient to have such an AI redesign the hardware it runs on as well. Such goals can be either hardwired or arise under the general goal of improvement plus an adequate knowledge of hardware or the ability to acquire it.
We ourselves are general purpose machines that happen to be biological and seek to some degree to understand ourselves enough to self-modify to become better.
I am talking about AIs designed for solving specific bounded problems. In this case the goal of the AI—which is to solve the problem efficiently—is as much of a constraint as its technical capabilities. Even if the AI has fundamental-self-modification routines at its disposal, I can hardly envisage a scenario in which the AI decides that the use of these routines would constitute an efficient use of its time for solving its specific problem.
“So instead of modifying its own source code, the AI programs a new, more powerful AI from scratch, that has the same values as the old AI, and has no prohibition against modifying its source code.”
Isn’t that the same as self-modifying code?
Or perhaps it’s the contrary: pervasive narrow AI fosters an undue sense of security. People become comfortable via familiarity, whether it’s justified or not. This morning I was peering down a 50 foot cliff, half way up, suspended by nothing but a half inch wide rope. No fear, no hesitation, perfect familiarity. Luckily, due to knowledge of numerous deaths of past climbers I can maintain a conscious alertness to safety and stave off complacency. But in the case of AI, what overt catastrophes will similarly stave off complacency toward existential risk short of an existential catastrophe itself?
What a strange thing to say.
Our current conception of AGI is based on a biased comparison of hypothetical AGI capabilities with our relatively unehanced capabilities. By the time AGI is viable, a typical professional with expert systems will be able to vastly outperform current professionals with our current tools.
What about the speed bottleneck from human decision making, compounded by human working memory bottleneck, if lots of relevant data is involved? Algorithmic trading already has automated systems doing stock trades since they can make decisions so much faster than a human expert.
Expert systems would be faster still. For AGI to be justified in this case, you would need a task that required both speed and creativity.
I imagine being very fast would be a great help in quite a few creative tasks. Off the top of my head, being able to develop new features in software in seconds instead of days would be a significant competitive advantage.
“AGI capability” is to rewrite the universe.
Yes, but it would have to take the resources from humans first.
You make some good points about economic and political realities. However, I’m deeply puzzled by some of your other remarks. For example, you make the claim that general AI wouldn’t provide any benefits above expert systems. I’m deeply puzzled by this claim since expert systems are by nature highly limited. Expert systems cannot construct new ideas nor can they handle anything that’s even vaguely cross-disciplinary. No number of expert systems will be able to engage in the same degree of scientific productivity as a single bright scientists.
You also claim that no general AI is better than friendly AI. This is deeply puzzling. This makes sense only if one is fantastically paranoid about the loss of jobs. But new technologies are often economically disruptive. There are all sorts of jobs that don’t exist now that were around a hundred years ago, or even fifty years ago. And yes, people lost jobs. But overall, they are better for it. You would need to make a much stronger case if you are trying to establish that no general AI is somehow better than general AI.
Why do you think expert systems cannot handle anything cross-disciplinary? I even say that expert systems can generate new ideas, by more or less the same process that humans do. An expert system only needs an understanding of manufacturing, physics, and chemistry to design better computer chips, for instance. If you’re talking about revolutionary, paradigm shifting ideas—we are probably already saturated with such ideas. The main bottleneck inhibiting paradigm shifts is not the ideas but the infrastructure and economic need for the paradigm shift. A company that can produce a 10% better product can already take over the market, a 200% better product is overkill, and especially unnecessary if there are substantial costs in overhauling the production line.
The reason why NO general AI is better than friendly (general) AI is very simple. IF general AI is an existential threat, than no organization claiming to put humans first could justify being pro-AGI (friendly or not), since no possible benefit* can justify the risk of destroying humanity.
*save for mitigating an even larger risk of annihilation, of course
Expert systems generally need very narrow problem domains to function. I’m not sure how you would expect an expert system to have an understanding of three very broad topics. Moreover, I don’t know exactly how humans come up with new ideas (sometimes when people ask me, I tell them that I bang my head against the wall. That’s not quite true but it does reflect that I only understand at a very gross level how I construct new ideas. I’m bright but not very bright, and I can see that much smarter people have the same trouble). So how you are convinced that expert systems could construct new ideas is not at all clear to me.
To be sure, there have been some limited work with computer systems coming up with new, interesting ideas. There’s been some limited success with computers in my own field. See for example Simon Colton’s work. There’s also been similar work in geometry and group theory. But none of these systems were expert systems as that term is normally used. Moreover, none of the ideas they’ve come up with have that impressive. The only exception I’m aware of that is the proof of the Robbins conjecture. So even in narrow areas we’ve had very little success using specialized AIs. Are you using a more general definition of expert system than is standard?
Multiple problems with that claim. First, the existential threat may be low. There’s some tiny risk for example that the LHC will destroy the Earth in some very fun way. There’s also some risk that work with genetic engineering might give fanatics the skill to make a humanity destroying pathogen. And there’s a chance that nanotech might turn everything into purple with green stripes goo (this is much more likely than gray goo of course). There’s even some risk that proving the wrong theorem might summon Lovecraftian horrors. All events have some degree of risk. Moreover, general AI might actually help mitigate some serious threats, such as making it easier to track and deal with rogue asteroids or other catastrophic threats.
Also, even if one accepted the general outline of your argument, one would conclude that that’s a reason why organizations shouldn’t try to make general friendly AI. It isn’t a reason that actually having no AI is better than having no friendly AI.
“First, the existential threat [of AGI] may be low.”
Let me trace back the argument tree for a second. I originally asked for a defense of the claim that “SIAI is tackling the world’s most important task.” Michael Porter responded, “The real question is, do you even believe that unfriendly AI is a threat to the human race, and if so, is there anyone else tackling the problem in even a semi-competent way?” So NOW in this argument tree, we’re assuming that unfriendly AI IS an existential threat, enough that preventing it is the “world’s most important task.”
Now in this branch of the argument, I assumed (but did not state) the following: If unfriendly AI is an existential threat, friendly AI is an existential threat, as long as there is some chance of it being modified into unfriendly AI. Furthermore, I assert that it’s a naive notion that any organization could protect friendly AI from being subverted.
AIs, including ones with Friendly goals, are apt to work to protect their goal systems from modification, as this will prevent their efforts from being directed towards things other than their (current) aims. There might be a window while the AI is mid-FOOM where it’s vulnerable, but not a wide one.
How are you going to protect the source code before you run it?
A Friendly AI ought to protect itself from being subverted into an unfriendly AI.
Let me posit that FAI may be much less capable than unfriendly AI. The power of unfriendly AI is that it can increase its growth rate by taking resources by force. An FAI would be limited to what resources it could ethically obtain. Therefore, a low-grade FAI might be quite vulnerable to human antagonists, while its unrestricted version could be magnitudes of order more dangerous. In short, FAI could be low-reward high-risk.
There are plenty of resources that an FAI could ethically obtain, and with a lead of time of less than 1 day, it could grow enough to be vastly more powerful than an unfriendly seed AI.
Really, asking which AI wins going head to head is the wrong question. The goal is to get an FAI running before unfriendly AGI is implemented.
Wrong. FAI will make whatever unethical steps it must, as long as it’s on the net the best path it can see, taking into account both the (ethically harmful) instrumental actions and their expected outcome. There is no such general disadvantage coming with AI being Friendly. Not that I expect any need for such drastic measures (in an apparent way), especially considering the likely fist-mover advantage it’ll have.
If a program can take an understanding of those subjects and design a better computer chip, I don’t think it’s just an “expert system” anymore. I would think it would take an AI to do that. That’s an AI complete problem.
Are you serious? I would think the exact opposite would be true: we have an infrastructure starving for paradigm shifting ideas. I’d love to hear some of these revolutionary ideas that we’re saturated with. I think we have some insights, but these insights need to be fleshed out and implemented, and figuring out how to do that is the paradigm shift that needs to occur
Wait a minute. If I could press a button now with a 10% chance of destroying humanity and a 90% chance of solving the world’s problems, I’d do it. Everything we do has some risks. Even the LHC had an (extremely miniscule) risk of destroying the universe, but doing a cost-benefit analysis should reveal that some things are worth minor chances of destroying humanity.
“If a program can take an understanding of those subjects and design a better computer chip, I don’t think it’s just an “expert system” anymore. I would think it would take an AI to do that. That’s an AI complete problem.”
What I had in mind was some sort of combinatorial approach to designing chips, i.e. take these materials and randomly generate a design, test it, and then start altering the search space based on the results. I didn’t mean “understanding” in the human sense of the word, sorry.
“I’d love to hear some of these revolutionary ideas that we’re saturated with. I think we have some insights, but these insights need to be fleshed out and implemented, and figuring out how to do that is the paradigm shift that needs to occur”
Example: many aspects of the legal and political systems could be reformed, and it’s not difficult to come up with ideas on how they could be reformed. The benefit is simply insufficient to justify spending much of the limited resources we have on solving those problems.
“Wait a minute. If I could press a button now with a 10% chance of destroying humanity and a 90% chance of solving the world’s problems, I’d do it. ”
So you think there’s a >10% chance that the world’s problems are going to destroy humanity in the near future?
Given the very large number of possibilities and the difficulty with making prototypes, this seems like an extremely inefficient process without more thought going into to it.
Oh, okay, fair enough, though I’m still not sure I would call that an “expert system” (this time for the opposite reason that it seems too stupid).
Ah. I was thinking of designing an AI, probably because I was primed by your expert system comment. Well, in those cases, I think the issue is that our legal and political systems were purposely set up to be difficult to change: change requires overturning precedents, obtaining majority or 3⁄5 or 2⁄3 votes in various legislative bodies, passing constitutional amendments, and so forth. And I can guarantee you that for any of these reforms, there are powerful interests who would be harmed by the reforms, and many people who don’t want reform: this is more of a persuasion problem than an infrastructure problem. But yes, you’re right that there are plenty of revolutionary ideas about how to reform, say, the education system: they’re just not widely accepted enough to happen.
I’m confused by this sentence. I’m not sure if I think that, but what does it have to do with the hypothetical button that has a 10% chance of destroying humanity? My point was that it’s worth taking a small risk of destroying humanity if the benefits are great enough.
Bear in mind that the people who used steam engines to make money didn’t make it by selling the engines: rather, the engines were useful in producing other goods. I don’t think that the creators of a cheap substitute for human labor (GAI could be one such example) would be looking to sell it necessarily. They could simply want to develop such a tool in order to produce a wide array of goods at low cost.
I may think that I’m clever enough, for example, to keep it in a box and ask it for stock market predictions now and again. :)
As for the “no free lunch” business, while its true that any real-world GAI could not efficiently solve every induction problem, it wouldn’t need to either for it to be quite fearsome. Indeed being able to efficiently solve at least the same set of induction problems that humans solve (particularly if its in silicon and the hardware is relatively cheap) is sufficient to pose a big threat (and be potentially quite useful economically).
Also, there is a non-zero possibility that there already exists a GAI and its creators, decided the safest, most lucrative, and beneficial thing to do is set the GAI on designing drugs: thereby avoiding giving the GAI too much information about the world. The creators could have then set up a biotech company that just so happens to produce a few good drugs now and again. Its kind of like how automated trading came from computer scientists and not the currently employed traders. I do think its unlikely that somebody working in medical research is going to develop GAI least of all because of the job threat. The creators of a GAI are probably going to be full time professionals who are are working on the project.
I’m surprised that nobody so far has pointed out a rather obvious counter to my argument that “AGI will be politically unjustifiable.” I don’t post flawed arguments on purpose, but I usually realize counteraguments shortly after I post them. In any case, even if the popular sentiment in democracies is to block AGI, this doesn’t mean that other governments couldn’t support AGI. I wonder what the SIAI plans to do for the possibility of a hostile government funding unfriendly AI for military purposes.