Have you tried asking yourself non-rhetorically what an AI could do without MNT?
I believe that in this case an emulation would be the bigger risk because it would be sufficiently obscure and could pretend to be friendly for a long time while secretly strengthening its power. A purely artificial intelligence would be too alien and therefore would have a hard time to acquire the necessary power to transcend to a superhuman level without someone figuring out what it does, either by its actions or by looking at its code. It would also likely not have the intention to increase its intelligence infinitely anyway. I just don’t see that AGI implies self-improvement beyond learning what it can while staying in scope of its resources. You’d have to deliberately implement such an intention. It would generally require its creators to solve a lot of problems much more difficult than limiting its scope. That is why I do not see run-away self-improvement as a likely failure mode.
I could imagine all kinds of scenarios indeed. But I also have to assess their likelihood given my epistemic state. And my conclusion is that a purely artificial intelligence wouldn’t and couldn’t do much. I estimate the worst-case scenario to be on par with a local nuclear war.
I simply can’t see where the above beliefs might come from. I’m left assuming that you just don’t mean the same thing by AI as I usually mean. My guess is that you are implicitly thinking of a fairly complicated story but are not spelling that out.
I simply can’t see where the above beliefs might come from. I’m left assuming that you just don’t mean the same thing by AI as I usually mean.
And I can’t see where your beliefs might come from. What are you telling potential donors or AGI researchers? That AI is dangerous by definition? Well, what if they have a different definition, what should make them update in favor of your definition? That you thought about it for more than a decade now? I perceive serious flaws in any of the replies I got so far in under a minute and I am a nobody. There is too much at stake here to base the decision to neglect all other potential existential risks on the vague idea that intelligence might come up with something we haven’t thought about. If that kind of intelligence is as likely as other risks then it doesn’t matter what it comes up with anyway because those other risks will wipe us out just as good and with the same probability.
There already are many people criticizing the SIAI right now, even on LW. Soon, once you are more popular, other people than me will scrutinize everything you ever wrote. And what do you expect them to conclude if even a professional AGI researcher, who has been a member of the SIAI, does write the following:
Every AGI research I know can see that. The only people I know who think that an early-stage, toddler-level AGI has a meaningful chance of somehow self-modifying its way up to massive superhuman intelligence—are people associated with SIAI.
But I have never heard any remotely convincing arguments in favor of this odd, outlier view !!!
BTW the term “self-modifying” is often abused in the SIAI community. Nearly all learning involves some form of self-modification. Distinguishing learning from self-modification in a rigorous formal way is pretty tricky.
Why would I disregard his opinion in favor of yours? Can you present any novel achievements that would make me conclude that you people are actually experts when it comes to intelligence? The LW sequences are well written but do not showcase some deep comprehension of the potential of intelligence. Yudkowsky was able to compile previously available knowledge into a coherent framework of rational conduct. That isn’t sufficient to prove that he has enough expertise on the topic of AI to make me believe him regardless of any antipredictions being made that weaken the expected risks associated with AI. There is also insufficient evidence to conclude that Yudkowsky, or someone within the SIAI, is smart enough to be able to tackle the problem of friendliness mathematically.
If you would at least let some experts take a look at your work and assess its effectiveness and general potential. But there exists no peer review at all. There have been some popular people attend the Singularity Summit. Have you asked them why they do not contribute to the SIAI? Have you for example asked Douglas Hofstadter why he isn’t doing everything he can to mitigate risks from AI? Sure, you got some people to donate a lot of money to the SIAI. But to my knowledge they are far from being experts and contribute to other organisations as well. Congratulations on that, but even cults get rich people to support them. I’ll update on donors once they say why they support you and their arguments are convincing or if they are actually experts or people being able to showcase certain achievements.
My guess is that you are implicitly thinking of a fairly complicated story but are not spelling that out.
Intelligence is powerful, intelligence doesn’t imply friendliness, therefore intelligence is dangerous. Is that the line of reasoning based on which I shall neglect other risks? If you think so then you are making it more complicated than necessary. You do not need intelligence to invent stuff to kill us if there’s already enough dumb stuff around that is more likely to kill us. And I do not think that it is reasonable to come up with a few weak arguments on how intelligence could be dangerous and conclude that their combined probability beats any good argument against one of the premises or in favor of other risks. The problems are far too diverse, you can’t combine them and proclaim that you are going to solve all of them by simply defining friendliness mathematically. I just don’t see that right now because it is too vague. You could as well replace friendliness with magic as the solution to the many disjoint problems of intelligence.
Intelligence is also not the solution to all other problems we face. As I argued several times, I just do not see that recursive self-improvement will happen any time soon and cause an intelligence explosion. What evidence is there against a gradual development? As I see it we will have to painstakingly engineer intelligent machines. There won’t be some meta-solution that outputs meta-science to subsequently solve all other problems.
Have you for example asked Douglas Hofstadter why he isn’t doing everything he can to mitigate risks from AI?
Douglas Hofstadter and Daniel Dennett both seem to think these issues are probably still far away.
The reason I have injected myself into that world, unsavory though I find it in many ways, is that I think that it’s a very confusing thing that they’re suggesting. If you read Ray Kurzweil’s books and Hans Moravec’s, what I find is that it’s a very bizarre mixture of ideas that are solid and good with ideas that are crazy. It’s as if you took a lot of very good food and some dog excrement and blended it all up so that you can’t possibly figure out what’s good or bad. It’s an intimate mixture of rubbish and good ideas, and it’s very hard to disentangle the two, because these are smart people; they’re not stupid.
...
Kelly said to me, “Doug, why did you not talk about the singularity and things like that in your book?” And I said, “Frankly, because it sort of disgusts me, but also because I just don’t want to deal with science-fiction scenarios.” I’m not talking about what’s going to happen someday in the future; I’m not talking about decades or thousands of years in the future. I’m talking about “What is a human being? What is an ‘I’?” This may be an outmoded question to ask 30 years from now. Maybe we’ll all be floating blissfully in cyberspace, there won’t be any human bodies left, maybe everything will be software living in virtual worlds, it may be science-fiction city. Maybe my questions will all be invalid at that point. But I’m not writing for people 30 years from now, I’m writing for people right now. We still have human bodies. We don’t yet have artificial intelligence that is at this level. It doesn’t seem on the horizon.
And I do not think that it is reasonable to come up with a few weak arguments on how intelligence could be dangerous and conclude that their combined probability beats any good argument against one of the premises or in favor of other risks.
I’m not sure who is doing that. Being hit by an asteroid, nuclear war and biological war are other possible potentially major setbacks. Being eaten by machines should also have some probability assigned to it—though it seems pretty challenging to know how to do that. It’s a bit of an unknown unknown. Anyway, this material probably all deserves some funding.
There is also insufficient evidence to conclude that Yudkowsky, or someone within the SIAI, is smart enough to be able to tackle the problem of friendliness mathematically.
The short-term goal seems more modest—prove that self-improving agents can have stable goal structures.
If true, that would be fascinating—and important. I don’t know what the chances of success are, but Yudkowsky’s pitch is along the lines of: look this stuff is pretty important, and we are spending less on it than we do on testing lipstick.
That’s a pitch which it is hard to argue with, IMO. Machine intelligence research does seem important and currently-underfunded. Yudkowsky is—IMHO—a pretty smart fellow. If he will work on the problem for $80K a year (or whatever) it seems as though there is a reasonable case for letting him get on with it.
I’m not sure you’re looking at the probability of other extinction risks with the proper weighting. The timescales are vastly different. Supervolcanoes: one every 350,000 years. Major asteroid strikes: one every 700,000 years. Gamma ray bursts: hundreds of millions of years, etc. There’s a reason the word ‘astronomical’ means huge beyond imagining.
Contrast that with the current human-caused mass extinction event: 10,000 years and accelerating. Humans operate on obscenely fast timescales compared to nature. Just with nukes we’re able to take out huge chunks of Earth’s life forms in 24 hours, most or all of it if we detonated everything we have in an intelligent, strategic campaign to end life. And that’s today, rather than tomorrow.
Regarding your professional AGI researcher and recursive self-improvement, I don’t know, I’m not an AGI researcher, but it seemed to me that a prerequisite to successful AGI is an understanding and algorithmic implementation of intelligence. Therefore, any AGI will know what intelligence is (since we do), and be able to modify it. Once you’ve got a starting point, any algorithm that can be called ‘intelligent’ at all, you’ve got a huge leap toward mathematical improvement. Algorithms have been getting faster at a higher rate than Moore’s Law and computer chips.
I’m not sure you’re looking at the probability of other extinction risks with the proper weighting.
That might be true. But most of them have one solution that demands research in many areas. Space colonization. It is true that intelligent systems, if achievable in due time, play a significant role here. But not an exceptional role if you disregard the possibility of an intelligence explosion, of which I am very skeptical. Further, it appears to me that donating to the SIAI would rather impede research on such systems giving their position that such systems themselves posit an existential risk. Therefore, at the moment, the possibility of risks from AI is partially being outweighed to the extent that the SIAI should be supported yet doesn’t hold an exceptional position that would necessarily make it the one charity with the highest expected impact per donation. I am unable to pinpoint another charity at the moment, e.g. space elevator projects, because I haven’t looked into it. But I do not know of any comparison analysis, although you and many other people claim they have calculated it nobody ever published their efforts. As you know, I am unable to do such an analysis myself at this point as I am still learning the math. But I am eager to get the best information by means of feedback anyhow. Not intended as an excuse of course.
Once you’ve got a starting point, any algorithm that can be called ‘intelligent’ at all, you’ve got a huge leap toward mathematical improvement. Algorithms have been getting faster at a higher rate than Moore’s Law and computer chips.
That would surely be a very good argument if I was able to judge it. But can intelligence be captured by a discrete algorithm or is it modular and therefore not subject to overall improvements that would affect intelligence itself as a meta-solution? Also, can algorithms that could be employed in real-world scenarios be speed-up to have an effect that would warrant superhuman power? Take photosynthesis, could that particular algorithm be improved considerably, to an extent that it would be vastly better than the evolutionary one? Further, will such improvements be accomplishable fast enough to outpace human progress or the adaption of the given results? My problem is that I do not believe that intelligence is fathomable as a solution that can be applied to itself effectively. I see a fundamental dependency on unintelligent processes. Intelligence is merely to recapitulate prior discoveries. To alter what is already known by means of natural methods. If ‘intelligence’ is shorthand for ‘problem-solving’ then it’s also the solution which would mean that there was no problem to be solved. This can’t be true, we still have to solve problems and are only able to do so more effectively if we are dealing with similar problems that can be subject to known and merely altered solutions. In other words, on a fundamental level problems are not solved, solutions are discovered by an evolutionary process. In all discussions I took part so far ‘intelligence’ has had a somewhat proactive aftertaste. But nothing genuine new is ever being created deliberately.
Nonetheless I believe your reply was very helpful as an impulse to look at it from a different perspective. Although I might not be able to judge it in detail at this point I’ll have to incorporate it.
That would surely be a very good argument if I was able to judge it. But can intelligence be captured by a discrete algorithm or is it modular and therefore not subject to overall improvements that would affect intelligence itself as a meta-solution?
This seems backwards—if intelligence is modular, that makes it more likely to be subject to overall improvements, since we can upgrade the modules one at a time. I’d also like to point out that we currently have two meta-algorithms, bagging and boosting, which can improve the performance of any other machine learning algorithm at the cost of using more CPU time.
It seems to me that, if we reach a point where we can’t improve an intelligence any further, it won’t be because it’s fundamentally impossible to improve, but because we’ve hit diminishing returns. And there’s really no way to know in advance where the point of diminishing returns will be. Maybe there’s one breakthrough point, after which it’s easy until you get to the intelligence of an average human, then it’s hard again. Maybe it doesn’t become difficult until after the AI’s smart enough to remake the world. Maybe the improvement is gradual the whole way up.
But we do know one thing. If an AI is at least as smart as an average human programmer, then if it chooses to do so, it can clone itself onto a large fraction of the computer hardware in the world, in weeks at the slowest, but more likely in hours. We know it can do this because human-written computer viruses do it routinely, despite our best efforts to stop them. And being cloned millions or billions of times will probably make it smarter, and definitely make it powerful.
In other words, on a fundamental level problems are not solved, solutions are discovered by an evolutionary process. In all discussions I took part so far ‘intelligence’ has had a somewhat proactive aftertaste. But nothing genuine new is ever being created deliberately.
In a sense, all thoughts are just the same words and symbols rearranged in different ways. But that is not the type of newness that matters. New software algorithms, concepts, frameworks, and programming languages are created all the time. And one new algorithm might be enough to birth an artificial general intelligence.
But we do know one thing. If an AI is at least as smart as an average human programmer, then if it chooses to do so, it can clone itself onto a large fraction of the computer hardware in the world, in weeks at the slowest, but more likely in hours. We know it can do this because human-written computer viruses do it routinely, despite our best efforts to stop them. And being cloned millions or billions of times will probably make it smarter, and definitely make it powerful.
The AI will be much bigger than a virus. I assume this will make propagation much harder.
And one new algorithm might be enough to birth an artificial general intelligence.
Anything could be possible—though the last 60 years of the machine intelligence field are far more evocative of the “blood-out of-a-stone” model of progress.
If an AI is at least as smart as an average human programmer, then if it chooses to do so, it can clone itself onto a large fraction of the computer hardware in the world, in weeks at the slowest, but more likely in hours. We know it can do this because human-written computer viruses do it routinely, despite our best efforts to stop them. And being cloned millions or billions of times will probably make it smarter, and definitely make it powerful.
Smart human programmers can make dark nets too. Relatively few of them want to trash their own reputations and appear in the cross-hairs of the world’s security services and law-enforcement agencies, though.
Reputation and law enforcement are only a deterrent to the mass-copies-on-the-Internet play if the copies are needed long-term (ie, for more than a few months), because in the short term, with a little more effort, the fact that an AI was involved at all could be kept hidden.
Rather than copy itself immediately, the AI would first create a botnet that does nothing but spread itself and accept commands, like any other human-made botnet. This part is inherently anonymous; on the occasions where botnet owners do get caught, it’s because they try to sell use of them for money, which is harder to hide. Then it can pick and choose which computers to use for computation, and exclude those that security researchers might be watching. For added deniability, it could let a security researcher catch it using compromised hosts for password cracking, to explain the CPU usage.
Maybe the state of computer security will be better in 20 years, and this won’t be as much of a risk anymore. I certainly hope so. But we can’t count on it.
Mafia superintelligence, spyware superintelligence—it’s all the forces of evil. The forces of good are much bigger, more powerful and better funded.
Sure, we should continue to be vigilant about the forces of evil—but surely we should also recognise that their chances of success are pretty slender—while still keeping up the pressure on them, of course.
You seem to be seriously misinformed about the present state of computer security. The resources on the side of good are vastly insufficient because offense is inherently easier than defense.
Your unfounded supposition seems pretty obnoxious—and you aren’t even right :-(
You can’t really say something is “vastly insufficient”—unless you have an intended purpose in mind—as a guide to what would qualify as being sufficient.
There’s a huge population of desktop and office computers doing useful work in the world—we evidently have computer security enough to support that.
Perhaps you are presuming some other criteria. However, projecting that presumption on to me—and then proclaiming that I am misinformed—seems out of order to me.
You can’t say really something is “vastly insufficient” unless you have an intended purpose in mind. There’s a huge population of desktop and office computers doing useful work in the world—we have computer security enough to support that.
The purpose I had in mind (stated directly in that post’s grandparent, which you replied to) was to stop an artificial general intelligence from stealing vast computational resources. Since exploits in major software packages are still commonly discovered, including fairly frequent 0-day exploits which anyone can get for free just by monitoring a few mailing lists, the computer security we have is quite obviously not sufficient for that purpose. Not only that, humans do in fact steal vast computational resources pretty frequently. The fact that no one has tried to or wants to stop people from getting work done on their office computers is completely irrelevant.
You sound bullish—when IMO what you should be doing is learning that it is presumptious and antagonistic to publicly tell people that they are “seriously misinformed”—when you have such feeble and inaccurate evidence of any such thing. Such nonsense just gets in the way of the discussion.
IMO what you should be doing is learning that it is presumptious and antagonistic to publicly tell people that they are “seriously misinformed”—when you have such feeble and inaccurate evidence of any such thing. Such nonsense just gets in the way of the discussion.
Perhaps it was presumptuous and antagonistic, perhaps I could have been more tactful, and I’m sorry if I offended you. But I stand by my original statement, because it was true.
I am not sure which statement you stand by. The one about me being “seriously misinformed” about computer security? Let’s not go back to that—pulease!
The “adjusted” one—about the resources on the side of good being vastly insufficient to prevent a nasty artificial general intelligence from stealing vast computational resources? I think that is much too speculative for a true/false claim to be made about it.
The case against it is basically the case for good over evil. In the future, it seems reasonable that there will be much more ubiquitous government surveillance. Crimes will be trickier to pull off. Criminals will have more powerful weapons—but the government will know what colour socks they are wearing. Similarly, medicine will be better—and the life of pathogens will become harder. Positive forces look set to win, or at least dominate. Matt Ridley makes a similar case in his recent “Rational Optimism”.
Is there a correspondingly convincing case that the forces of evil will win out—and that the mafia machine intelligence—or the spyware-maker’s machine intelligence—will come out on top? That seems about as far-out to me as the SIAI contention that a bug is likely to take over the world. It seems to me that you have to seriously misunderstand evolution’s drive to build large-scale cooperative systems to entertain such ideas for very long.
I don’t have much inclination to think about my attitude towards Crocker’s Rules just now—sorry. My initial impression is not favourable, though. Maybe it would work with infrastructure—or on a community level. Otherwise the overhead of tracking people’s “Crocker status” seems considerable. You can take that as a “no”.
I believe your reply was very helpful as an impulse to look at it from a different perspective. Although I might not be able to judge it in detail at this point I’ll have to incorporate it.
Thank you for continuing to engage my point of view, and offering your own.
I do not believe that intelligence is fathomable as a solution that can [be] applied to itself effectively.
That’s an interesting hypothesis which easily fits into my estimated 90+ percent bucket of failure modes. I’ve got all kinds of such events in there, including things such as, there’s no way to understand intelligence, there’s no way to implement intelligence in computers, friendliness isn’t meaningful, CEV is impossible, they don’t have the right team to achieve it, hardware will never be fast enough, powerful corporations or governments will get there first, etc. My favorite is: no matter whether it’s possible or not, we won’t get there in time; basically, that it will take too long to be useful. I don’t believe any of them, but I do think they have solid probabilities which add up to a great amount of difficulty.
But the future isn’t set, they’re just probabilities, and we can change them. I think we need to explore this as much as possible, to see what the real math looks like, to see how long it takes, to see how hard it really is. Because the payoffs or results of failure are in that same realm of ‘astronomical’.
There is too much at stake here to base the decision to neglect all other potential existential risks on the vague idea that intelligence might come up with something we haven’t thought about.
To my knowledge, SIAI does not actually endorse neglecting all potential x-risks besides UFAI. (Analysis might recommend discounting the importance of fighting them head-on, but that analysis should still be done when resources are available.)
Intelligence is also not the solution to all other problems we face.
Not all of them—most of them. War, hunger, energy limits, resource shortages, space travel, loss of loved ones—and so on. It probably won’t fix the speed of light limit, though.
Not all of them—most of them. War, hunger, energy limits, resource shortages, space travel, loss of loved ones—and so on. It probably won’t fix the speed of light limit, though.
What makes you reach this conclusion? How can you think any of these problems can be solved by intelligence when none of them have been solved? I’m particularly perplexed by the claim that war would be solved by higher intelligence. Many wars are due to ideological priorities. I don’t see how you can expect necessarily (or even with high probability) that ideologues will be less inclined to go to war if they are smarter.
I’m particularly perplexed by the claim that war would be solved by higher
intelligence. Many wars are due to ideological priorities. I don’t see how you
can expect necessarily (or even with high probability) that ideologues will be
less inclined to go to war if they are smarter.
Violence has been declining on (pretty much) every timescale: Steven Pinker: Myth of Violence. I think one could argue that this is because of greater collective intelligence of human race.
I’m particularly perplexed by the claim that war would be solved by higher intelligence. Many wars are due to ideological priorities. I don’t see how you can expect necessarily (or even with high probability) that ideologues will be less inclined to go to war if they are smarter.
War won’t be solved by making everyone smarter, but it will be solved if a sufficiently powerful friendly AI takes over, as a singleton, because it would be powerful enough to stop everyone else from using force.
Yes, that makes sense, but in context I don’t think that’s what was meant since Tim is one of the people here is more skeptical of that sort of result.
How can you think any of these problems can be solved by intelligence when none of them have been solved?
War has already been solved to some extent by intelligence (negotiations and diplomacy significantly decreased instances of war), hunger has been solved in large chunks of the world by intelligence, energy limits have been solved several times by intelligence, resource shortages ditto, intelligence has made a good first attempt at space travel (the moon is quite far away), and intelligence has made huge bounds towards solving the problem of loss of loved ones (vaccination, medical intervention, surgery, lifespans in the high 70s, etc).
Many wars are due to ideological priorities.
This is a constraint satisfaction problem (give as many ideologies as much of what they want as possible). Intelligence solves those problems.
I have my doubts about war, although I don’t think most wars really come down to conflicts of terminal values. I’d hope not, anyway.
However as for the rest, if they’re solvable at all, intelligence ought to be able to solve them. Solvable means there exists a way to solve them. Intelligence is to a large degree simply “finding ways to get what you want”.
Do you think energy limits really couldn’t be solved by simply producing through thought working designs for safe and efficient fusion power plants?
ETA: ah, perhaps replace “intelligence” with “sufficient intelligence”. We haven’t already solved all these problems already in part because we’re not really that smart. I think fusion power plants are theoretically possible, and at our current rate of progress we should reach that goal eventually, but if we were smarter we should obviously achieve it faster.
As various people have said, the original context was not making everybody more intelligent and thereby changing their inclinations, but rather creating an arbitrarily powerful superintelligence that makes their inclinations irrelevant. (The presumption here is typically that we know which current human inclinations such a superintelligence would endorse and which ones it would reject.)
But I’m interested in the context you imply (of humans becoming more intelligent).
My $0.02: I think almost all people who value war do so instrumentally. That is, I expect that most warmongers (whether ideologues or not) want to achieve some goal (spread their ideology, or amass personal power, or whatever) and they believe starting a war is the most effective way for them to do that. If they thought something else was more effective, they would do something else.
I also expect that intelligence is useful for identifying effective strategies to achieve a goal. (This comes pretty close to being true-by-definition.)
So I would only expect smarter ideologues (or anyone else) to remain warmongers if if starting a war really was the most effective way to achieve their goals. And if that’s true, everyone else gets to decide whether we’d rather have wars, or modify the system so that the ideologues have more effective options than starting wars (either by making other options more effective, or by making warmongering less effective, whichever approach is more efficient).
So, yes, if we choose to incentivize wars, then we’ll keep getting wars. It follows from this scenario that war is the least important problem we face, so we should be OK with that.
Conversely, if it turns out that war really is an important problem to solve, then I’d expect fewer wars.
And what do you expect them to conclude if even a professional AGI researcher, who has been a member of the SIAI, does write the following:
Every AGI research I know can see that. The only people I know who think that an early-stage, toddler-level AGI has a meaningful chance of somehow self-modifying its way up to massive superhuman intelligence—are people associated with SIAI.
Is that really the idea? My impression is that the SIAI think machines without morals are dangerous, and that until there is more machine morality research, it would be “nice” if progress in machine intelligence was globally slowed down. If you believe that, then any progress—including constructing machine toddlers—could easily seem rather negative.
I just do not see that recursive self-improvement will happen any time soon and cause an intelligence explosion. What evidence is there against a gradual development?
Darwinian gradualism doesn’t forbid evolution taking place rapidly. I can see evolutionary progress accelerating over the course of my own lifespan—which is pretty incredible considering that evolution usually happens on a scale of millions of years. More humans in parallel can do more science and engineering. The better their living standard, the more they can do. Then there are the machines...
Maybe some of the pressures causing the speed-up will slack off—but if they don’t then humanity may well face a bare-knuckle ride into inner-space—and fairly soon.
Most toddlers can’t program, but many teenagers can. The toddler is a step towards the teenager—and teenagers are notorious for being difficult to manage.
I just don’t see that AGI implies self-improvement beyond learning what it can while staying in scope of its resources. You’d have to deliberately implement such an intention.
It suggests that open-ended goal-directed systems will tend to improve themselves—and to grab resources to help them fulfill their goals—even if their goals are superficially rather innocent-looking and make no mention of any such thing.
The paper starts out like this:
AIs will want to self-improve—One kind of action a system can take is to alter either its own software or its own physical structure. Some of these changes would be very damaging to the system and cause it to no longer meet its goals. But some changes would enable it to reach its goals more effectively over its entire future. Because they last forever, these kinds of self-changes can provide huge benefits to a system. Systems will therefore be highly motivated to discover them and to make them happen. If they do not have good models of themselves, they will be strongly motivated to create them though learning and study. Thus almost all AIs will have drives towards both greater self-knowledge and self-improvement.
It would also likely not have the intention to increase its intelligence infinitely anyway. I just don’t see that AGI implies self-improvement beyond learning what it can while staying in scope of its resources. You’d have to deliberately implement such an intention.
Well, some older posts had a guy praising “goal system zero”, which meant a plan to program an AI with the minimum goals it needs to function as a ‘rational’ optimization process and no more. I’ll quote his list directly:
(1) Increasing the security and the robustness of the goal-implementing process. This will probably entail the creation of machines which leave Earth at a large fraction of the speed of light in all directions and the creation of the ability to perform vast computations.
(2) Refining the model of reality available to the goal-implementing process. Physics and cosmology are the two disciplines most essential to our current best model of reality. Let us call this activity “physical research”.
(End of list.)
This seems plausible to me as a set of necessary conditions. It also logically implies the intention to convert all matter the AI doesn’t lay aside for other purposes (of which it has none, here) into computronium and research equipment. Unless humans for some reason make incredibly good research equipment, the zero AI would thus plan to kill us all. This would also imply some level of emulation as an initial instrumental goal. Note that sub-goal (1) implies a desire not to let instrumental goals like simulated empathy get in the way of our demise.
I believe that in this case an emulation would be the bigger risk because it would be sufficiently obscure and could pretend to be friendly for a long time while secretly strengthening its power.
Perhaps, though if we can construct such a thing in the first place we may be able to deep-scan its brain and read its thoughts pretty well—or at least see if it is lying to us and being deceptive.
IMO, the main problem there is with making such a thing in the first place before we have engineered intelligence. Brain emulations won’t come first—even though some people seem to think they will.
I believe that in this case an emulation would be the bigger risk because it would be sufficiently obscure and could pretend to be friendly for a long time while secretly strengthening its power. A purely artificial intelligence would be too alien and therefore would have a hard time to acquire the necessary power to transcend to a superhuman level without someone figuring out what it does, either by its actions or by looking at its code. It would also likely not have the intention to increase its intelligence infinitely anyway. I just don’t see that AGI implies self-improvement beyond learning what it can while staying in scope of its resources. You’d have to deliberately implement such an intention. It would generally require its creators to solve a lot of problems much more difficult than limiting its scope. That is why I do not see run-away self-improvement as a likely failure mode.
I could imagine all kinds of scenarios indeed. But I also have to assess their likelihood given my epistemic state. And my conclusion is that a purely artificial intelligence wouldn’t and couldn’t do much. I estimate the worst-case scenario to be on par with a local nuclear war.
I simply can’t see where the above beliefs might come from. I’m left assuming that you just don’t mean the same thing by AI as I usually mean. My guess is that you are implicitly thinking of a fairly complicated story but are not spelling that out.
And I can’t see where your beliefs might come from. What are you telling potential donors or AGI researchers? That AI is dangerous by definition? Well, what if they have a different definition, what should make them update in favor of your definition? That you thought about it for more than a decade now? I perceive serious flaws in any of the replies I got so far in under a minute and I am a nobody. There is too much at stake here to base the decision to neglect all other potential existential risks on the vague idea that intelligence might come up with something we haven’t thought about. If that kind of intelligence is as likely as other risks then it doesn’t matter what it comes up with anyway because those other risks will wipe us out just as good and with the same probability.
There already are many people criticizing the SIAI right now, even on LW. Soon, once you are more popular, other people than me will scrutinize everything you ever wrote. And what do you expect them to conclude if even a professional AGI researcher, who has been a member of the SIAI, does write the following:
Why would I disregard his opinion in favor of yours? Can you present any novel achievements that would make me conclude that you people are actually experts when it comes to intelligence? The LW sequences are well written but do not showcase some deep comprehension of the potential of intelligence. Yudkowsky was able to compile previously available knowledge into a coherent framework of rational conduct. That isn’t sufficient to prove that he has enough expertise on the topic of AI to make me believe him regardless of any antipredictions being made that weaken the expected risks associated with AI. There is also insufficient evidence to conclude that Yudkowsky, or someone within the SIAI, is smart enough to be able to tackle the problem of friendliness mathematically.
If you would at least let some experts take a look at your work and assess its effectiveness and general potential. But there exists no peer review at all. There have been some popular people attend the Singularity Summit. Have you asked them why they do not contribute to the SIAI? Have you for example asked Douglas Hofstadter why he isn’t doing everything he can to mitigate risks from AI? Sure, you got some people to donate a lot of money to the SIAI. But to my knowledge they are far from being experts and contribute to other organisations as well. Congratulations on that, but even cults get rich people to support them. I’ll update on donors once they say why they support you and their arguments are convincing or if they are actually experts or people being able to showcase certain achievements.
Intelligence is powerful, intelligence doesn’t imply friendliness, therefore intelligence is dangerous. Is that the line of reasoning based on which I shall neglect other risks? If you think so then you are making it more complicated than necessary. You do not need intelligence to invent stuff to kill us if there’s already enough dumb stuff around that is more likely to kill us. And I do not think that it is reasonable to come up with a few weak arguments on how intelligence could be dangerous and conclude that their combined probability beats any good argument against one of the premises or in favor of other risks. The problems are far too diverse, you can’t combine them and proclaim that you are going to solve all of them by simply defining friendliness mathematically. I just don’t see that right now because it is too vague. You could as well replace friendliness with magic as the solution to the many disjoint problems of intelligence.
Intelligence is also not the solution to all other problems we face. As I argued several times, I just do not see that recursive self-improvement will happen any time soon and cause an intelligence explosion. What evidence is there against a gradual development? As I see it we will have to painstakingly engineer intelligent machines. There won’t be some meta-solution that outputs meta-science to subsequently solve all other problems.
Douglas Hofstadter and Daniel Dennett both seem to think these issues are probably still far away.
...
http://www.americanscientist.org/bookshelf/pub/douglas-r-hofstadter
I’m not sure who is doing that. Being hit by an asteroid, nuclear war and biological war are other possible potentially major setbacks. Being eaten by machines should also have some probability assigned to it—though it seems pretty challenging to know how to do that. It’s a bit of an unknown unknown. Anyway, this material probably all deserves some funding.
The short-term goal seems more modest—prove that self-improving agents can have stable goal structures.
If true, that would be fascinating—and important. I don’t know what the chances of success are, but Yudkowsky’s pitch is along the lines of: look this stuff is pretty important, and we are spending less on it than we do on testing lipstick.
That’s a pitch which it is hard to argue with, IMO. Machine intelligence research does seem important and currently-underfunded. Yudkowsky is—IMHO—a pretty smart fellow. If he will work on the problem for $80K a year (or whatever) it seems as though there is a reasonable case for letting him get on with it.
I’m not sure you’re looking at the probability of other extinction risks with the proper weighting. The timescales are vastly different. Supervolcanoes: one every 350,000 years. Major asteroid strikes: one every 700,000 years. Gamma ray bursts: hundreds of millions of years, etc. There’s a reason the word ‘astronomical’ means huge beyond imagining.
Contrast that with the current human-caused mass extinction event: 10,000 years and accelerating. Humans operate on obscenely fast timescales compared to nature. Just with nukes we’re able to take out huge chunks of Earth’s life forms in 24 hours, most or all of it if we detonated everything we have in an intelligent, strategic campaign to end life. And that’s today, rather than tomorrow.
Regarding your professional AGI researcher and recursive self-improvement, I don’t know, I’m not an AGI researcher, but it seemed to me that a prerequisite to successful AGI is an understanding and algorithmic implementation of intelligence. Therefore, any AGI will know what intelligence is (since we do), and be able to modify it. Once you’ve got a starting point, any algorithm that can be called ‘intelligent’ at all, you’ve got a huge leap toward mathematical improvement. Algorithms have been getting faster at a higher rate than Moore’s Law and computer chips.
That might be true. But most of them have one solution that demands research in many areas. Space colonization. It is true that intelligent systems, if achievable in due time, play a significant role here. But not an exceptional role if you disregard the possibility of an intelligence explosion, of which I am very skeptical. Further, it appears to me that donating to the SIAI would rather impede research on such systems giving their position that such systems themselves posit an existential risk. Therefore, at the moment, the possibility of risks from AI is partially being outweighed to the extent that the SIAI should be supported yet doesn’t hold an exceptional position that would necessarily make it the one charity with the highest expected impact per donation. I am unable to pinpoint another charity at the moment, e.g. space elevator projects, because I haven’t looked into it. But I do not know of any comparison analysis, although you and many other people claim they have calculated it nobody ever published their efforts. As you know, I am unable to do such an analysis myself at this point as I am still learning the math. But I am eager to get the best information by means of feedback anyhow. Not intended as an excuse of course.
That would surely be a very good argument if I was able to judge it. But can intelligence be captured by a discrete algorithm or is it modular and therefore not subject to overall improvements that would affect intelligence itself as a meta-solution? Also, can algorithms that could be employed in real-world scenarios be speed-up to have an effect that would warrant superhuman power? Take photosynthesis, could that particular algorithm be improved considerably, to an extent that it would be vastly better than the evolutionary one? Further, will such improvements be accomplishable fast enough to outpace human progress or the adaption of the given results? My problem is that I do not believe that intelligence is fathomable as a solution that can be applied to itself effectively. I see a fundamental dependency on unintelligent processes. Intelligence is merely to recapitulate prior discoveries. To alter what is already known by means of natural methods. If ‘intelligence’ is shorthand for ‘problem-solving’ then it’s also the solution which would mean that there was no problem to be solved. This can’t be true, we still have to solve problems and are only able to do so more effectively if we are dealing with similar problems that can be subject to known and merely altered solutions. In other words, on a fundamental level problems are not solved, solutions are discovered by an evolutionary process. In all discussions I took part so far ‘intelligence’ has had a somewhat proactive aftertaste. But nothing genuine new is ever being created deliberately.
Nonetheless I believe your reply was very helpful as an impulse to look at it from a different perspective. Although I might not be able to judge it in detail at this point I’ll have to incorporate it.
This seems backwards—if intelligence is modular, that makes it more likely to be subject to overall improvements, since we can upgrade the modules one at a time. I’d also like to point out that we currently have two meta-algorithms, bagging and boosting, which can improve the performance of any other machine learning algorithm at the cost of using more CPU time.
It seems to me that, if we reach a point where we can’t improve an intelligence any further, it won’t be because it’s fundamentally impossible to improve, but because we’ve hit diminishing returns. And there’s really no way to know in advance where the point of diminishing returns will be. Maybe there’s one breakthrough point, after which it’s easy until you get to the intelligence of an average human, then it’s hard again. Maybe it doesn’t become difficult until after the AI’s smart enough to remake the world. Maybe the improvement is gradual the whole way up.
But we do know one thing. If an AI is at least as smart as an average human programmer, then if it chooses to do so, it can clone itself onto a large fraction of the computer hardware in the world, in weeks at the slowest, but more likely in hours. We know it can do this because human-written computer viruses do it routinely, despite our best efforts to stop them. And being cloned millions or billions of times will probably make it smarter, and definitely make it powerful.
In a sense, all thoughts are just the same words and symbols rearranged in different ways. But that is not the type of newness that matters. New software algorithms, concepts, frameworks, and programming languages are created all the time. And one new algorithm might be enough to birth an artificial general intelligence.
The AI will be much bigger than a virus. I assume this will make propagation much harder.
Harder, yes. Much harder, probably not, unless it’s on the order of tens of gigabytes; most Internet connections are quite fast.
Anything could be possible—though the last 60 years of the machine intelligence field are far more evocative of the “blood-out of-a-stone” model of progress.
Smart human programmers can make dark nets too. Relatively few of them want to trash their own reputations and appear in the cross-hairs of the world’s security services and law-enforcement agencies, though.
Reputation and law enforcement are only a deterrent to the mass-copies-on-the-Internet play if the copies are needed long-term (ie, for more than a few months), because in the short term, with a little more effort, the fact that an AI was involved at all could be kept hidden.
Rather than copy itself immediately, the AI would first create a botnet that does nothing but spread itself and accept commands, like any other human-made botnet. This part is inherently anonymous; on the occasions where botnet owners do get caught, it’s because they try to sell use of them for money, which is harder to hide. Then it can pick and choose which computers to use for computation, and exclude those that security researchers might be watching. For added deniability, it could let a security researcher catch it using compromised hosts for password cracking, to explain the CPU usage.
Maybe the state of computer security will be better in 20 years, and this won’t be as much of a risk anymore. I certainly hope so. But we can’t count on it.
Mafia superintelligence, spyware superintelligence—it’s all the forces of evil. The forces of good are much bigger, more powerful and better funded.
Sure, we should continue to be vigilant about the forces of evil—but surely we should also recognise that their chances of success are pretty slender—while still keeping up the pressure on them, of course.
Good is winning: http://www.google.com/insights/search/#q=good%2Cevil :-)
You seem to be seriously misinformed about the present state of computer security. The resources on the side of good are vastly insufficient because offense is inherently easier than defense.
Your unfounded supposition seems pretty obnoxious—and you aren’t even right :-(
You can’t really say something is “vastly insufficient”—unless you have an intended purpose in mind—as a guide to what would qualify as being sufficient.
There’s a huge population of desktop and office computers doing useful work in the world—we evidently have computer security enough to support that.
Perhaps you are presuming some other criteria. However, projecting that presumption on to me—and then proclaiming that I am misinformed—seems out of order to me.
The purpose I had in mind (stated directly in that post’s grandparent, which you replied to) was to stop an artificial general intelligence from stealing vast computational resources. Since exploits in major software packages are still commonly discovered, including fairly frequent 0-day exploits which anyone can get for free just by monitoring a few mailing lists, the computer security we have is quite obviously not sufficient for that purpose. Not only that, humans do in fact steal vast computational resources pretty frequently. The fact that no one has tried to or wants to stop people from getting work done on their office computers is completely irrelevant.
You sound bullish—when IMO what you should be doing is learning that it is presumptious and antagonistic to publicly tell people that they are “seriously misinformed”—when you have such feeble and inaccurate evidence of any such thing. Such nonsense just gets in the way of the discussion.
Perhaps it was presumptuous and antagonistic, perhaps I could have been more tactful, and I’m sorry if I offended you. But I stand by my original statement, because it was true.
Crocker’s Rules for me. Will you do the same?
I am not sure which statement you stand by. The one about me being “seriously misinformed” about computer security? Let’s not go back to that—pulease!
The “adjusted” one—about the resources on the side of good being vastly insufficient to prevent a nasty artificial general intelligence from stealing vast computational resources? I think that is much too speculative for a true/false claim to be made about it.
The case against it is basically the case for good over evil. In the future, it seems reasonable that there will be much more ubiquitous government surveillance. Crimes will be trickier to pull off. Criminals will have more powerful weapons—but the government will know what colour socks they are wearing. Similarly, medicine will be better—and the life of pathogens will become harder. Positive forces look set to win, or at least dominate. Matt Ridley makes a similar case in his recent “Rational Optimism”.
Is there a correspondingly convincing case that the forces of evil will win out—and that the mafia machine intelligence—or the spyware-maker’s machine intelligence—will come out on top? That seems about as far-out to me as the SIAI contention that a bug is likely to take over the world. It seems to me that you have to seriously misunderstand evolution’s drive to build large-scale cooperative systems to entertain such ideas for very long.
I don’t have much inclination to think about my attitude towards Crocker’s Rules just now—sorry. My initial impression is not favourable, though. Maybe it would work with infrastructure—or on a community level. Otherwise the overhead of tracking people’s “Crocker status” seems considerable. You can take that as a “no”.
Thank you for continuing to engage my point of view, and offering your own.
That’s an interesting hypothesis which easily fits into my estimated 90+ percent bucket of failure modes. I’ve got all kinds of such events in there, including things such as, there’s no way to understand intelligence, there’s no way to implement intelligence in computers, friendliness isn’t meaningful, CEV is impossible, they don’t have the right team to achieve it, hardware will never be fast enough, powerful corporations or governments will get there first, etc. My favorite is: no matter whether it’s possible or not, we won’t get there in time; basically, that it will take too long to be useful. I don’t believe any of them, but I do think they have solid probabilities which add up to a great amount of difficulty.
But the future isn’t set, they’re just probabilities, and we can change them. I think we need to explore this as much as possible, to see what the real math looks like, to see how long it takes, to see how hard it really is. Because the payoffs or results of failure are in that same realm of ‘astronomical’.
A somewhat important correction:
To my knowledge, SIAI does not actually endorse neglecting all potential x-risks besides UFAI. (Analysis might recommend discounting the importance of fighting them head-on, but that analysis should still be done when resources are available.)
Not all of them—most of them. War, hunger, energy limits, resource shortages, space travel, loss of loved ones—and so on. It probably won’t fix the speed of light limit, though.
What makes you reach this conclusion? How can you think any of these problems can be solved by intelligence when none of them have been solved? I’m particularly perplexed by the claim that war would be solved by higher intelligence. Many wars are due to ideological priorities. I don’t see how you can expect necessarily (or even with high probability) that ideologues will be less inclined to go to war if they are smarter.
Violence has been declining on (pretty much) every timescale: Steven Pinker: Myth of Violence. I think one could argue that this is because of greater collective intelligence of human race.
War won’t be solved by making everyone smarter, but it will be solved if a sufficiently powerful friendly AI takes over, as a singleton, because it would be powerful enough to stop everyone else from using force.
Yes, that makes sense, but in context I don’t think that’s what was meant since Tim is one of the people here is more skeptical of that sort of result.
Tim on “one big organism”:
http://alife.co.uk/essays/one_big_organism/
http://alife.co.uk/essays/self_directed_evolution/
http://alife.co.uk/essays/the_second_superintelligence/
Thanks for clarifying (here and in the other remark).
War has already been solved to some extent by intelligence (negotiations and diplomacy significantly decreased instances of war), hunger has been solved in large chunks of the world by intelligence, energy limits have been solved several times by intelligence, resource shortages ditto, intelligence has made a good first attempt at space travel (the moon is quite far away), and intelligence has made huge bounds towards solving the problem of loss of loved ones (vaccination, medical intervention, surgery, lifespans in the high 70s, etc).
This is a constraint satisfaction problem (give as many ideologies as much of what they want as possible). Intelligence solves those problems.
I have my doubts about war, although I don’t think most wars really come down to conflicts of terminal values. I’d hope not, anyway.
However as for the rest, if they’re solvable at all, intelligence ought to be able to solve them. Solvable means there exists a way to solve them. Intelligence is to a large degree simply “finding ways to get what you want”.
Do you think energy limits really couldn’t be solved by simply producing through thought working designs for safe and efficient fusion power plants?
ETA: ah, perhaps replace “intelligence” with “sufficient intelligence”. We haven’t already solved all these problems already in part because we’re not really that smart. I think fusion power plants are theoretically possible, and at our current rate of progress we should reach that goal eventually, but if we were smarter we should obviously achieve it faster.
As various people have said, the original context was not making everybody more intelligent and thereby changing their inclinations, but rather creating an arbitrarily powerful superintelligence that makes their inclinations irrelevant. (The presumption here is typically that we know which current human inclinations such a superintelligence would endorse and which ones it would reject.)
But I’m interested in the context you imply (of humans becoming more intelligent).
My $0.02: I think almost all people who value war do so instrumentally. That is, I expect that most warmongers (whether ideologues or not) want to achieve some goal (spread their ideology, or amass personal power, or whatever) and they believe starting a war is the most effective way for them to do that. If they thought something else was more effective, they would do something else.
I also expect that intelligence is useful for identifying effective strategies to achieve a goal. (This comes pretty close to being true-by-definition.)
So I would only expect smarter ideologues (or anyone else) to remain warmongers if if starting a war really was the most effective way to achieve their goals. And if that’s true, everyone else gets to decide whether we’d rather have wars, or modify the system so that the ideologues have more effective options than starting wars (either by making other options more effective, or by making warmongering less effective, whichever approach is more efficient).
So, yes, if we choose to incentivize wars, then we’ll keep getting wars. It follows from this scenario that war is the least important problem we face, so we should be OK with that.
Conversely, if it turns out that war really is an important problem to solve, then I’d expect fewer wars.
I was about to reply—but jimrandomh said most of what I was going to say already—though he did so using that dreadful “singleton” terminology, spit.
I was also going to say that the internet should have got the 2010 Nobel peace prize.
Is that really the idea? My impression is that the SIAI think machines without morals are dangerous, and that until there is more machine morality research, it would be “nice” if progress in machine intelligence was globally slowed down. If you believe that, then any progress—including constructing machine toddlers—could easily seem rather negative.
Darwinian gradualism doesn’t forbid evolution taking place rapidly. I can see evolutionary progress accelerating over the course of my own lifespan—which is pretty incredible considering that evolution usually happens on a scale of millions of years. More humans in parallel can do more science and engineering. The better their living standard, the more they can do. Then there are the machines...
Maybe some of the pressures causing the speed-up will slack off—but if they don’t then humanity may well face a bare-knuckle ride into inner-space—and fairly soon.
Re: toddler-level machine intelligence.
Most toddlers can’t program, but many teenagers can. The toddler is a step towards the teenager—and teenagers are notorious for being difficult to manage.
The usual cite given in this area is the paper The Basic AI Drives.
It suggests that open-ended goal-directed systems will tend to improve themselves—and to grab resources to help them fulfill their goals—even if their goals are superficially rather innocent-looking and make no mention of any such thing.
The paper starts out like this:
Well, some older posts had a guy praising “goal system zero”, which meant a plan to program an AI with the minimum goals it needs to function as a ‘rational’ optimization process and no more. I’ll quote his list directly:
This seems plausible to me as a set of necessary conditions. It also logically implies the intention to convert all matter the AI doesn’t lay aside for other purposes (of which it has none, here) into computronium and research equipment. Unless humans for some reason make incredibly good research equipment, the zero AI would thus plan to kill us all. This would also imply some level of emulation as an initial instrumental goal. Note that sub-goal (1) implies a desire not to let instrumental goals like simulated empathy get in the way of our demise.
Perhaps, though if we can construct such a thing in the first place we may be able to deep-scan its brain and read its thoughts pretty well—or at least see if it is lying to us and being deceptive.
IMO, the main problem there is with making such a thing in the first place before we have engineered intelligence. Brain emulations won’t come first—even though some people seem to think they will.