Yes, you may argue: the Scary Idea hasn’t been rigorously shown to be true… but what if it IS true?
OK but … pointing out that something scary is possible, is a very different thing from having an argument that it’s likely.
The Scary Idea is certainly something to keep in mind, but there are also many other risks to keep in mind, some much more definite and palpable.
[...]
Also, there are always possibilities like: the alien race that is watching us and waiting for us to achieve an IQ of 333, at which point it will swoop down upon us and eat us, or merge with us. We can’t rule this out via any formal proof, and we can’t meaningfully estimate the odds of it either. Yes, this sounds science-fictional and outlandish; but is it really more outlandish and speculative than the Scary Idea?
The issue with potential risks posed by unfriendly AI are numerous. The only organisation that takes those issues serious is the SIAI, as its name already implies. But I believe most people simply don’t see a difference between the SIAI and one or a few highly intelligent people telling them that a particle collider could destroy the world while all experts working directly on it claim there’s no risk. Now I think I understand the argument that if the whole world is at stake it does outweigh the low probability of the event. But does it? I think it is completely justified to have at least one organisation working on FAI, but is the risk as serious as portrayed and perceived within the SIAI? Right now if I had to hazard a guess I’d say that it will probably be a gradual development of many exponential growth phases. That is, we’ll have this conceptual revolution and optimize it very rapidly. Then the next revolution will be necessary. Sure, I might be wrong there, as the plateau argument of self-improvement recursion might hold. But even if that is true, I think we’ll need at least two paradigm-shattering conceptual revolutions before we get there. But what does that mean though? How quickly can such revolutions happen? I’m guessing that this could take a long time, if it isn’t completely impossible. That is, if we are not the equivalent of Universal Turing Machine of abstract reasoning. Just imagine we are merely better chimps. Maybe it doesn’t matter if a billion humans does science for a million years, we won’t come up with the AI equivalent of Shakespeare’s plays. This would mean that we are doomed to evolve slowly, to tweak ourselves incrementally into a posthuman state. Yet, there are also other possibilities, that AGI might for example be a gradual development over many centuries. Human intelligence might turn out to be close to the maximum.
There is so much we do not know yet (http://bit.ly/ckeQo6). Take for example a constrained well-understood domain like Go. AI does still perform awfully at Go. Or take P vs. NP.:
P vs. NP is an absolutely enormous problem, and one way of seeing that is that there are already vastly, vastly easier questions that would be implied by P not equal to NP but that we already don’t know how to answer. So basically, if someone is claiming to prove P not equal to NP, then they’re sort of jumping 20 or 30 nontrivial steps beyond what we know today. [...] We have very strong reasons to believe that these problems cannot be solved without major — enormous — advances in human knowledge. [...] So in order to prove such a thing, a prerequisite to it is to understand the space of all possible efficient algorithms. That is an unbelievably tall order. So the expectation is that on the way to proving such a thing, we’re going to learn an enormous amount about efficient algorithms, beyond what we already know, and very, very likely discover new algorithms that will likely have applications that we can’t even foresee right now. (http://web.mit.edu/newsoffice/2010/3q-pnp.html).
But that is just my highly uneducated guess which I never seriously contemplated. I believe that for most academics the problem here is mainly about the missing proof of concept. Missing evidence. They are not the kind of people who wait before testing the first nuke because it might ignite the atmosphere. If there’s no good evidence, a position supported by years worth of disjunctive lines of reasoning won’t convince them either.
The paperclip maximizer (http://wiki.lesswrong.com/wiki/Paperclip_maximizer) scenario needs serious consideration. But given what needs to be done, what insights may be necessary to create something creative that is effective in the real world, it’s hard to believe that this is a serious risk. It’s similar with the kind of grey goo scenario that nanotechnology might hold. It will likely be a gradual development that once it becomes sophisticated enough to pose a serious risk is also understood and controlled by countermeasures.
I also wonder why we don’t see any alien paperclip maximizer’s out there? If there are any in the observable universe our FAI will lose anyway since it is far behind in its development.
It will likely be a gradual development that once it becomes sophisticated enough to pose a serious risk is also understood and controlled by countermeasures.
Indeed. Companies illustrate this. They are huge, superhuman powerful entities too.
A major upvote for this. The SIAI should create a sister organization to publicize the logical (and exceptionally) dangerous conclusion to the course that corporations are currently on. We have created powerful, superhuman entities with the sole top-level goal (required by LAW in for-profit corporations) of “Optimize money acquisition and retention”. My personal and professional opinion is that this is a far more immediate (and greater) risk than UnFriendly AI).
Companies are probably the number 1 bet for the type of organisation most likely to produce machine intelligence—with number 2 being governments. So, there’s a good chance that early machine intelligences will be embedded into the infrastructure of companies. So, these issues are probably linked.
Money is the nearest global equivalent of “utility”. Law-abiding maximisation of it does not seem unreasonable. There are some problems where it is difficult to measure and price things, though.
Money is the nearest global equivalent of “utility”. Law-abiding maximisation of it does not seem unreasonable.
On the other hand, maximization of money, including accurate terms for expected financial costs of legal penalties, can cause remarkable unreasonable behavior. As was repeated recently “It’s hard for the idea of an agent with different terminal values to really sink in”, in particular “something that could result in powerful minds that actually don’t care about morality”. A business that actually behaved as a pure profit maximizer would be such an entity.
Morality is represented by legal constraints. That results in a “negative” morality, and - arguably -not a very good one.
Fortunately companies are also subject to many of the same forces that produce cooperation and niceness in the rest of biology—including reputations, reciprocal altruism and kin selection.
Algorithmic trading is indeed an example for the kind of risks posed by complication (unmanageable) systems but also shows that we evolve our security measures with each small-scale catastrophe. There is no example of some existential risk from true runaway technological development yet although many people believe there are such risks, e.g. nuclear weapons. Unstoppable recursive self-improvement is just a hypothesis that you shouldn’t take as a foundation for a whole lot of further inductions.
An all-out nuclear war between Russia and the United States would be the worst catastrophe in history, a tragedy so huge it is difficult to comprehend. Even so, it would be far from the end of human life on earth. The dangers from nuclear weapons have been distorted and exaggerated, for varied reasons. These exaggerations have become demoralizing myths, believed by millions of Americans.
Apparently I don’t understand what you mean by “serious risk”. (Before I pick this apart, by the way, I agree that we should try not to Godwin people—because I think it doesn’t work.)
I consider it likely that AGI will take a long time to develop. A rational species would likely figure out the flaw and take corrective steps by then. But look around you. Nearly all of us seem to agree, if you look at what we actually want according to our actions, that we should try to prevent an asteroid strike that might destroy humanity. As far as I can tell we haven’t started yet. No doubt you can think of other examples: the evidence says that if we put off FAI theory ‘until we need it’, we could easily put it off longer than that.
We’ll never be able to make up for it. But I wouldn’t worry about it anyway, if there was just one hard take-off in the visible universe we should be able to detect it (soon) as it would have transformed entire galaxies and super-clusters if you believe current ideas (even without a hard take-off I guess). But there seems to be nothing, so either technological life is rare (once in the visible universe) or there are other more important risks than paperclipping.
Given that Burning the Cosmic Commons is already portrayed as a risk from uFAI that are paperclipping I don’t think it is unreasonable to ask why we don’t see any effects of such outcomes in the visible universe if the risk of such events is >70% for technological civilisations. I’m just looking for empirical evidence here that might support the conclusions. Of course, it might take one or two decades before we are capable of detecting such anomalies. Still, given the age of the universe you’d expect to see some artifacts if your premise is that technological civilizations haven’t evolved only once in the visible universe. If there are other reasons, then as I said before there might be other risks more important than uFAI to explain the fermi-paradox, which would be an importing observation too.
I don’t think you could see any of those things very easily in other galaxies if they were there. What you can see in other galaxies is chemical compounds—through mass spectroscopy. That shows some of them are rich in “life-like” stuff. Also, there is no shortage of darkish matter out there.
Thanks for the original pointer goes to Kevin.
Key points, some of which I already mentioned in the post Should I believe what the SIAI claims?:
[...]
My comment from the discussion post:
Should I believe what the SIAI claims? I’m still not sure, although I learnt some things since that post. But what I know is how serious the people here take this stuff. Also read the comments on this post for how people associated with LW overreact to completely harmless AI research.
The issue with potential risks posed by unfriendly AI are numerous. The only organisation that takes those issues serious is the SIAI, as its name already implies. But I believe most people simply don’t see a difference between the SIAI and one or a few highly intelligent people telling them that a particle collider could destroy the world while all experts working directly on it claim there’s no risk. Now I think I understand the argument that if the whole world is at stake it does outweigh the low probability of the event. But does it? I think it is completely justified to have at least one organisation working on FAI, but is the risk as serious as portrayed and perceived within the SIAI? Right now if I had to hazard a guess I’d say that it will probably be a gradual development of many exponential growth phases. That is, we’ll have this conceptual revolution and optimize it very rapidly. Then the next revolution will be necessary. Sure, I might be wrong there, as the plateau argument of self-improvement recursion might hold. But even if that is true, I think we’ll need at least two paradigm-shattering conceptual revolutions before we get there. But what does that mean though? How quickly can such revolutions happen? I’m guessing that this could take a long time, if it isn’t completely impossible. That is, if we are not the equivalent of Universal Turing Machine of abstract reasoning. Just imagine we are merely better chimps. Maybe it doesn’t matter if a billion humans does science for a million years, we won’t come up with the AI equivalent of Shakespeare’s plays. This would mean that we are doomed to evolve slowly, to tweak ourselves incrementally into a posthuman state. Yet, there are also other possibilities, that AGI might for example be a gradual development over many centuries. Human intelligence might turn out to be close to the maximum.
There is so much we do not know yet (http://bit.ly/ckeQo6). Take for example a constrained well-understood domain like Go. AI does still perform awfully at Go. Or take P vs. NP.:
But that is just my highly uneducated guess which I never seriously contemplated. I believe that for most academics the problem here is mainly about the missing proof of concept. Missing evidence. They are not the kind of people who wait before testing the first nuke because it might ignite the atmosphere. If there’s no good evidence, a position supported by years worth of disjunctive lines of reasoning won’t convince them either.
The paperclip maximizer (http://wiki.lesswrong.com/wiki/Paperclip_maximizer) scenario needs serious consideration. But given what needs to be done, what insights may be necessary to create something creative that is effective in the real world, it’s hard to believe that this is a serious risk. It’s similar with the kind of grey goo scenario that nanotechnology might hold. It will likely be a gradual development that once it becomes sophisticated enough to pose a serious risk is also understood and controlled by countermeasures.
I also wonder why we don’t see any alien paperclip maximizer’s out there? If there are any in the observable universe our FAI will lose anyway since it is far behind in its development.
I suppose the actual risk could be taking mere idea too serious.
Indeed. Companies illustrate this. They are huge, superhuman powerful entities too.
A major upvote for this. The SIAI should create a sister organization to publicize the logical (and exceptionally) dangerous conclusion to the course that corporations are currently on. We have created powerful, superhuman entities with the sole top-level goal (required by LAW in for-profit corporations) of “Optimize money acquisition and retention”. My personal and professional opinion is that this is a far more immediate (and greater) risk than UnFriendly AI).
Companies are probably the number 1 bet for the type of organisation most likely to produce machine intelligence—with number 2 being governments. So, there’s a good chance that early machine intelligences will be embedded into the infrastructure of companies. So, these issues are probably linked.
Money is the nearest global equivalent of “utility”. Law-abiding maximisation of it does not seem unreasonable. There are some problems where it is difficult to measure and price things, though.
On the other hand, maximization of money, including accurate terms for expected financial costs of legal penalties, can cause remarkable unreasonable behavior. As was repeated recently “It’s hard for the idea of an agent with different terminal values to really sink in”, in particular “something that could result in powerful minds that actually don’t care about morality”. A business that actually behaved as a pure profit maximizer would be such an entity.
Morality is represented by legal constraints. That results in a “negative” morality, and - arguably -not a very good one.
Fortunately companies are also subject to many of the same forces that produce cooperation and niceness in the rest of biology—including reputations, reciprocal altruism and kin selection.
Algorithmic trading is indeed an example for the kind of risks posed by complication (unmanageable) systems but also shows that we evolve our security measures with each small-scale catastrophe. There is no example of some existential risk from true runaway technological development yet although many people believe there are such risks, e.g. nuclear weapons. Unstoppable recursive self-improvement is just a hypothesis that you shouldn’t take as a foundation for a whole lot of further inductions.
Dispelling Stupid Myths About Nuclear War
Apparently I don’t understand what you mean by “serious risk”. (Before I pick this apart, by the way, I agree that we should try not to Godwin people—because I think it doesn’t work.)
I consider it likely that AGI will take a long time to develop. A rational species would likely figure out the flaw and take corrective steps by then. But look around you. Nearly all of us seem to agree, if you look at what we actually want according to our actions, that we should try to prevent an asteroid strike that might destroy humanity. As far as I can tell we haven’t started yet. No doubt you can think of other examples: the evidence says that if we put off FAI theory ‘until we need it’, we could easily put it off longer than that.
If there’s only one, we could perhaps run away from it at near the speed of light—while developing technologically—and building our strength.
We’ll never be able to make up for it. But I wouldn’t worry about it anyway, if there was just one hard take-off in the visible universe we should be able to detect it (soon) as it would have transformed entire galaxies and super-clusters if you believe current ideas (even without a hard take-off I guess). But there seems to be nothing, so either technological life is rare (once in the visible universe) or there are other more important risks than paperclipping.
How would a hard-takeoff galaxy look different from ordinary galaxies?
Very dim? Or very unusually given paperclipping?
http://en.wikipedia.org/wiki/Stellar_engine
http://en.wikipedia.org/wiki/Matrioshka_brain
http://en.wikipedia.org/wiki/Astroengineering
http://en.wikipedia.org/wiki/Megascale_engineering
Given that Burning the Cosmic Commons is already portrayed as a risk from uFAI that are paperclipping I don’t think it is unreasonable to ask why we don’t see any effects of such outcomes in the visible universe if the risk of such events is >70% for technological civilisations. I’m just looking for empirical evidence here that might support the conclusions. Of course, it might take one or two decades before we are capable of detecting such anomalies. Still, given the age of the universe you’d expect to see some artifacts if your premise is that technological civilizations haven’t evolved only once in the visible universe. If there are other reasons, then as I said before there might be other risks more important than uFAI to explain the fermi-paradox, which would be an importing observation too.
I don’t think you could see any of those things very easily in other galaxies if they were there. What you can see in other galaxies is chemical compounds—through mass spectroscopy. That shows some of them are rich in “life-like” stuff. Also, there is no shortage of darkish matter out there.
The Carina Nebula and the Orion Nebula may well be full of advanced life. We don’t really know what that looks like yet, though.