(This was about 14:30 of writing time. I think it would probably fit into a 15-minute chunk of interview time. I deliberately avoided deleting or changing much as I went. I don’t think anything in it is in any way original.)
So, first of all, why believe in existential risk? We know that sometimes species, empires, and the like come to an end. The human race is ingenious and adaptable, but there’s no reason to think we’re magically immune to every catastrophe that could wipe us out. (Which wouldn’t necessarily mean killing every single human being; I would consider something an existential risk if it wiped out our civilization in a way that made it difficult to start over successfully.)
Since humanity has been around for a long time, it’s reasonable to suppose that existential risks are rare. So why care about them? Because the possible impact is very large. For instance, suppose you believe the following things (I am not claiming they’re right, only that a reasonable person could believe them): 1. We are probably alone in the universe, or at least in this galaxy. (Because otherwise we’d expect that some other intelligent species would have expanded to take up a non-negligible fraction of the galaxy, and we would expect to see signs of that.) 2. If all goes well for us, there’s a reasonable chance that we will travel to, and colonize, other star systems, and eventually maybe most of this galaxy. If these things are true then it is possible that the number of future humans will be vastly greater than the present population, and that the future of the human race is the future of sentient life in our galaxy. In that case, if something wipes us out early on, the potential loss is staggeringly great.
One might also simply care a lot about the fate of the human race, and feel that its extinction would be a terrible thing whether or not the alternative involves billions of billions of lives.
Now, what about AI? The following is highly speculative, and I don’t think anyone should be very confident about it. But: first of all, if we are able to make human-level intelligences then it is likely not very difficult to make substantially-smarter-than-human intelligences. After all, humans have varied a lot in intellectual capabilities, and I know of no reason to think that although dogs and ants are much less smart than humans nothing can be much more smart than humans. And at least some forms of smarterness seem easy; e.g., it’s often possible to take a computer program and run it much faster, by means of more expensive hardware or more attention to optimizing the software. So, if we make human-level AI then soon afterwards we will have better-than-human-level AI. Now the smarter-than-human AI can work on making better AI, and will do it better and faster than we do, and naive mathematical models of the resulting progress suggest that after a short while progress will be infinitely fast. Obviously that is unlikely to happen, but it does suggest that we shouldn’t be too surprised if soon after the first human-level AI there are artificial intelligences that are as much cleverer than Einstein as Einstein was cleverer than a dog.
And, as the history of humanity on earth shows, intelligence and power are closely related. A vastly superhuman intelligence might be astonishingly effective at persuading people of things. It might be astonishingly effective at predicting markets. It might be astonishingly effective at inventing new technology. One way or another, it would not be surprising if soon after the first human-level AI we had AI that vastly outstrips us in power.
So we had better, somehow, do a really good job of making sure that any AI we build, and any AI that builds, behaves in ways that don’t wipe us out. Unfortunately, that seems to be a hard problem.
(This was about 14:30 of writing time. I think it would probably fit into a 15-minute chunk of interview time. I deliberately avoided deleting or changing much as I went. I don’t think anything in it is in any way original.)
So, first of all, why believe in existential risk? We know that sometimes species, empires, and the like come to an end. The human race is ingenious and adaptable, but there’s no reason to think we’re magically immune to every catastrophe that could wipe us out. (Which wouldn’t necessarily mean killing every single human being; I would consider something an existential risk if it wiped out our civilization in a way that made it difficult to start over successfully.)
Since humanity has been around for a long time, it’s reasonable to suppose that existential risks are rare. So why care about them? Because the possible impact is very large. For instance, suppose you believe the following things (I am not claiming they’re right, only that a reasonable person could believe them): 1. We are probably alone in the universe, or at least in this galaxy. (Because otherwise we’d expect that some other intelligent species would have expanded to take up a non-negligible fraction of the galaxy, and we would expect to see signs of that.) 2. If all goes well for us, there’s a reasonable chance that we will travel to, and colonize, other star systems, and eventually maybe most of this galaxy. If these things are true then it is possible that the number of future humans will be vastly greater than the present population, and that the future of the human race is the future of sentient life in our galaxy. In that case, if something wipes us out early on, the potential loss is staggeringly great.
One might also simply care a lot about the fate of the human race, and feel that its extinction would be a terrible thing whether or not the alternative involves billions of billions of lives.
Now, what about AI? The following is highly speculative, and I don’t think anyone should be very confident about it. But: first of all, if we are able to make human-level intelligences then it is likely not very difficult to make substantially-smarter-than-human intelligences. After all, humans have varied a lot in intellectual capabilities, and I know of no reason to think that although dogs and ants are much less smart than humans nothing can be much more smart than humans. And at least some forms of smarterness seem easy; e.g., it’s often possible to take a computer program and run it much faster, by means of more expensive hardware or more attention to optimizing the software. So, if we make human-level AI then soon afterwards we will have better-than-human-level AI. Now the smarter-than-human AI can work on making better AI, and will do it better and faster than we do, and naive mathematical models of the resulting progress suggest that after a short while progress will be infinitely fast. Obviously that is unlikely to happen, but it does suggest that we shouldn’t be too surprised if soon after the first human-level AI there are artificial intelligences that are as much cleverer than Einstein as Einstein was cleverer than a dog.
And, as the history of humanity on earth shows, intelligence and power are closely related. A vastly superhuman intelligence might be astonishingly effective at persuading people of things. It might be astonishingly effective at predicting markets. It might be astonishingly effective at inventing new technology. One way or another, it would not be surprising if soon after the first human-level AI we had AI that vastly outstrips us in power.
So we had better, somehow, do a really good job of making sure that any AI we build, and any AI that builds, behaves in ways that don’t wipe us out. Unfortunately, that seems to be a hard problem.