True. My point was that if it was easy for an organization to become much more powerful than it is now, and the organization was motivated to do so, then it would already be much more powerful than it is now, so we should not expect a sudden increase in organizations’ self-improvement abilities unless we can identify a good reason that it is particularly likely. The increased ease of self-modification offered by being completely digital is such a reason, but since organizations are not completely digital, this does not offer a way for organizations to suddenly increase their rate of self-improvement unless we can upload an organization.
We don’t expect a sudden increase in organizations’ self-improvement abilities.We don’t expect a sudden increase in the self-improvement abilities of machines either. The bottom line is that evolution happens gradually. Going digital isn’t a reason to expect a sudden increase in self-improvement abilities. We know that since the digital revolution has been going on for decades now, and the resulting rate of improvement is clearly gradual. It is gradual because digitization affects one system at a time, and there are many systems involved, each of which is instantiated many times—and their replacement takes time. So, for example, the human memory system has already been superseded in practically every way by machine memories. The human retina has already been superseded in practically every way by digital cameras. Humans won’t suddenly be replaced by machines. They will coevolve for an extended period—indeed they have already been doing that for thousands of years now.
We don’t expect a sudden increase in the self-improvement abilities of machines either.
Maybe you don’t expect that, but surely you must be aware that many of us do.
Anyway, nothing seems particularly close to powerful enough to be catastrophically dangerous at the moment except for nuclear-armed nations, which have been fairly stable in their power and, with the exception of North Korea, which isn’t powerful enough, the rest of the nuclear powers are not much of a threat because they would prefer not to cause massive destruction. Every organization that’s not a country is far enough away from that level of power that I don’t expect them to become catastrophically dangerous any time soon without a sudden increase in self-improvement.
I think that at some point in the development of Artificial Intelligence, we are likely to see a fast, local increase in capability—“AI go FOOM”.
We are witness to Moore’s law. A straightforwards extrapolation of that says that at some point things will be changing rapidly. I don’t have an argument with that. What I would object to are saltations. Those are suggested by the term “suddenly”—but are contrary to evolutionary theory.
Probably, things will be progressing fastest well after the human era is over. It’s a remote era which we can really only speculate about. We have far more immediate issues to worry about that what is likely to happen then.
Every organization that’s not a country is far enough away from that level of power that I don’t expect them to become catastrophically dangerous any time soon without a sudden increase in self-improvement.
So: giant oaks from tiny acorns grow—and it is easiest to influence creatures when they are young.
True. My point was that if it was easy for an organization to become much more powerful than it is now, and the organization was motivated to do so, then it would already be much more powerful than it is now, so we should not expect a sudden increase in organizations’ self-improvement abilities unless we can identify a good reason that it is particularly likely. The increased ease of self-modification offered by being completely digital is such a reason, but since organizations are not completely digital, this does not offer a way for organizations to suddenly increase their rate of self-improvement unless we can upload an organization.
We don’t expect a sudden increase in organizations’ self-improvement abilities.We don’t expect a sudden increase in the self-improvement abilities of machines either. The bottom line is that evolution happens gradually. Going digital isn’t a reason to expect a sudden increase in self-improvement abilities. We know that since the digital revolution has been going on for decades now, and the resulting rate of improvement is clearly gradual. It is gradual because digitization affects one system at a time, and there are many systems involved, each of which is instantiated many times—and their replacement takes time. So, for example, the human memory system has already been superseded in practically every way by machine memories. The human retina has already been superseded in practically every way by digital cameras. Humans won’t suddenly be replaced by machines. They will coevolve for an extended period—indeed they have already been doing that for thousands of years now.
Maybe you don’t expect that, but surely you must be aware that many of us do.
Anyway, nothing seems particularly close to powerful enough to be catastrophically dangerous at the moment except for nuclear-armed nations, which have been fairly stable in their power and, with the exception of North Korea, which isn’t powerful enough, the rest of the nuclear powers are not much of a threat because they would prefer not to cause massive destruction. Every organization that’s not a country is far enough away from that level of power that I don’t expect them to become catastrophically dangerous any time soon without a sudden increase in self-improvement.
I am aware that there’s an argument that at some point things will be changing rapidly:
We are witness to Moore’s law. A straightforwards extrapolation of that says that at some point things will be changing rapidly. I don’t have an argument with that. What I would object to are saltations. Those are suggested by the term “suddenly”—but are contrary to evolutionary theory.
Probably, things will be progressing fastest well after the human era is over. It’s a remote era which we can really only speculate about. We have far more immediate issues to worry about that what is likely to happen then.
So: giant oaks from tiny acorns grow—and it is easiest to influence creatures when they are young.