I think the best justification is by analogy. Humans do not physically have a decisive strategic advantage over other large animals—chimps, lions, elephants, etc. And for hundreds of thousands of years, we were not at the top of the food chain, despite our intelligence. However, intelligence eventually won out, and allowed us to conquer the planet.
Moreover, the benefit of intelligence increased exponentially in proportion to the exponential advance of technology. There was a long, slow burn, followed by what (on evolutionary timescales) was an extremely “fast takeoff”: a very rapid improvement in technology (and thus power) over only a few hundred years. Technological progress is now so rapid that human minds have trouble keeping up within a single lifetime, and genetic evolution has been left in the dust.
That’s the world into which AGI will enter—a technological world in which a difference in intellectual ability can be easily translated into a difference in technological ability, and thus power. Any future technologies that the laws of physics don’t explicitly prohibit, we must assume that an AGI will master faster than we can.
I think the best justification is by analogy. Humans do not physically have a decisive strategic advantage over other large animals—chimps, lions, elephants, etc. And for hundreds of thousands of years, we were not at the top of the food chain, despite our intelligence. However, intelligence eventually won out, and allowed us to conquer the planet.
Moreover, the benefit of intelligence increased exponentially in proportion to the exponential advance of technology. There was a long, slow burn, followed by what (on evolutionary timescales) was an extremely “fast takeoff”: a very rapid improvement in technology (and thus power) over only a few hundred years. Technological progress is now so rapid that human minds have trouble keeping up within a single lifetime, and genetic evolution has been left in the dust.
That’s the world into which AGI will enter—a technological world in which a difference in intellectual ability can be easily translated into a difference in technological ability, and thus power. Any future technologies that the laws of physics don’t explicitly prohibit, we must assume that an AGI will master faster than we can.