An interesting parallel might be a parallel Earth making nanotechnology breakthroughs instead of AI breakthroughs, such that it’s apparent they’ll be capable of creating gray goo and not apparent they’ll be able to avoid creating gray goo.
I guess a slow takeoff could be if, like, the first self-replicators took a day to double, so if somebody accidentally made a gram of gray goo you’d have weeks to figure it out and nuke the lab or whatever, but self-replication speed went down as technology improved, and so accidental unconstrained replicators happened periodically but could be contained until one couldn’t be.
Whereas hard takeoff could be if you had nanobots that built stuff in seconds but couldn’t self-replicate using random environmental mass, and then the first nanobot that can do that, can do it in seconds and eats the planet.
Should we consider the second scenario less likely because of smooth trend lines? Does Paul think we should? (I’m pretty sure Eliezer thinks that Paul thinks we should)
An interesting parallel might be a parallel Earth making nanotechnology breakthroughs instead of AI breakthroughs, such that it’s apparent they’ll be capable of creating gray goo and not apparent they’ll be able to avoid creating gray goo.
I guess a slow takeoff could be if, like, the first self-replicators took a day to double, so if somebody accidentally made a gram of gray goo you’d have weeks to figure it out and nuke the lab or whatever, but self-replication speed went down as technology improved, and so accidental unconstrained replicators happened periodically but could be contained until one couldn’t be.
Whereas hard takeoff could be if you had nanobots that built stuff in seconds but couldn’t self-replicate using random environmental mass, and then the first nanobot that can do that, can do it in seconds and eats the planet.
Should we consider the second scenario less likely because of smooth trend lines? Does Paul think we should? (I’m pretty sure Eliezer thinks that Paul thinks we should)