One point that Holden elided (so maybe he wouldn’t want me to argue this way) is that a population of human-level AIs is not going to stay human level for long.
Humans aren’t at some special gateway in the space of minds—we’re just the first ape species that got smart enough to discover writing. I’m not optimal, and I have a missing appendix to prove it. The point is, whatever process smartened these AIs up to the nebulous human level isn’t going to suddenly hit a brick wall, because there is no wall here.
But as I said, if Holden was writing this reply he’d probably try to argue without appeal to this fact. He’d probably say something about even if we just treat AIs as “like humans, but with different reproductive cycles and conditions for life,” having a couple million of them trying to kill all humans is still dire news. Maybe something about how even North Korea’s dictators still have some restraint born of self-preservation, but AIs might be happy to make the earth uninhabitable for human life because they have different conditions for survival.
One point that Holden elided (so maybe he wouldn’t want me to argue this way) is that a population of human-level AIs is not going to stay human level for long.
Humans aren’t at some special gateway in the space of minds—we’re just the first ape species that got smart enough to discover writing. I’m not optimal, and I have a missing appendix to prove it. The point is, whatever process smartened these AIs up to the nebulous human level isn’t going to suddenly hit a brick wall, because there is no wall here.
But as I said, if Holden was writing this reply he’d probably try to argue without appeal to this fact. He’d probably say something about even if we just treat AIs as “like humans, but with different reproductive cycles and conditions for life,” having a couple million of them trying to kill all humans is still dire news. Maybe something about how even North Korea’s dictators still have some restraint born of self-preservation, but AIs might be happy to make the earth uninhabitable for human life because they have different conditions for survival.