It’s mostly a challenge to the analogy (with a willingness to be surprised by the creativity of suggestions to the contrary.) UFAI is scary not just because it can easily attain power but because the sort of power it could obtain would allow it to pursue its goals unilaterally; emperors, meanwhile, get their power from the consent of the governed (for values of “consent” that include compliance with coercion.) History is basically one unfriendly emperor (or social institution more broadly) after another, but this has posed existential risk only recently.
If I can build infrastructure faster than people can reproduce, which should be easy once technology advances far enough, I just have to eventually have enough automation that I can take everyone out. Or I can just ignore the humans until I’ve mined most of the solar system, and then take them out.
How does that matter? No matter what my goal is, I’d start by taking over the world.
It’s mostly a challenge to the analogy (with a willingness to be surprised by the creativity of suggestions to the contrary.) UFAI is scary not just because it can easily attain power but because the sort of power it could obtain would allow it to pursue its goals unilaterally; emperors, meanwhile, get their power from the consent of the governed (for values of “consent” that include compliance with coercion.) History is basically one unfriendly emperor (or social institution more broadly) after another, but this has posed existential risk only recently.
If I can build infrastructure faster than people can reproduce, which should be easy once technology advances far enough, I just have to eventually have enough automation that I can take everyone out. Or I can just ignore the humans until I’ve mined most of the solar system, and then take them out.