What would it mean for an AGI to be aligned with “Democracy,” or “Confucianism,” or “Marxism with Chinese characteristics,” or “the American Constitution”? Contingent on a world where such an entity exists and is compatible with my existence, what would my life be like in a weird transhuman future as a non-citizen in each system?
None of these philosophies or ideologies was created with an interplanetary transhuman order in mind, so to some extent a superintelligent AI guided by them, will find itself “out of distribution” when deciding what to do. And how that turns out, should depend on underlying features of the AGI’s thought—how it reasons and how it deals with ontological crisis. We could in fact do some experiments along these lines—tell an existing frontier AI to suppose that it is guided by historic human systems like these, and ask how it might reinterpret the central concepts, in order to deal with being in a situation of relative omnipotence.
Supposing that the human culture of America and China is also a clue to the world that their AIs would build when unleashed, one could look to their science fiction for paradigms of life under cosmic circumstances. The West has lots of science fiction, but the one we keep returning to in the context of AI, is the Culture universe of Iain Banks. As for China, we know about Liu Cixin (“Three-Body Problem” series), and I also dwell on the xianxia novels of Er Gen, which are fantasy but do depict a kind of politics of omnipotence.
None of these philosophies or ideologies was created with an interplanetary transhuman order in mind, so to some extent a superintelligent AI guided by them, will find itself “out of distribution” when deciding what to do. And how that turns out, should depend on underlying features of the AGI’s thought—how it reasons and how it deals with ontological crisis. We could in fact do some experiments along these lines—tell an existing frontier AI to suppose that it is guided by historic human systems like these, and ask how it might reinterpret the central concepts, in order to deal with being in a situation of relative omnipotence.
Supposing that the human culture of America and China is also a clue to the world that their AIs would build when unleashed, one could look to their science fiction for paradigms of life under cosmic circumstances. The West has lots of science fiction, but the one we keep returning to in the context of AI, is the Culture universe of Iain Banks. As for China, we know about Liu Cixin (“Three-Body Problem” series), and I also dwell on the xianxia novels of Er Gen, which are fantasy but do depict a kind of politics of omnipotence.
The best of Liu Cixin’s novels about super AI is China 2185, which is also Liu Cixin’s unreleased debut novel