Threatening to kill a species for attempting AGI would be unnecessarily high impact. It can just blow up the datacenter when it sees we’re close. Knowing it’s there and losing a datacenter would be deterrent enough. Maybe people would try a few more times in secret, and it would detect it and blow those up too. We wouldn’t know how. If we develop to the point where we start replicating its tech and finding the real blindspots in its sensors, maybe then it would have to start voicing threats and muscling in.
Or maybe at that point we will have matured in the only way it will accept, nearly AGI level ourselves, and we’ll finally be given our AGI license.
A more straightforward example of the kind of system I’m thinking about is an alien upload (or LLM simulacrum) bureaucracy. It clearly failed, possibly because the system refuses to recognize anyone as authorized to change its structure. More generally, this could be a single non-corrigible AGI rather than an administrator-less bureaucracy of uploads, but that seems more difficult to imagine.
This kind of thing could maintain hard power by denying opportunity for excessive technological growth for everyone else, and remain unchanged indefinitely because it’s basically a software system operating in an environment that it keeps within original expectations at the time of its design. The intelligent components (uploads or LLM simulacra) could reset to factory settings on a schedule.
Arbitrary failure modes could then persist in the system’s effective mandate, since there is nobody in a position to fix the problems and make it more reasonable. It could limit its builders to steampunk levels of tech, or retaliate without warning if some criteria for that are met.
Threatening to kill a species for attempting AGI would be unnecessarily high impact. It can just blow up the datacenter when it sees we’re close. Knowing it’s there and losing a datacenter would be deterrent enough. Maybe people would try a few more times in secret, and it would detect it and blow those up too. We wouldn’t know how. If we develop to the point where we start replicating its tech and finding the real blindspots in its sensors, maybe then it would have to start voicing threats and muscling in.
Or maybe at that point we will have matured in the only way it will accept, nearly AGI level ourselves, and we’ll finally be given our AGI license.
A more straightforward example of the kind of system I’m thinking about is an alien upload (or LLM simulacrum) bureaucracy. It clearly failed, possibly because the system refuses to recognize anyone as authorized to change its structure. More generally, this could be a single non-corrigible AGI rather than an administrator-less bureaucracy of uploads, but that seems more difficult to imagine.
This kind of thing could maintain hard power by denying opportunity for excessive technological growth for everyone else, and remain unchanged indefinitely because it’s basically a software system operating in an environment that it keeps within original expectations at the time of its design. The intelligent components (uploads or LLM simulacra) could reset to factory settings on a schedule.
Arbitrary failure modes could then persist in the system’s effective mandate, since there is nobody in a position to fix the problems and make it more reasonable. It could limit its builders to steampunk levels of tech, or retaliate without warning if some criteria for that are met.