my understanding is that crypto is secured not by trust, guns, or rules, but by fundamental computational limits
While there are hard physical limits on computation (or at least there seem to be, based on our current knowledge of physics), cryptographic systems are not generally based on those limits, and are not known to be difficult to break. It’s just that we haven’t discovered an easy way to break them yet—except for all the cryptosystems where we have discovered a way, and so we don’t use those systems anymore. This should not inspire too much confidence in the currently used systems, especially against a superhuman adversary.
the ability of any one actor (including AI) to gain arbitrary power without the consent of everyone else would be limited
As long as the AI has something of value to offer, people will have an incentive to trade with it. Even if the increments are small, it could gain control of lots of resources over time. By analogy, it’s not hard to find people who disapprove of how Jeff Bezos spends his money, but who still shop on Amazon.
While there are hard physical limits on computation (or at least there seem to be, based on our current knowledge of physics), cryptographic systems are not generally based on those limits, and are not known to be difficult to break. It’s just that we haven’t discovered an easy way to break them yet—except for all the cryptosystems where we have discovered a way, and so we don’t use those systems anymore. This should not inspire too much confidence in the currently used systems, especially against a superhuman adversary.
As long as the AI has something of value to offer, people will have an incentive to trade with it. Even if the increments are small, it could gain control of lots of resources over time. By analogy, it’s not hard to find people who disapprove of how Jeff Bezos spends his money, but who still shop on Amazon.