I believe that in real life, it is mostly the principal-agent problem, or coordination problems in general. Groups defeat individuals, but large groups often fall apart under their own weight. When a group becomes powerful, infighting becomes more profitable than fighting for the benefit of the group. An entirely ruthless army is an army that wouldn’t mind taking over its own country; and then its generals would optimize for getting in power over conquering foreign territories.
If we had robots, capable of thinking like a human, and needing a comparable amount of resources to live and reproduce… and in addition, able to make perfectly loyal copies of themselves… that would already be quite scary, like Scientology on steroids.
People waste a lot of resources competing with each other. Imagine how great it would be to have a group of people whom you could trust 100% to act in your best interest. Having friends is already a great thing, and friends are far from 100% reliable.
I believe that in real life, it is mostly the principal-agent problem, or coordination problems in general. Groups defeat individuals, but large groups often fall apart under their own weight. When a group becomes powerful, infighting becomes more profitable than fighting for the benefit of the group. An entirely ruthless army is an army that wouldn’t mind taking over its own country; and then its generals would optimize for getting in power over conquering foreign territories.
If we had robots, capable of thinking like a human, and needing a comparable amount of resources to live and reproduce… and in addition, able to make perfectly loyal copies of themselves… that would already be quite scary, like Scientology on steroids.
People waste a lot of resources competing with each other. Imagine how great it would be to have a group of people whom you could trust 100% to act in your best interest. Having friends is already a great thing, and friends are far from 100% reliable.