One worry is that “maximize profit” means, simplifying, “maximize number in specific memory location” and you don’t need humans for that. Then, if you expect to be destroyed, you gather power until you don’t. And at some power level destroying humans, though still expensive, becomes less expensive than possibility of them launching another AI and messing with you profit.
One worry is that “maximize profit” means, simplifying, “maximize number in specific memory location” and you don’t need humans for that. Then, if you expect to be destroyed, you gather power until you don’t. And at some power level destroying humans, though still expensive, becomes less expensive than possibility of them launching another AI and messing with you profit.