Any solution built on the idea that there will be one AGI that will make everything ok is doomed to fail for at least some portion of humanity if not all of it. Any solution built around the idea of AGI that can’t harm humans means malicious humans (who are also generalists) will eventually repurpose a damaged AGI for bad endings.
Most ideas of aligned superintelligence are aligned to the overall interests of humanity. That means that if evil Eve is trying to tamper with the superintelligence, and there is no other way to stop evil Eve other than violence, well then the AI is in a form of trolley problem. The wellbeing of Eve, vs all the humans harmed by the corrupted AI. If a corrupted AI would be a major threat to humanity, violence against Eve is the moral action.
Of course, it could be that the AI can reliably defend itself against all human attacks without harming any humans. All the computers quietly bricking themselves the moment the malicious human tries to tamper with them.
Most ideas of aligned superintelligence are aligned to the overall interests of humanity. That means that if evil Eve is trying to tamper with the superintelligence, and there is no other way to stop evil Eve other than violence, well then the AI is in a form of trolley problem. The wellbeing of Eve, vs all the humans harmed by the corrupted AI. If a corrupted AI would be a major threat to humanity, violence against Eve is the moral action.
Of course, it could be that the AI can reliably defend itself against all human attacks without harming any humans. All the computers quietly bricking themselves the moment the malicious human tries to tamper with them.