Since existing high-status people’s cooperation may be required to build a CEV-like AI
Yes, this is a concern I’ve had: a CEV-valuing FAI would also have to be a very good politician to have the social support necessary to keep the project from being shut down. But good politicians are a danger in and of themselves.
The other possibility is that someone builds CEV before the existing elites get to a high enough future-shock level to even pay attention. Dictators will wake up one morning to find that the FAI has just robbed them of the only thing that makes anyone care about them: their power over other people. And furthermore, there will be a very large number of people whose volitions say something like “I want to punish [Dictator X] because he killed my family”.
Yes, this is a concern I’ve had: a CEV-valuing FAI would also have to be a very good politician to have the social support necessary to keep the project from being shut down. But good politicians are a danger in and of themselves.
The other possibility is that someone builds CEV before the existing elites get to a high enough future-shock level to even pay attention. Dictators will wake up one morning to find that the FAI has just robbed them of the only thing that makes anyone care about them: their power over other people. And furthermore, there will be a very large number of people whose volitions say something like “I want to punish [Dictator X] because he killed my family”.
Once an AGI is out, you won’t be able to shut it down. The Internet makes it pretty much impossible, even if the AGI is not superintelligent.