Great post. I even worry about the emphasis on FAI, as it seems to depend on friendly superintelligent AIs effectively defending us against deliberately criminal AIs. Scott Alexander speculated:
For example, it might program a virus that will infect every computer in the world, causing them to fill their empty memory with partial copies of the superintelligence, which when networked together become full copies of the superintelligence.
But way before that, we will have humans looking to get rich programming such a virus, and you better believe they won’t be using safeguards. It won’t take over every computer in the world—just the ones that aren’t defended by a more-powerful superintelligence (i.e. almost all computers) and that aren’t interacting with the internet using formally verified software. We’ll be attacked by a superintelligence running on billions of smart phones. Might be distributed initially through a compromised build of the hottest new social app for anonymous VR fucking.
Great post. I even worry about the emphasis on FAI, as it seems to depend on friendly superintelligent AIs effectively defending us against deliberately criminal AIs. Scott Alexander speculated:
But way before that, we will have humans looking to get rich programming such a virus, and you better believe they won’t be using safeguards. It won’t take over every computer in the world—just the ones that aren’t defended by a more-powerful superintelligence (i.e. almost all computers) and that aren’t interacting with the internet using formally verified software. We’ll be attacked by a superintelligence running on billions of smart phones. Might be distributed initially through a compromised build of the hottest new social app for anonymous VR fucking.