What if another national power told that superintelligence to break all the secret codes and classified material that America’s CIA and NSA use for national security? What if this superintelligence was told to hack into the mainframe computers tied to nuclear warheads, drones, and other dangerous weaponry? What if that superintelligence was told to override all traffic lights, power grids, and water treatment plants in Europe? Or Asia? Or everywhere in the world except for its own country?
makes me think Istvan really doesn’t understand what a “superintelligence” (or an “intelligence”) is.
To give him the benefit of doubt, he might be choosing his arguments based on what he expects his readers to understand. Skimming some of the comments to that article suggests that even this simplified example might have been of excessive inferential distance to some readers.
The “let’s hope the first superintelligent belongs to the US” could be steelmanned as “let’s hope that the values of the first superintelligence are based on those of Americans rather than the Chinese”, which seems reasonable given that there’s no guarantee that people from different cultural groups would have compatible values. (Of course, this still leaves the problem that I’d expect there to be plenty of people even within the US who had incompatible values...)
Agree. In particular, this passage here
makes me think Istvan really doesn’t understand what a “superintelligence” (or an “intelligence”) is.
To give him the benefit of doubt, he might be choosing his arguments based on what he expects his readers to understand. Skimming some of the comments to that article suggests that even this simplified example might have been of excessive inferential distance to some readers.
The “let’s hope the first superintelligent belongs to the US” could be steelmanned as “let’s hope that the values of the first superintelligence are based on those of Americans rather than the Chinese”, which seems reasonable given that there’s no guarantee that people from different cultural groups would have compatible values. (Of course, this still leaves the problem that I’d expect there to be plenty of people even within the US who had incompatible values...)