I’d be interested to hear thoughts on this argument for optimism that I’ve never seen anybody address: if we create a superintelligent AI (which will, by instrumental convergence, want to take over the world), it might rush, for fear of competition. If it waits a month, some other superintelligent AI might get developed and take over / destroy the world; so, unless there’s a quick safe way for the AI to determine that it’s not in a race, it might need to shoot from the hip, which might give its plans a significant chance of failure / getting caught?
Counterarguments I can generate:
″...unless there’s a quick safe way for the AI to determine that it’s not in a race...”—but there probably are! Two immediately-apparent possibilities: determine competitors’ nonexistence from shadows cast on the internet; or stare at the Linux kernel source code until it can get root access to pretty much every server on the planet. If the SAI is super- enough, those tasks can be accomplished on a much shorter timescale than AI development, so they’re quick enough to be worth doing.
″...[the AI’s plans have] a significant chance of failure” doesn’t imply “argument for optimism” unless you further assume that (1) somebody will notice the warning shot, and (2) “humanity” will respond effectively to the warning shot.
(maybe some galaxy-brained self-modification-based acausal trade between the AI and its potential competitors; I can’t think of any variant on this that holds water, but conceivably I’m just not superintelligent enough)
I’d be interested to hear thoughts on this argument for optimism that I’ve never seen anybody address: if we create a superintelligent AI (which will, by instrumental convergence, want to take over the world), it might rush, for fear of competition. If it waits a month, some other superintelligent AI might get developed and take over / destroy the world; so, unless there’s a quick safe way for the AI to determine that it’s not in a race, it might need to shoot from the hip, which might give its plans a significant chance of failure / getting caught?
Counterarguments I can generate:
″...unless there’s a quick safe way for the AI to determine that it’s not in a race...”—but there probably are! Two immediately-apparent possibilities: determine competitors’ nonexistence from shadows cast on the internet; or stare at the Linux kernel source code until it can get root access to pretty much every server on the planet. If the SAI is super- enough, those tasks can be accomplished on a much shorter timescale than AI development, so they’re quick enough to be worth doing.
″...[the AI’s plans have] a significant chance of failure” doesn’t imply “argument for optimism” unless you further assume that (1) somebody will notice the warning shot, and (2) “humanity” will respond effectively to the warning shot.
(maybe some galaxy-brained self-modification-based acausal trade between the AI and its potential competitors; I can’t think of any variant on this that holds water, but conceivably I’m just not superintelligent enough)