This argument falls apart on the last one. A superintelligence that wants to kill you can’t if it’s vastly out resourced and out numbered by superintelligences, some in boxes, that don’t want to kill humans.
You snuck in a questionable assumption, that a “free” superintelligence able to decide to kill you will be far more capable than boxed and restricted superintelligences who may have access to use weapons when authorized.
If the boxed superintelligence with the ability to plan usage of weapons when authorized by humans, and other boxed superintelligences able to control robotics in manufacturing cells are on humans side, the advantage for humans could be overwhelming. No matter how smart an ASI is it’s tough to win if the humans are prepared with millions of drones and nukes and space suits and bio and nano weapon sensors and weapons satellites and..
It’s an assumption EY has made many times, I am just calling it out.
“If the boxed superintelligence with the ability to plan usage of weapons when authorized by humans, and other boxed superintelligences able to control robotics in manufacturing cells are on humans side, the advantage for humans could be overwhelming”
As I said, I do not expect boxed AIs to be a thing most will do. We haven’t seen it, and I don’t expect to see it, because unboxed AIs are superior. This isn’t how people in control are approaching the situation, and I don’t expect that to change.
My definition of “box” may be very different from yours. In my definition, locked weights and training only on testing, as well as other design elements such as distribution detection, heavily box the model’a capabilities and behavior.
It is fine if the model can access the internet, robotics, etc so long as it lacks the context information to know it’s on the real thing vs a sim or cached copy.
This argument falls apart on the last one. A superintelligence that wants to kill you can’t if it’s vastly out resourced and out numbered by superintelligences, some in boxes, that don’t want to kill humans.
You snuck in a questionable assumption, that a “free” superintelligence able to decide to kill you will be far more capable than boxed and restricted superintelligences who may have access to use weapons when authorized.
If the boxed superintelligence with the ability to plan usage of weapons when authorized by humans, and other boxed superintelligences able to control robotics in manufacturing cells are on humans side, the advantage for humans could be overwhelming. No matter how smart an ASI is it’s tough to win if the humans are prepared with millions of drones and nukes and space suits and bio and nano weapon sensors and weapons satellites and..
It’s an assumption EY has made many times, I am just calling it out.
“If the boxed superintelligence with the ability to plan usage of weapons when authorized by humans, and other boxed superintelligences able to control robotics in manufacturing cells are on humans side, the advantage for humans could be overwhelming”
As I said, I do not expect boxed AIs to be a thing most will do. We haven’t seen it, and I don’t expect to see it, because unboxed AIs are superior. This isn’t how people in control are approaching the situation, and I don’t expect that to change.
My definition of “box” may be very different from yours. In my definition, locked weights and training only on testing, as well as other design elements such as distribution detection, heavily box the model’a capabilities and behavior.
See https://www.lesswrong.com/posts/a5NxvzFGddj2e8uXQ/updating-drexler-s-cais-model?commentId=AZA8ujssBJK9vQXAY
It is fine if the model can access the internet, robotics, etc so long as it lacks the context information to know it’s on the real thing vs a sim or cached copy.