We tried to find the strongest argument in the literature. This is how we came up with our version:
“ Premise 1: Superintelligent AI is a realistic prospect, and it would be out of human control. (Singularity claim)
Premise 2: Any level of intelligence can go with any goals. (Orthogonality thesis)
Conclusion: Superintelligent AI poses an existential risk for humanity ”
==== A more formal version with the same propositions might be this:
1. IF there is a realistic prospect that there will be a superintelligent AI system that is a) out of human control and b) can have any goals, THEN there is existential risk for humanity from AI
2. There is a realistic prospect that there will be a superintelligent AI system that is a) out of human control and b) can have any goals
->
3. There is existential risk for humanity from AI
====
And now our concern is whether a superintelligence can be both a) and b) - given that a) must be understood in a way that is strong enough to generate existential risk, including “widening the frame”, and b) must be understood as strong enough to exclude reflection on goals. Perhaps that will work only if “intelligent” is understood in two different ways? Thus Premise 2 is doubtful.
We tried to find the strongest argument in the literature. This is how we came up with our version:
“
Premise 1: Superintelligent AI is a realistic prospect, and it would be out of human control. (Singularity claim)
Premise 2: Any level of intelligence can go with any goals. (Orthogonality thesis)
Conclusion: Superintelligent AI poses an existential risk for humanity
”
====
A more formal version with the same propositions might be this:
1. IF there is a realistic prospect that there will be a superintelligent AI system that is a) out of human control and b) can have any goals, THEN there is existential risk for humanity from AI
2. There is a realistic prospect that there will be a superintelligent AI system that is a) out of human control and b) can have any goals
->
3. There is existential risk for humanity from AI
====
And now our concern is whether a superintelligence can be both a) and b) - given that a) must be understood in a way that is strong enough to generate existential risk, including “widening the frame”, and b) must be understood as strong enough to exclude reflection on goals. Perhaps that will work only if “intelligent” is understood in two different ways? Thus Premise 2 is doubtful.