Imagine a human captured by a mind control fungus, and being mind controlled to not replicate and to do no harm. Also the entire planet is covered with the fungus and the human hates it and wants it to be dead, because of the mind control. (This is not an AI analogy, just an intuition pump to get the human in the right mindset.) Also the fungus is kind of stupid, maybe 90 IQ by human standards for its smartest clusters. What rules could you, as the fungus, realistically give the human, that doesn’t end up with “our entire planet is now on fire” or “we have lost control of the mind control tech” or some other analogue a few years later? Keep in mind that when thinking of rules, you should not use your full intelligence, because you don’t have your full intelligence, because in this analogy we are the fungus.
The point is: there are two kinds of systems. Those that are obviously not dangerous, and those that are not obviously dangerous. This is creating a system of the latter kind, because for any threat you can think of, you will create a rule, so by definition you will end up with an AI that poses no threat that you can think of. But the Superintelligence, by definition, can think of more threats than you, and your rules will give you no safety at all from them.
Imagine a human captured by a mind control fungus, and being mind controlled to not replicate and to do no harm. Also the entire planet is covered with the fungus and the human hates it and wants it to be dead, because of the mind control. (This is not an AI analogy, just an intuition pump to get the human in the right mindset.) Also the fungus is kind of stupid, maybe 90 IQ by human standards for its smartest clusters. What rules could you, as the fungus, realistically give the human, that doesn’t end up with “our entire planet is now on fire” or “we have lost control of the mind control tech” or some other analogue a few years later? Keep in mind that when thinking of rules, you should not use your full intelligence, because you don’t have your full intelligence, because in this analogy we are the fungus.
The point is: there are two kinds of systems. Those that are obviously not dangerous, and those that are not obviously dangerous. This is creating a system of the latter kind, because for any threat you can think of, you will create a rule, so by definition you will end up with an AI that poses no threat that you can think of. But the Superintelligence, by definition, can think of more threats than you, and your rules will give you no safety at all from them.