Besides, even if an AI is Friendliness-complete and knows the “right thing” to be achieved, it doesn’t mean it can actually achieve it. Being superhumanly smart doesn’t mean being superhumanly powerful. We often make such an assumption because it’s the safe one in the Least Convenient World if the AI is not Friendly. But in the Least Convenient World, a proven-Friendly AI is at least as intelligent as a human, but no more powerful than an average big corp.
Besides, even if an AI is Friendliness-complete and knows the “right thing” to be achieved, it doesn’t mean it can actually achieve it. Being superhumanly smart doesn’t mean being superhumanly powerful. We often make such an assumption because it’s the safe one in the Least Convenient World if the AI is not Friendly. But in the Least Convenient World, a proven-Friendly AI is at least as intelligent as a human, but no more powerful than an average big corp.