On further reflection, this business of “FAI-complete” is very puzzling. What we should make of it depends on what we mean by FAI:
If we define FAI broadly, then yes, the problem of getting AI to have a decent understanding of our intentions does seem to be FAI-complete
If we defined FAI as a utopia-machine, claims of FAI completeness look very dubious. I have a human’s values, but my understanding of my own values isn’t perfect. If I found myself in the position of the titular character in Bruce Almighty, I’d trust myself to try to make some very large improvements in the world, but I wouldn’t trust myself to try to create a utopia in one fell swoop. If my self-assessment is right, that means it’s possible to have a mind that can be trusted to attempt some good actions but not others, which looks like a problem for claims of FAI completeness.
Edit: Though in Bruce Almighty, he just wills things to happen and they happen. There are often unintended consequences, but never any need to worry about what means the genie will use to get the desired result. So it’s not a perfect analogy for trying to use super-AI.
Besides, even if an AI is Friendliness-complete and knows the “right thing” to be achieved, it doesn’t mean it can actually achieve it. Being superhumanly smart doesn’t mean being superhumanly powerful. We often make such an assumption because it’s the safe one in the Least Convenient World if the AI is not Friendly. But in the Least Convenient World, a proven-Friendly AI is at least as intelligent as a human, but no more powerful than an average big corp.
On further reflection, this business of “FAI-complete” is very puzzling. What we should make of it depends on what we mean by FAI:
If we define FAI broadly, then yes, the problem of getting AI to have a decent understanding of our intentions does seem to be FAI-complete
If we defined FAI as a utopia-machine, claims of FAI completeness look very dubious. I have a human’s values, but my understanding of my own values isn’t perfect. If I found myself in the position of the titular character in Bruce Almighty, I’d trust myself to try to make some very large improvements in the world, but I wouldn’t trust myself to try to create a utopia in one fell swoop. If my self-assessment is right, that means it’s possible to have a mind that can be trusted to attempt some good actions but not others, which looks like a problem for claims of FAI completeness.
Edit: Though in Bruce Almighty, he just wills things to happen and they happen. There are often unintended consequences, but never any need to worry about what means the genie will use to get the desired result. So it’s not a perfect analogy for trying to use super-AI.
Besides, even if an AI is Friendliness-complete and knows the “right thing” to be achieved, it doesn’t mean it can actually achieve it. Being superhumanly smart doesn’t mean being superhumanly powerful. We often make such an assumption because it’s the safe one in the Least Convenient World if the AI is not Friendly. But in the Least Convenient World, a proven-Friendly AI is at least as intelligent as a human, but no more powerful than an average big corp.