Huh. Seeing this answer twice, I can’t help but think that the standard strategy for any UFAI then is to first convince you that it is an FAI, and then to convince you that there is another UFAI “almost ready” somewhere.
Heck, if it can do #2, it might be able to skip #1 if it was able to argue that it was the less-dangerous of the two.
That’s probably why EY is so cautious about it and does not want any meaningful AGI research progress to happen until a “provably friendly AI” theory is developed. An admirable goal, though many remain skeptical of the odds of success of such an approach, or even the rationale behind it.
I think the theory is that the only thing that is powerful enough to contain UFAI is FAI, so the first self-improving AI had damn well better be FAI.
Huh. Seeing this answer twice, I can’t help but think that the standard strategy for any UFAI then is to first convince you that it is an FAI, and then to convince you that there is another UFAI “almost ready” somewhere.
Heck, if it can do #2, it might be able to skip #1 if it was able to argue that it was the less-dangerous of the two.
That’s probably why EY is so cautious about it and does not want any meaningful AGI research progress to happen until a “provably friendly AI” theory is developed. An admirable goal, though many remain skeptical of the odds of success of such an approach, or even the rationale behind it.