I think if there was an FAI, and it successfully built utopia, it would have to solve this problem of convincing us that it did (?using something like the Arthur/Merlin protocol?)
edit: I realize that there is a nearly limitless room for trickery here, I am just saying that if I had to define FAI success it would have to be in terms of some sort of convincing game with asymmetric computing power setup, how else would we even do it? Importantly, Merlin is allowed to lie.
I think if there was an FAI, and it successfully built utopia, it would have to solve this problem of convincing us that it did (?using something like the Arthur/Merlin protocol?)
edit: I realize that there is a nearly limitless room for trickery here, I am just saying that if I had to define FAI success it would have to be in terms of some sort of convincing game with asymmetric computing power setup, how else would we even do it? Importantly, Merlin is allowed to lie.