The entire premise of Xyrik’s scenario is that everything will be hunky-dory. Xyrik is just making a wish, and not thinking about how anything will actually work.
Well, to be fair, I never claimed that I had any ideas for how to actually achieve a scenario with a flawless AGI, and I don’t think I even said I was under the impression that this would be a good idea, although in the case that we DID have a flawless AGI, I would be open to a reasoning that proclaimed so.
But all I was asking was what potential downsides this could have, and people have risen to the occasion.
You know, this seems amusingly analogous to the scene in the seventh Harry Potter novel in which Xenophillius Lovegood asks Hermione to falsify the existence of the Resurrection Stone.