A posthuman king is not centrally a king (not mortal, very different incentives), and “an AI” is a very vague bag-of-everything that might include things like simulated worlds or bureaucracies with checks and balances as special cases. The reason His Majesty’s Democratic Government doesn’t really work while the king retains ultimate authority is that the next king can be incompetent or malevolent, or its activities start threatening the king’s position and so the king is motivated to restrict them. So even “giving keys to the universe back” is not necessarily that important in the case of a posthuman god-king, but it remains a possibility after the acute risk period passes and it’s more clear how to make the next thing work.
A posthuman king is not centrally a king (not mortal, very different incentives), and “an AI” is a very vague bag-of-everything that might include things like simulated worlds or bureaucracies with checks and balances as special cases. The reason His Majesty’s Democratic Government doesn’t really work while the king retains ultimate authority is that the next king can be incompetent or malevolent, or its activities start threatening the king’s position and so the king is motivated to restrict them. So even “giving keys to the universe back” is not necessarily that important in the case of a posthuman god-king, but it remains a possibility after the acute risk period passes and it’s more clear how to make the next thing work.