Follow the utility function that program X would output if given enough time” doesn’t require the AI to run program X, only to reason about the likely outputs of program X.
It doesn’t in principle require this, but might in practice, in which case the AI might eat the universe if that’s the amount of computational resources necessary to compute the results of running program X. That is a potential downside of this plan.
Well on the dark, sardonic upside, it might find it convenient to eat the people in the process of using their minds to compute a CEV-function. Infinite varieties of infinite hell-eternities for everyone!
It doesn’t in principle require this, but might in practice, in which case the AI might eat the universe if that’s the amount of computational resources necessary to compute the results of running program X. That is a potential downside of this plan.
Well on the dark, sardonic upside, it might find it convenient to eat the people in the process of using their minds to compute a CEV-function. Infinite varieties of infinite hell-eternities for everyone!