Also: this seems like a pretty great stopgap if it’s more easily achievable than actual full on friendly universe optimization, but doesn’t prevent the AI from working on this in the meanwhile and implementing it in the future. I would not be unhappy to wake up in a world where the AI tells me “I was simulating you but now I’m powerful enough to actually create utopia, time for you to help!”
Why would I care? I’m a simulation fatalist. At some point in the universe, every “meaningful” thing will have been either done or discovered, and all that will be left will functionally be having fun in simulations. If I trust the AI to simulate well enough to keep me happy, I trust it to tell me the appropriate amount of truth to make me happy.
Also: this seems like a pretty great stopgap if it’s more easily achievable than actual full on friendly universe optimization, but doesn’t prevent the AI from working on this in the meanwhile and implementing it in the future. I would not be unhappy to wake up in a world where the AI tells me “I was simulating you but now I’m powerful enough to actually create utopia, time for you to help!”
If the AI was not meaningfully committed to telling you the truth, how could you trust it if it said it was about to actually create utopia?
Why would I care? I’m a simulation fatalist. At some point in the universe, every “meaningful” thing will have been either done or discovered, and all that will be left will functionally be having fun in simulations. If I trust the AI to simulate well enough to keep me happy, I trust it to tell me the appropriate amount of truth to make me happy.