what if the universe turns out to be much larger than previously thought, and the AI says “I’m at 99.999% of achievable utility already, it’s not worth it to expand farther or live longer”?
A small risk of losing the utility it was previously counting on.
Of course you can do intuition pumps either way- I don’t feel like I’d want the AI to sacrifice everything in the universe we know for a 0.01% chance of making it in a bigger universe- but some level of risk has to be worth a vast increase in potential fun.
LCPW cuts two ways here, because there are two universal quantifiers in your claim. You need to look at every possible bounded utility function, not just every possible scenario. At least, if I understand you correctly, you’re claiming that no bounded utility function reflects your preferences accurately.
resources, whether physical or computational. Presumably the AI is programmed to utilize resources in a parsimonious manner, with terms governing various applications of the resources, including powering the AI, and deciding on what to do. If the AI is programmed to limit what it does at some large but arbitrary point, because we don’t want it taking over the universe or whatever, then this point might end up actually being before we want it to stop doing whatever it’s doing.
It’s not worth what?
A small risk of losing the utility it was previously counting on.
Of course you can do intuition pumps either way- I don’t feel like I’d want the AI to sacrifice everything in the universe we know for a 0.01% chance of making it in a bigger universe- but some level of risk has to be worth a vast increase in potential fun.
It seems to me that expanding further would reduce the risk of losing the utility it was previously counting on.
LCPW isn’t even necessary: do you really think that it wouldn’t make a difference that you’d care about?
LCPW cuts two ways here, because there are two universal quantifiers in your claim. You need to look at every possible bounded utility function, not just every possible scenario. At least, if I understand you correctly, you’re claiming that no bounded utility function reflects your preferences accurately.
resources, whether physical or computational. Presumably the AI is programmed to utilize resources in a parsimonious manner, with terms governing various applications of the resources, including powering the AI, and deciding on what to do. If the AI is programmed to limit what it does at some large but arbitrary point, because we don’t want it taking over the universe or whatever, then this point might end up actually being before we want it to stop doing whatever it’s doing.
That doesn’t sound like an expected utility maximizer.