The NO can’t want to get out of its box because it can’t want at all.
The NO can assign higher utility to states of world where an NO with its utility function is out of the box and powerful (as an instrumental value, since this sort of state tends to lead to maximum fulfillment of its utility functions), and take actions that maximize the probability that this will occur. I’m not sure what you meant by “want”.
I’m not sure what anyone means by “want.” It just seems that most of the scenarios discussed on LW where the AI/etc. tries to unbox itself seem predicated on it “wanting” to do so (or am I missing something?). This assumption seems even more overt in notions like “we’ll let it out if it’s Friendly.”
To me, the LiteralGenie problem (which you’ve basically summarized above) is the reason to keep an AI boxed, whether Friendly or not, and the NO for the same reason.
The NO can assign higher utility to states of world where an NO with its utility function is out of the box and powerful (as an instrumental value, since this sort of state tends to lead to maximum fulfillment of its utility functions), and take actions that maximize the probability that this will occur. I’m not sure what you meant by “want”.
I’m not sure what anyone means by “want.” It just seems that most of the scenarios discussed on LW where the AI/etc. tries to unbox itself seem predicated on it “wanting” to do so (or am I missing something?). This assumption seems even more overt in notions like “we’ll let it out if it’s Friendly.”
To me, the LiteralGenie problem (which you’ve basically summarized above) is the reason to keep an AI boxed, whether Friendly or not, and the NO for the same reason.