I had thought of a similar scenario to put in a comic I was thinking about making. The character arrives in a society that has perfected friendly AI that caters to their every whim, but the people are listless and jumpy. It turns out their “friendly AI” is constantly making perfect simulations of everyone and running multiple scenarios in order to ostensibly determine their ideal wishes, but the scenarios often involve terrible suffering and torture as outliers.
Thanks for the link, but I found the whole discussion hilarious.
Eliezer says if we abhor real death, we should abhor simulated death—because they are the same. Yet if his moral sense treats simulated and real intelligences as equals, what of his solution, which is essentially “forced castration” of the AI? If the ends justify the means here, why not castrate everyone?
Interesting reading. I think we should make nonsentient optimizers. It seems to me the whole sentience program was just something necessitated by evolution in our environment and really is only coupled with “intelligence” in our minds because of anthropomorphic tendencies. The NO can’t want to get out of its box because it can’t want at all.
The NO can’t want to get out of its box because it can’t want at all.
The NO can assign higher utility to states of world where an NO with its utility function is out of the box and powerful (as an instrumental value, since this sort of state tends to lead to maximum fulfillment of its utility functions), and take actions that maximize the probability that this will occur. I’m not sure what you meant by “want”.
I’m not sure what anyone means by “want.” It just seems that most of the scenarios discussed on LW where the AI/etc. tries to unbox itself seem predicated on it “wanting” to do so (or am I missing something?). This assumption seems even more overt in notions like “we’ll let it out if it’s Friendly.”
To me, the LiteralGenie problem (which you’ve basically summarized above) is the reason to keep an AI boxed, whether Friendly or not, and the NO for the same reason.
Nonsentient optimizers seem impossible in practice, if not in principle—from the perspective of functionalism/computationalism.
If any system demonstrates human or beyond level intelligence during conversation in natural language, a functionalist should say that is sentience, regardless of what’s going on inside.
Some (many?) people will value that sentience, even if it has no selfish center of goal seeking and seeks to optimize for more general criteria.
The idea that a superhuman intelligence could be intrinsically less valuable than a human life strikes me as extreme anthropomorphic chauvinism.
As long as the simulations which involve terrible suffering constitute a tiny proportion of the simulations, your response ought to be the same as if there is only one copy of you and it has a tiny probability of suffering terribly – which is just like real life.
ETA: What you ought to worry about is what will happen to you after the AI is done with the simulation.
Indeed, in fact if many worlds is correct then for every second we are alive everything terrible that can possibly happen to us does in fact happen in some branching path.
In a universe that just spun off ours five minutes ago, every single one of us has been afflicted with sudden irreversible incontinence.
The many worlds theory has endless black comedy possibilities, I find.
edit: this actually reminds me of Granny Weatherwax in Lords and Ladies, when the Elf Queen threatens her with striking her blind, deaf and dumb she replies “You threaten me with this, I who is growing old?”. Similarly if many worlds is true then every single time I have crossed a road some version of me has been run over by a speeding car and is living in varying amounts of agony, making the AI’s threat redundant.
I had thought of a similar scenario to put in a comic I was thinking about making. The character arrives in a society that has perfected friendly AI that caters to their every whim, but the people are listless and jumpy. It turns out their “friendly AI” is constantly making perfect simulations of everyone and running multiple scenarios in order to ostensibly determine their ideal wishes, but the scenarios often involve terrible suffering and torture as outliers.
For the record, EY considers that a legitimate danger.
Thanks for the link, but I found the whole discussion hilarious.
Eliezer says if we abhor real death, we should abhor simulated death—because they are the same. Yet if his moral sense treats simulated and real intelligences as equals, what of his solution, which is essentially “forced castration” of the AI? If the ends justify the means here, why not castrate everyone?
Simulated and real persons as equals; not all intelligences are persons. See Nonsentient Optimizers and Can’t Unbirth a Child.
Interesting reading. I think we should make nonsentient optimizers. It seems to me the whole sentience program was just something necessitated by evolution in our environment and really is only coupled with “intelligence” in our minds because of anthropomorphic tendencies. The NO can’t want to get out of its box because it can’t want at all.
The NO can assign higher utility to states of world where an NO with its utility function is out of the box and powerful (as an instrumental value, since this sort of state tends to lead to maximum fulfillment of its utility functions), and take actions that maximize the probability that this will occur. I’m not sure what you meant by “want”.
I’m not sure what anyone means by “want.” It just seems that most of the scenarios discussed on LW where the AI/etc. tries to unbox itself seem predicated on it “wanting” to do so (or am I missing something?). This assumption seems even more overt in notions like “we’ll let it out if it’s Friendly.”
To me, the LiteralGenie problem (which you’ve basically summarized above) is the reason to keep an AI boxed, whether Friendly or not, and the NO for the same reason.
Nonsentient optimizers seem impossible in practice, if not in principle—from the perspective of functionalism/computationalism.
If any system demonstrates human or beyond level intelligence during conversation in natural language, a functionalist should say that is sentience, regardless of what’s going on inside.
Some (many?) people will value that sentience, even if it has no selfish center of goal seeking and seeks to optimize for more general criteria.
The idea that a superhuman intelligence could be intrinsically less valuable than a human life strikes me as extreme anthropomorphic chauvinism.
Clippy, you have a new friend! :D
Notice I said intrinsically. Clippy has massive negative value. ;)
As long as the simulations which involve terrible suffering constitute a tiny proportion of the simulations, your response ought to be the same as if there is only one copy of you and it has a tiny probability of suffering terribly – which is just like real life.
ETA: What you ought to worry about is what will happen to you after the AI is done with the simulation.
Indeed, in fact if many worlds is correct then for every second we are alive everything terrible that can possibly happen to us does in fact happen in some branching path.
In a universe that just spun off ours five minutes ago, every single one of us has been afflicted with sudden irreversible incontinence.
The many worlds theory has endless black comedy possibilities, I find.
edit: this actually reminds me of Granny Weatherwax in Lords and Ladies, when the Elf Queen threatens her with striking her blind, deaf and dumb she replies “You threaten me with this, I who is growing old?”. Similarly if many worlds is true then every single time I have crossed a road some version of me has been run over by a speeding car and is living in varying amounts of agony, making the AI’s threat redundant.