I would instead ask “What preferences would this agent have, in a counterfactual universe in which they were fully-informed and rational but otherwise identical?”.
“The problem with trying to extrapolate what a person would want with perfect information is, perfect information is a lot of fucking information. The human brain can’t handle that much information, so if you want your extrapolatory homunculus to do anything but scream and die like someone put into the Total Perspective Vortex, you need to enhance its information processing capabilities. And once you’ve reached that point, why not improve its general intelligence too, so it can make better decisions? Maybe teach it a little bit about heuristics and biases, to help it make more rational choices. And you know it wouldn’t really hate blacks except for those pesky emotions that get in the way, so lets throw those out the window. You know what, let’s just replace it with a copy of me, I want all the cool things anyway.
Truly, the path of a utilitarian is a thorny one. That’s why I prefer a whimsicalist moral philosophy. Whimsicalism is a humanism!”
The sophisticated reader presented with a slippery slope argument like that one first checks whether there really is a force driving us in a particular direction, that makes the metaphorical terrain a slippery slope rather than just a slippery field, and secondly they check whether there are any defensible points of cleavage in the metaphorical terrain that could be used to build a fence and stop the slide at some point.
The slippery slope argument you are quoting, when uprooted and placed in this context, seems to me to fail both tests. There’s no reason at all to descend progressively into the problems described, and even if there was you could draw a line and say “we’re just going to inform our mental model of any relevant facts we know that it doesn’t, and fix any mental processes our construct has that are clearly highly irrational”.
You haven’t given us a link but going by the principle of charity I imagine that what you’ve done here is take a genuine problem with building a weakly God-like friendly AI and tried to transplant the argument into the context of intervening in a suicide attempt, where it doesn’t belong.
I would instead ask “What preferences would this agent have, in a counterfactual universe in which they were fully-informed and rational but otherwise identical?”.
Quoting a forum post from a couple years ago...
“The problem with trying to extrapolate what a person would want with perfect information is, perfect information is a lot of fucking information. The human brain can’t handle that much information, so if you want your extrapolatory homunculus to do anything but scream and die like someone put into the Total Perspective Vortex, you need to enhance its information processing capabilities. And once you’ve reached that point, why not improve its general intelligence too, so it can make better decisions? Maybe teach it a little bit about heuristics and biases, to help it make more rational choices. And you know it wouldn’t really hate blacks except for those pesky emotions that get in the way, so lets throw those out the window. You know what, let’s just replace it with a copy of me, I want all the cool things anyway.
Truly, the path of a utilitarian is a thorny one. That’s why I prefer a whimsicalist moral philosophy. Whimsicalism is a humanism!”
The sophisticated reader presented with a slippery slope argument like that one first checks whether there really is a force driving us in a particular direction, that makes the metaphorical terrain a slippery slope rather than just a slippery field, and secondly they check whether there are any defensible points of cleavage in the metaphorical terrain that could be used to build a fence and stop the slide at some point.
The slippery slope argument you are quoting, when uprooted and placed in this context, seems to me to fail both tests. There’s no reason at all to descend progressively into the problems described, and even if there was you could draw a line and say “we’re just going to inform our mental model of any relevant facts we know that it doesn’t, and fix any mental processes our construct has that are clearly highly irrational”.
You haven’t given us a link but going by the principle of charity I imagine that what you’ve done here is take a genuine problem with building a weakly God-like friendly AI and tried to transplant the argument into the context of intervening in a suicide attempt, where it doesn’t belong.