Like, yes, when I say ‘the moral evaluation function takes the dog’s brain as an input, not the cuteness of its overt behaviors’, I am talking about a moral evaluation function that we have to extract from the human’s brain.
But the human moral evaluation function is a totally different function from the ‘does-this-thing-make-noises-and-facial-expressions-that-naturally-make-me-feel-sympathy-for-it-before-I-learn-any-neuroscience?’ function, even though both are located in the human brain.
Thinking (with very low confidence) about an idealized, heavily self-modified, reflectively consistent, CEV-ish version of me:
If it turns out that squirrels are totally unconscious automata, then I think Ideal Me would probably at least weakly prefer to not go around stepping on squirrels for fun. I think this would be for two reasons:
The kind of reverence-for-beauty that makes me not want to randomly shred flowers to pieces. Squirrels can be beautiful even if they have no moral value. Gorgeous sunsets plausibly deserve a similar kind of reverence.
The kind of disgust that makes me not want to draw pictures of mutilated humans. There may be nothing morally important about the cognitive algorithms in squirrels’ brains; but squirrels still have a lot of anatomical similarities to humans, and the visual resemblance between the two is reason enough to be grossed out by roadkill.
In both cases, these don’t seem like obviously bad values to me. (And I’m pretty conservative about getting rid of my values! Though a lot can and should change eventually, as humanity figures out all the risks and implications of various self-modifications. Indeed, I think the above descriptions would probably look totally wrong, quaint, and confused to a real CEV of mine; but it’s my best guess for now.)
In contrast, conflating the moral worth of genuinely-totally-conscious things (insofar as anything is genuinely conscious) with genuinely-totally-unconscious things seems… actively bad, to me? Not a value worth endorsing or protecting?
Like, maybe you think it’s implausible that squirrels, with all their behavioral complexity, could have ‘the lights be off’ in the way that a roomba with a cute face glued to it has ‘the lights off’. I disagree somewhat, but I find that view vastly less objectionable than ‘it doesn’t even matter what the squirrel’s mind is like, it just matters how uneducated humans naively emotionally respond to the squirrel’s overt behaviors’.
Maybe a way of gesturing at the thing is: Phenomenal consciousness is an illusion, but the illusion adds up to normality. It doesn’t add up to ‘therefore the difference between automata / cartoon characters and things-that-actually-have-the-relevant-mental-machinery-in-their-brains suddenly becomes unimportant (or even less important)’.
Like, yes, when I say ‘the moral evaluation function takes the dog’s brain as an input, not the cuteness of its overt behaviors’, I am talking about a moral evaluation function that we have to extract from the human’s brain.
But the human moral evaluation function is a totally different function from the ‘does-this-thing-make-noises-and-facial-expressions-that-naturally-make-me-feel-sympathy-for-it-before-I-learn-any-neuroscience?’ function, even though both are located in the human brain.
Thinking (with very low confidence) about an idealized, heavily self-modified, reflectively consistent, CEV-ish version of me:
If it turns out that squirrels are totally unconscious automata, then I think Ideal Me would probably at least weakly prefer to not go around stepping on squirrels for fun. I think this would be for two reasons:
The kind of reverence-for-beauty that makes me not want to randomly shred flowers to pieces. Squirrels can be beautiful even if they have no moral value. Gorgeous sunsets plausibly deserve a similar kind of reverence.
The kind of disgust that makes me not want to draw pictures of mutilated humans. There may be nothing morally important about the cognitive algorithms in squirrels’ brains; but squirrels still have a lot of anatomical similarities to humans, and the visual resemblance between the two is reason enough to be grossed out by roadkill.
In both cases, these don’t seem like obviously bad values to me. (And I’m pretty conservative about getting rid of my values! Though a lot can and should change eventually, as humanity figures out all the risks and implications of various self-modifications. Indeed, I think the above descriptions would probably look totally wrong, quaint, and confused to a real CEV of mine; but it’s my best guess for now.)
In contrast, conflating the moral worth of genuinely-totally-conscious things (insofar as anything is genuinely conscious) with genuinely-totally-unconscious things seems… actively bad, to me? Not a value worth endorsing or protecting?
Like, maybe you think it’s implausible that squirrels, with all their behavioral complexity, could have ‘the lights be off’ in the way that a roomba with a cute face glued to it has ‘the lights off’. I disagree somewhat, but I find that view vastly less objectionable than ‘it doesn’t even matter what the squirrel’s mind is like, it just matters how uneducated humans naively emotionally respond to the squirrel’s overt behaviors’.
Maybe a way of gesturing at the thing is: Phenomenal consciousness is an illusion, but the illusion adds up to normality. It doesn’t add up to ‘therefore the difference between automata / cartoon characters and things-that-actually-have-the-relevant-mental-machinery-in-their-brains suddenly becomes unimportant (or even less important)’.