One approach to treating an AI ethically is to design it to not be a person. Of course, this means building it the hard way, but, as Tetronian notes, that is already a requirement of making it Friendly.
I’m inclined to think that any computer complex enough to be useful will at least have to have a model of itself and a model of what changes to the self (or possibly to the model of itself, which gets to be an interesting distinction) are acceptable. This is at least something like being a person, though presumably it wouldn’t need to be able to experience pain.
I’m not going to exclude the possibility of something like pain, though, either—it might be the most efficient way of modeling “don’t do that”.
Huh—this makes p-zombies interesting. Could an AI need qualia?
“Um—okay, look, putting aside the obvious objection that any sufficiently powerful intelligence will be able to model itself—”
Lob’s Sentence contains an exact recipe for a copy of itself, including the recipe for the recipe; it has a perfect self-model. Does that make it sentient?
One approach to treating an AI ethically is to design it to not be a person. Of course, this means building it the hard way, but, as Tetronian notes, that is already a requirement of making it Friendly.
What are the boundaries of not being a person?
I’m inclined to think that any computer complex enough to be useful will at least have to have a model of itself and a model of what changes to the self (or possibly to the model of itself, which gets to be an interesting distinction) are acceptable. This is at least something like being a person, though presumably it wouldn’t need to be able to experience pain.
I’m not going to exclude the possibility of something like pain, though, either—it might be the most efficient way of modeling “don’t do that”.
Huh—this makes p-zombies interesting. Could an AI need qualia?
Eliezer has anticipated your argument:
I think it’s relevant that the self-model for an AI would change as the AI changes.