If you try to formalize what “people” or “morally valuable agents” are—also known as tabooing the word “people”—then you run into problems with bad definitions that don’t match your intuition and maybe think plucked chickens are people.
That’s exactly why I’m arguing that you should not formalize or taboo “people”, because it’s not a natural category; it’s something that is best defined by pointing to a human brain and saying “whatever the brain recognizes as people, that’s people”.
I think I don’t understand what you’re saying.
If you try to formalize what “people” or “morally valuable agents” are—also known as tabooing the word “people”—then you run into problems with bad definitions that don’t match your intuition and maybe think plucked chickens are people.
That’s exactly why I’m arguing that you should not formalize or taboo “people”, because it’s not a natural category; it’s something that is best defined by pointing to a human brain and saying “whatever the brain recognizes as people, that’s people”.
Are you going to put a human brain in your superhuman AI so it can use it for a reference?
I could if I had to. Or I could tell it to analyze some brains and remember the results.