You shouldn’t try to taboo “people”. Actual human brains really do think in terms of the category “people”. If the world changes and the category no longer carves it at its joints (say, if superhuman AI is developed), human brains will remain to some extent hardwired with their category of “people”. The only answer to the question of what constitutes a person is to go look at how human brains pattern-match things to recognize persons, which is that they look and behave like humans.
That kind of attitude is an extremely effective way of -preventing- you from developing superhuman AI, or at least the kind you’d -want- to develop. Your superhuman AI needs to know the difference between plucked chickens and Greek philosophers.
If you try to formalize what “people” or “morally valuable agents” are—also known as tabooing the word “people”—then you run into problems with bad definitions that don’t match your intuition and maybe think plucked chickens are people.
That’s exactly why I’m arguing that you should not formalize or taboo “people”, because it’s not a natural category; it’s something that is best defined by pointing to a human brain and saying “whatever the brain recognizes as people, that’s people”.
You shouldn’t try to taboo “people”. Actual human brains really do think in terms of the category “people”. If the world changes and the category no longer carves it at its joints (say, if superhuman AI is developed), human brains will remain to some extent hardwired with their category of “people”. The only answer to the question of what constitutes a person is to go look at how human brains pattern-match things to recognize persons, which is that they look and behave like humans.
That kind of attitude is an extremely effective way of -preventing- you from developing superhuman AI, or at least the kind you’d -want- to develop. Your superhuman AI needs to know the difference between plucked chickens and Greek philosophers.
I think I don’t understand what you’re saying.
If you try to formalize what “people” or “morally valuable agents” are—also known as tabooing the word “people”—then you run into problems with bad definitions that don’t match your intuition and maybe think plucked chickens are people.
That’s exactly why I’m arguing that you should not formalize or taboo “people”, because it’s not a natural category; it’s something that is best defined by pointing to a human brain and saying “whatever the brain recognizes as people, that’s people”.
Are you going to put a human brain in your superhuman AI so it can use it for a reference?
I could if I had to. Or I could tell it to analyze some brains and remember the results.