“Generally intelligent optimization processes” is a natural category, don’t you think?
Categories are kludges used to get around inability to make analysis more precise. The “laws” expressed in terms of natural categories are only binding as long as you remain unable to see the world at a deeper level. The question is not whether “minds” constitute a natural category, this is forlorn with smarter AIs, but whether “minds” deductively implies “things to treat as you would be treated by your superiors” (whatever that means).
The difference in rules comes from ability of more powerful AI to look at a situation and see it in detail, taking moves in favor of AI’s goals via the most unexpected exceptions to the most reliable heuristic rules. You can’t rely on natural categories when they are fought with magical intent. You can fight magic only with magic, and in this case this means postulating the particular magic in the fabric of the world that helps to justify your argument. Your complex wish can’t be granted by natural laws.
Categories are kludges used to get around inability to make analysis more precise. The “laws” expressed in terms of natural categories are only binding as long as you remain unable to see the world at a deeper level. The question is not whether “minds” constitute a natural category, this is forlorn with smarter AIs, but whether “minds” deductively implies “things to treat as you would be treated by your superiors” (whatever that means).
The difference in rules comes from ability of more powerful AI to look at a situation and see it in detail, taking moves in favor of AI’s goals via the most unexpected exceptions to the most reliable heuristic rules. You can’t rely on natural categories when they are fought with magical intent. You can fight magic only with magic, and in this case this means postulating the particular magic in the fabric of the world that helps to justify your argument. Your complex wish can’t be granted by natural laws.