“Generally intelligent optimization processes” is a natural category, don’t you think? (Though there might be no non-magical-thinking reason for it to be game-theoretically relevant in this way. Or the most game-theoretically natural (Schelling point) border might exclude humans.)
“Generally intelligent optimization processes” is a natural category, don’t you think?
Categories are kludges used to get around inability to make analysis more precise. The “laws” expressed in terms of natural categories are only binding as long as you remain unable to see the world at a deeper level. The question is not whether “minds” constitute a natural category, this is forlorn with smarter AIs, but whether “minds” deductively implies “things to treat as you would be treated by your superiors” (whatever that means).
The difference in rules comes from ability of more powerful AI to look at a situation and see it in detail, taking moves in favor of AI’s goals via the most unexpected exceptions to the most reliable heuristic rules. You can’t rely on natural categories when they are fought with magical intent. You can fight magic only with magic, and in this case this means postulating the particular magic in the fabric of the world that helps to justify your argument. Your complex wish can’t be granted by natural laws.
“Generally intelligent optimization processes” is a natural category, don’t you think? (Though there might be no non-magical-thinking reason for it to be game-theoretically relevant in this way. Or the most game-theoretically natural (Schelling point) border might exclude humans.)
Categories are kludges used to get around inability to make analysis more precise. The “laws” expressed in terms of natural categories are only binding as long as you remain unable to see the world at a deeper level. The question is not whether “minds” constitute a natural category, this is forlorn with smarter AIs, but whether “minds” deductively implies “things to treat as you would be treated by your superiors” (whatever that means).
The difference in rules comes from ability of more powerful AI to look at a situation and see it in detail, taking moves in favor of AI’s goals via the most unexpected exceptions to the most reliable heuristic rules. You can’t rely on natural categories when they are fought with magical intent. You can fight magic only with magic, and in this case this means postulating the particular magic in the fabric of the world that helps to justify your argument. Your complex wish can’t be granted by natural laws.