“That is, it is hard for humans to coordinate to exclude some humans from benefiting from these institutions.”
Humans do this all the time: much of the world is governed by kleptocracies that select policy apparently on the basis of preventing successful rebellion and extracting production. The strength of the apparatus of oppression, which is affected by technological and organizational factors, can dramatically affect the importance of the threat of rebellion. In North Korea the regime can allow millions of citizens to starve so long as the soldiers are paid and top officials rewarded. The small size of the winning coalition can be masked if positive treatment of the subjects increases the size of the tax base, enables military recruitment, or otherwise pays off for self-interested rulers. However, if human labor productivity is insufficient to justify a subsistence wage, then there is no longer a ‘tax farmer’ case for not slaughtering and expropriating the citizenry.
“If AIs use these same institutions, perhaps somewhat modified, to coordinate with each other, humans would similarly benefit from AI coordination.”
What is difficult for humans need not be comparably difficult for entities capable of making digital copies of themselves, reverting to saved versions, and modifying their psychological processes relatively easily. I have a paper underway on this, which would probably enable a more productive discussion, so I’ll suggest a postponement.
“That is, it is hard for humans to coordinate to exclude some humans from benefiting from these institutions.”
Humans do this all the time: much of the world is governed by kleptocracies that select policy apparently on the basis of preventing successful rebellion and extracting production. The strength of the apparatus of oppression, which is affected by technological and organizational factors, can dramatically affect the importance of the threat of rebellion. In North Korea the regime can allow millions of citizens to starve so long as the soldiers are paid and top officials rewarded. The small size of the winning coalition can be masked if positive treatment of the subjects increases the size of the tax base, enables military recruitment, or otherwise pays off for self-interested rulers. However, if human labor productivity is insufficient to justify a subsistence wage, then there is no longer a ‘tax farmer’ case for not slaughtering and expropriating the citizenry.
“If AIs use these same institutions, perhaps somewhat modified, to coordinate with each other, humans would similarly benefit from AI coordination.”
What is difficult for humans need not be comparably difficult for entities capable of making digital copies of themselves, reverting to saved versions, and modifying their psychological processes relatively easily. I have a paper underway on this, which would probably enable a more productive discussion, so I’ll suggest a postponement.