As usual, a converse problem is likely to give useful insights. I would approach this issue from the other direction: what prevents humans from coordinating with each other at low cost? Do we expect AIs to have similar traits/issues?
One possible way for AIs to coordinate with each other is for two or more AIs to modify their individual utility functions into some compromise utility function, in a mutually verifiable way, or equivalently to jointly construct a successor AI with the same compromise utility function and then hand over control of resources to the successor AI. This simply isn’t something that humans can do.
modify their individual utility functions into some compromise utility function, in a mutually verifiable way, or equivalently to jointly construct a successor AI with the same compromise utility function and then hand over control of resources to the successor AI
This is precisely equivalent to Coasean efficiency, FWIW—indeed, correspondence with some “compromise” welfare function is what it means for an outcome to be efficient in this sense. It’s definitely the case that humans, and agents more generally, can face obstacles to achieving this, so that they’re limited to some constrained-efficient outcome—something that does maximize some welfare function, but only after taking some inevitable constraints into account!
(For instance, if the pricing of some commodity, service or whatever is bounded due to an information problem, so that “cheap” versions of it predominate, then the marginal rates of transformation won’t necessarily be equalized across agents. Agent A might put her endowment towards goal X, while agent B will use her own resources to pursue some goal
Y. But that’s a constraint that could in principle be well-defined—a transaction cost. Put them all together, and you’ll understand how these constraints determine what you lose to inefficiency—the “price of anarchy”, so to speak.)
But jointly constructing a successor with compromise values and then giving them the reins is something humans can sort of do via parenting, there’s just more fuzziness and randomness and drift involved, no? That is, assuming human children take a bunch of the structure of their mindsets from what their parents teach them, which certainly seems to be the case on the face of it.
Yes, but humans generally hand off resources to their children as late as possible (whereas the AIs in my scheme would do so as soon as possible) which suggests that coordination is not the primary purpose for humans to have children.
Humans tend to highly value their own personal experiences—getting to do things that feel fun, acquiring personal status, putting intrinsic value on their own survival, etc. This limits the extent to which they’ll co-operate with a group, since their own interests and the interests of the group are only partially the same. AIs with less personal interests would be better incentivized to coordinate—if you only care about the amount of paperclips in the universe, you will be able to better further that goal with others than if each AI was instead optimizing for the amount of paperclips that they personally got to produce.
Some academics argue that religion etc. evolved for the purpose of suppressing personal interests and giving them a common impersonal goal, partially getting around this problem. I discussed this and its connection to these matters a bit in my old post Intelligence Explosion vs. Co-operative Explosion.
As usual, a converse problem is likely to give useful insights. I would approach this issue from the other direction: what prevents humans from coordinating with each other at low cost? Do we expect AIs to have similar traits/issues?
One possible way for AIs to coordinate with each other is for two or more AIs to modify their individual utility functions into some compromise utility function, in a mutually verifiable way, or equivalently to jointly construct a successor AI with the same compromise utility function and then hand over control of resources to the successor AI. This simply isn’t something that humans can do.
This is precisely equivalent to Coasean efficiency, FWIW—indeed, correspondence with some “compromise” welfare function is what it means for an outcome to be efficient in this sense. It’s definitely the case that humans, and agents more generally, can face obstacles to achieving this, so that they’re limited to some constrained-efficient outcome—something that does maximize some welfare function, but only after taking some inevitable constraints into account!
(For instance, if the pricing of some commodity, service or whatever is bounded due to an information problem, so that “cheap” versions of it predominate, then the marginal rates of transformation won’t necessarily be equalized across agents. Agent A might put her endowment towards goal X, while agent B will use her own resources to pursue some goal Y. But that’s a constraint that could in principle be well-defined—a transaction cost. Put them all together, and you’ll understand how these constraints determine what you lose to inefficiency—the “price of anarchy”, so to speak.)
Strong upvote, very good to know
I internalised the meaning of these variables only to find you didn’t refer to them again. What was the point of this sentence.
But jointly constructing a successor with compromise values and then giving them the reins is something humans can sort of do via parenting, there’s just more fuzziness and randomness and drift involved, no? That is, assuming human children take a bunch of the structure of their mindsets from what their parents teach them, which certainly seems to be the case on the face of it.
Yes, but humans generally hand off resources to their children as late as possible (whereas the AIs in my scheme would do so as soon as possible) which suggests that coordination is not the primary purpose for humans to have children.
I’m pretty sure nobility frequently arranged marriages to do exactly this, for this purpose, to avoid costly conflicts.
Humans tend to highly value their own personal experiences—getting to do things that feel fun, acquiring personal status, putting intrinsic value on their own survival, etc. This limits the extent to which they’ll co-operate with a group, since their own interests and the interests of the group are only partially the same. AIs with less personal interests would be better incentivized to coordinate—if you only care about the amount of paperclips in the universe, you will be able to better further that goal with others than if each AI was instead optimizing for the amount of paperclips that they personally got to produce.
Some academics argue that religion etc. evolved for the purpose of suppressing personal interests and giving them a common impersonal goal, partially getting around this problem. I discussed this and its connection to these matters a bit in my old post Intelligence Explosion vs. Co-operative Explosion.