One possible way for AIs to coordinate with each other is for two or more AIs to modify their individual utility functions into some compromise utility function, in a mutually verifiable way, or equivalently to jointly construct a successor AI with the same compromise utility function and then hand over control of resources to the successor AI. This simply isn’t something that humans can do.
modify their individual utility functions into some compromise utility function, in a mutually verifiable way, or equivalently to jointly construct a successor AI with the same compromise utility function and then hand over control of resources to the successor AI
This is precisely equivalent to Coasean efficiency, FWIW—indeed, correspondence with some “compromise” welfare function is what it means for an outcome to be efficient in this sense. It’s definitely the case that humans, and agents more generally, can face obstacles to achieving this, so that they’re limited to some constrained-efficient outcome—something that does maximize some welfare function, but only after taking some inevitable constraints into account!
(For instance, if the pricing of some commodity, service or whatever is bounded due to an information problem, so that “cheap” versions of it predominate, then the marginal rates of transformation won’t necessarily be equalized across agents. Agent A might put her endowment towards goal X, while agent B will use her own resources to pursue some goal
Y. But that’s a constraint that could in principle be well-defined—a transaction cost. Put them all together, and you’ll understand how these constraints determine what you lose to inefficiency—the “price of anarchy”, so to speak.)
But jointly constructing a successor with compromise values and then giving them the reins is something humans can sort of do via parenting, there’s just more fuzziness and randomness and drift involved, no? That is, assuming human children take a bunch of the structure of their mindsets from what their parents teach them, which certainly seems to be the case on the face of it.
Yes, but humans generally hand off resources to their children as late as possible (whereas the AIs in my scheme would do so as soon as possible) which suggests that coordination is not the primary purpose for humans to have children.
One possible way for AIs to coordinate with each other is for two or more AIs to modify their individual utility functions into some compromise utility function, in a mutually verifiable way, or equivalently to jointly construct a successor AI with the same compromise utility function and then hand over control of resources to the successor AI. This simply isn’t something that humans can do.
This is precisely equivalent to Coasean efficiency, FWIW—indeed, correspondence with some “compromise” welfare function is what it means for an outcome to be efficient in this sense. It’s definitely the case that humans, and agents more generally, can face obstacles to achieving this, so that they’re limited to some constrained-efficient outcome—something that does maximize some welfare function, but only after taking some inevitable constraints into account!
(For instance, if the pricing of some commodity, service or whatever is bounded due to an information problem, so that “cheap” versions of it predominate, then the marginal rates of transformation won’t necessarily be equalized across agents. Agent A might put her endowment towards goal X, while agent B will use her own resources to pursue some goal Y. But that’s a constraint that could in principle be well-defined—a transaction cost. Put them all together, and you’ll understand how these constraints determine what you lose to inefficiency—the “price of anarchy”, so to speak.)
Strong upvote, very good to know
I internalised the meaning of these variables only to find you didn’t refer to them again. What was the point of this sentence.
But jointly constructing a successor with compromise values and then giving them the reins is something humans can sort of do via parenting, there’s just more fuzziness and randomness and drift involved, no? That is, assuming human children take a bunch of the structure of their mindsets from what their parents teach them, which certainly seems to be the case on the face of it.
Yes, but humans generally hand off resources to their children as late as possible (whereas the AIs in my scheme would do so as soon as possible) which suggests that coordination is not the primary purpose for humans to have children.
I’m pretty sure nobility frequently arranged marriages to do exactly this, for this purpose, to avoid costly conflicts.