Today we have a number of approaches to coordination—we sign contracts, create firms with multiple shareholders, vote in democracies, and so forth. I think the starting point for multiple AIs interacting with multiple humans is:
AI systems use similar mechanisms to coordinate amongst each other—e.g. my AI and your AI may sign a contract, or voters may have AI delegates who vote for them or advise them on how to vote.
Some AI systems are deployed by (and hopefully aligned with) a collective decision-making process—e.g. a firm may decide to deploy an AI CEO which is overseen by the board, or a government agency may deploy an AI to enforce regulations which is overseen by a bureaucratic process.
We may interleave those two basic approaches in complex ways, e.g. a firm with AI shareholders may itself deploy an AI, which may sign contracts with other firms’ AIs, which are in turn enforced by other AIs who are overseen by a process defined by that contract...
(And regardless of what happens on the object level, AIs and humans will continue improving our methods for cooperation/governance/oversight.)
When I think about this topic I’m mostly interested in ways that this “default” falls short (or may be unrealistic/impractical).
AI may also facilitate new forms of cooperation; those might be needed to cope with new challenges or the new pace introduced by AI, or may result in an improvement over the status quo. Some versions of this:
We can deploy AI systems with novel forms of oversight, rather than merely representing some existing collective decision-making process. For example, we could imagine “merging utility functions” (as in Servant of Many Masters) or some kind of collective idealization (as in CEV).
Because some kinds of AI labor are very cheap, we could use coordination mechanisms that are currently impractical—e.g. our AI systems could write many more contracts and negotiate them with low latency; or we could vote directly on a much larger number of decisions made by the state; or we could reduce agency costs by engaging in more careful oversight more sparingly.
Whether or not (or for however long) the “default” is adequate to avoid existential catastrophe, it seems useful to use AI as an opportunity to improve our coordination. In some sense “most” of that work will presumably be done by AI systems, but doing the work ourselves may unlock those benefits much earlier. That may be critically important if the transition to AI creates a ton of chaos before we have AI systems who are much better than us at designing new cooperative arrangements. (This is fairly similar to the situation with alignment, where we could also wait to delegate the problem to AI but that may be too late to avoid trouble.)
I think both sets of bullets (multi-multi (eco?)systems either replicating cooperation-etc-as-we-know-it or making new forms of cooperation etc) are important, I think I’ll call them prosaic cooperation and nonprosaic cooperation, respectively, going forward. When I say “cooperation etc.” I mean cooperation, coordination, competition, negotiation, compromise.
You’ve provided crisp scenarios, so thanks for that!
In some sense “most” of that work will presumably be done by AI systems, but doing the work ourselves may unlock those benefits much earlier.
But if the AI does that work there will be an interpretability problem, an inferential distance. I’m imagining people ask a somewhat single-single aligned AI for solutions to multi-multi problems and the black box returns something inscrutable. Putting ourselves in a position where we can grok what its recommendations are seems aligned with researching it for ourselves so we won’t have to ask the black box in the first place, though this probably only applies to prosaic cooperation.
Today we have a number of approaches to coordination—we sign contracts, create firms with multiple shareholders, vote in democracies, and so forth. I think the starting point for multiple AIs interacting with multiple humans is:
AI systems use similar mechanisms to coordinate amongst each other—e.g. my AI and your AI may sign a contract, or voters may have AI delegates who vote for them or advise them on how to vote.
Some AI systems are deployed by (and hopefully aligned with) a collective decision-making process—e.g. a firm may decide to deploy an AI CEO which is overseen by the board, or a government agency may deploy an AI to enforce regulations which is overseen by a bureaucratic process.
We may interleave those two basic approaches in complex ways, e.g. a firm with AI shareholders may itself deploy an AI, which may sign contracts with other firms’ AIs, which are in turn enforced by other AIs who are overseen by a process defined by that contract...
(And regardless of what happens on the object level, AIs and humans will continue improving our methods for cooperation/governance/oversight.)
When I think about this topic I’m mostly interested in ways that this “default” falls short (or may be unrealistic/impractical).
AI may also facilitate new forms of cooperation; those might be needed to cope with new challenges or the new pace introduced by AI, or may result in an improvement over the status quo. Some versions of this:
We can deploy AI systems with novel forms of oversight, rather than merely representing some existing collective decision-making process. For example, we could imagine “merging utility functions” (as in Servant of Many Masters) or some kind of collective idealization (as in CEV).
Because some kinds of AI labor are very cheap, we could use coordination mechanisms that are currently impractical—e.g. our AI systems could write many more contracts and negotiate them with low latency; or we could vote directly on a much larger number of decisions made by the state; or we could reduce agency costs by engaging in more careful oversight more sparingly.
Whether or not (or for however long) the “default” is adequate to avoid existential catastrophe, it seems useful to use AI as an opportunity to improve our coordination. In some sense “most” of that work will presumably be done by AI systems, but doing the work ourselves may unlock those benefits much earlier. That may be critically important if the transition to AI creates a ton of chaos before we have AI systems who are much better than us at designing new cooperative arrangements. (This is fairly similar to the situation with alignment, where we could also wait to delegate the problem to AI but that may be too late to avoid trouble.)
I think both sets of bullets (multi-multi (eco?)systems either replicating cooperation-etc-as-we-know-it or making new forms of cooperation etc) are important, I think I’ll call them prosaic cooperation and nonprosaic cooperation, respectively, going forward. When I say “cooperation etc.” I mean cooperation, coordination, competition, negotiation, compromise.
You’ve provided crisp scenarios, so thanks for that!
But if the AI does that work there will be an interpretability problem, an inferential distance. I’m imagining people ask a somewhat single-single aligned AI for solutions to multi-multi problems and the black box returns something inscrutable. Putting ourselves in a position where we can grok what its recommendations are seems aligned with researching it for ourselves so we won’t have to ask the black box in the first place, though this probably only applies to prosaic cooperation.