You can peek into everyone’s heads, gather all the evidence, remove double-counting, and perform a joint update. That’s basically what Aumann agreement does—it doesn’t vote on beliefs, but instead tries to reach an end state that’s updated on all the evidence behind these beliefs.
Right, this is where strong Bayesianism is required. You have to assume, for example, that everyone agrees on the set of hypotheses under consideration and the exact models to be used. This is not just an abstract plan for slicing the universe into manageable events, but the actual structure and properties of the measurement instruments that generate “evidence.” If we wish to act as well we also have to specify the set of possible interventions and their expected outcomes. These choices are well outside the scope of a Bayesian update (see e.g. Gelman and Shalizi or John Norton).
Also, I do not have a super-intelligent AI. I’m working on narrow AI alignment, and many of these systems have social choice problems too, for example recommender systems.
Imagine that after doing the joint update, the agents agree to cooperate instead of fighting, and have a set of possible joint policies. Each joint policy leads to a tuple of expected utilities for all agents. The resulting set of points in N-dimensional space has a Pareto frontier.
The Pareto frontier is a very weak constraint, and lots of points on it are bad. For a self-driving car that wants to drive both quickly and safely, both not moving at all and driving as fast as possible are on the frontier. For a distribution of wealth problem, “one person gets everything” is on the frontier. The hard problem is choosing between points on the frontier, that is, trading off one person’s utility against another. There is a long tradition of work within political economy which considers this problem in detail. It is, of course, partly a normative question, which is why norm-generation processes like voting are relevant.
Right, this is where strong Bayesianism is required. You have to assume, for example, that everyone agrees on the set of hypotheses under consideration and the exact models to be used.
But under these assumptions, combining evidence always gives the right answer. Compare with the example in the post: “vote on a, vote on b, vote on a^b” which just seems strange. Shouldn’t we try to use methods that give right answers to simple questions?
The hard problem is choosing between points on the frontier… which is why norm-generation processes like voting are relevant.
I think if you have a set of coefficients for comparing different people’s utilities (maybe derived by looking into their brains and measuring how much fun they feel), then that linear combination of utilities is almost tautologically the right solution. But if your only inputs are each person’s choices in some mechanism like voting, then each person’s utility function is only determined up to affine transform, and that’s not enough information to solve the problem.
For example, imagine two agents with utility functions A and B such that A<0, B<0, AB=1. So the Pareto frontier is one branch of a hyperbola. But if the agents instead had utility functions A’=2A and B’=B/2, the frontier would be the same hyperbola. Basically there’s no affine-invariant way to pick a point on that curve.
You could say that’s because the example uses unbounded utility functions. But they are unbounded only in the negative direction, which maybe isn’t so unrealistic. And anyway, the example suggests that even for bounded utility functions, any method would have to be sensitive to the far negative reaches of utility, which seems strange. Compare to what happens when you do have coefficients for comparing utilities, then the method is nicely local.
But under these assumptions, combining evidence always gives the right answer. Compare with the example in the post: “vote on a, vote on b, vote on a^b” which just seems strange. Shouldn’t we try to use methods that give right answers to simple questions?
a) “Everyone does Bayesian updating according to the same hypothesis set, model, and measurement methods” strikes me as an extremelystrong assumption, especially since we do not have strong theory that tells us the “right” way to select these hypothesis sets, models, and measurement instruments. I would argue that this makes Aumann agreement essentially useless in “open world” scenarios.
b) Why should uniquely consistent aggregation methods exist at all? A long line of folks including Condorcet, Arrow, Sen and Parfit have pointed out that when you start aggregating beliefs, utility, or preferences, there do not exist methods that always give unambiguously “correct” answers.
I think if you have a set of coefficients for comparing different people’s utilities (maybe derived by looking into their brains and measuring how much fun they feel), then that linear combination of utilities is almost tautologically the right solution.
Sure, but finding the set of coefficients for comparing different people’s utilities is a hard problem in AI alignment, or political economy generally. Not only are there tremendous normative uncertainties here (“how much inequality is too much?”) but the problem of combining utilities a minefield of paradoxes even if you are just summing or averaging.
Right, this is where strong Bayesianism is required. You have to assume, for example, that everyone agrees on the set of hypotheses under consideration and the exact models to be used. This is not just an abstract plan for slicing the universe into manageable events, but the actual structure and properties of the measurement instruments that generate “evidence.” If we wish to act as well we also have to specify the set of possible interventions and their expected outcomes. These choices are well outside the scope of a Bayesian update (see e.g. Gelman and Shalizi or John Norton).
Also, I do not have a super-intelligent AI. I’m working on narrow AI alignment, and many of these systems have social choice problems too, for example recommender systems.
The Pareto frontier is a very weak constraint, and lots of points on it are bad. For a self-driving car that wants to drive both quickly and safely, both not moving at all and driving as fast as possible are on the frontier. For a distribution of wealth problem, “one person gets everything” is on the frontier. The hard problem is choosing between points on the frontier, that is, trading off one person’s utility against another. There is a long tradition of work within political economy which considers this problem in detail. It is, of course, partly a normative question, which is why norm-generation processes like voting are relevant.
But under these assumptions, combining evidence always gives the right answer. Compare with the example in the post: “vote on a, vote on b, vote on a^b” which just seems strange. Shouldn’t we try to use methods that give right answers to simple questions?
I think if you have a set of coefficients for comparing different people’s utilities (maybe derived by looking into their brains and measuring how much fun they feel), then that linear combination of utilities is almost tautologically the right solution. But if your only inputs are each person’s choices in some mechanism like voting, then each person’s utility function is only determined up to affine transform, and that’s not enough information to solve the problem.
For example, imagine two agents with utility functions A and B such that A<0, B<0, AB=1. So the Pareto frontier is one branch of a hyperbola. But if the agents instead had utility functions A’=2A and B’=B/2, the frontier would be the same hyperbola. Basically there’s no affine-invariant way to pick a point on that curve.
You could say that’s because the example uses unbounded utility functions. But they are unbounded only in the negative direction, which maybe isn’t so unrealistic. And anyway, the example suggests that even for bounded utility functions, any method would have to be sensitive to the far negative reaches of utility, which seems strange. Compare to what happens when you do have coefficients for comparing utilities, then the method is nicely local.
Does that make sense?
a) “Everyone does Bayesian updating according to the same hypothesis set, model, and measurement methods” strikes me as an extremely strong assumption, especially since we do not have strong theory that tells us the “right” way to select these hypothesis sets, models, and measurement instruments. I would argue that this makes Aumann agreement essentially useless in “open world” scenarios.
b) Why should uniquely consistent aggregation methods exist at all? A long line of folks including Condorcet, Arrow, Sen and Parfit have pointed out that when you start aggregating beliefs, utility, or preferences, there do not exist methods that always give unambiguously “correct” answers.
Sure, but finding the set of coefficients for comparing different people’s utilities is a hard problem in AI alignment, or political economy generally. Not only are there tremendous normative uncertainties here (“how much inequality is too much?”) but the problem of combining utilities a minefield of paradoxes even if you are just summing or averaging.
Yeah. I was more trying to argue that, compared to Bayesian ideas, voting doesn’t win you all that much.