Right, this is where strong Bayesianism is required. You have to assume, for example, that everyone agrees on the set of hypotheses under consideration and the exact models to be used.
But under these assumptions, combining evidence always gives the right answer. Compare with the example in the post: “vote on a, vote on b, vote on a^b” which just seems strange. Shouldn’t we try to use methods that give right answers to simple questions?
The hard problem is choosing between points on the frontier… which is why norm-generation processes like voting are relevant.
I think if you have a set of coefficients for comparing different people’s utilities (maybe derived by looking into their brains and measuring how much fun they feel), then that linear combination of utilities is almost tautologically the right solution. But if your only inputs are each person’s choices in some mechanism like voting, then each person’s utility function is only determined up to affine transform, and that’s not enough information to solve the problem.
For example, imagine two agents with utility functions A and B such that A<0, B<0, AB=1. So the Pareto frontier is one branch of a hyperbola. But if the agents instead had utility functions A’=2A and B’=B/2, the frontier would be the same hyperbola. Basically there’s no affine-invariant way to pick a point on that curve.
You could say that’s because the example uses unbounded utility functions. But they are unbounded only in the negative direction, which maybe isn’t so unrealistic. And anyway, the example suggests that even for bounded utility functions, any method would have to be sensitive to the far negative reaches of utility, which seems strange. Compare to what happens when you do have coefficients for comparing utilities, then the method is nicely local.
But under these assumptions, combining evidence always gives the right answer. Compare with the example in the post: “vote on a, vote on b, vote on a^b” which just seems strange. Shouldn’t we try to use methods that give right answers to simple questions?
a) “Everyone does Bayesian updating according to the same hypothesis set, model, and measurement methods” strikes me as an extremelystrong assumption, especially since we do not have strong theory that tells us the “right” way to select these hypothesis sets, models, and measurement instruments. I would argue that this makes Aumann agreement essentially useless in “open world” scenarios.
b) Why should uniquely consistent aggregation methods exist at all? A long line of folks including Condorcet, Arrow, Sen and Parfit have pointed out that when you start aggregating beliefs, utility, or preferences, there do not exist methods that always give unambiguously “correct” answers.
I think if you have a set of coefficients for comparing different people’s utilities (maybe derived by looking into their brains and measuring how much fun they feel), then that linear combination of utilities is almost tautologically the right solution.
Sure, but finding the set of coefficients for comparing different people’s utilities is a hard problem in AI alignment, or political economy generally. Not only are there tremendous normative uncertainties here (“how much inequality is too much?”) but the problem of combining utilities a minefield of paradoxes even if you are just summing or averaging.
But under these assumptions, combining evidence always gives the right answer. Compare with the example in the post: “vote on a, vote on b, vote on a^b” which just seems strange. Shouldn’t we try to use methods that give right answers to simple questions?
I think if you have a set of coefficients for comparing different people’s utilities (maybe derived by looking into their brains and measuring how much fun they feel), then that linear combination of utilities is almost tautologically the right solution. But if your only inputs are each person’s choices in some mechanism like voting, then each person’s utility function is only determined up to affine transform, and that’s not enough information to solve the problem.
For example, imagine two agents with utility functions A and B such that A<0, B<0, AB=1. So the Pareto frontier is one branch of a hyperbola. But if the agents instead had utility functions A’=2A and B’=B/2, the frontier would be the same hyperbola. Basically there’s no affine-invariant way to pick a point on that curve.
You could say that’s because the example uses unbounded utility functions. But they are unbounded only in the negative direction, which maybe isn’t so unrealistic. And anyway, the example suggests that even for bounded utility functions, any method would have to be sensitive to the far negative reaches of utility, which seems strange. Compare to what happens when you do have coefficients for comparing utilities, then the method is nicely local.
Does that make sense?
a) “Everyone does Bayesian updating according to the same hypothesis set, model, and measurement methods” strikes me as an extremely strong assumption, especially since we do not have strong theory that tells us the “right” way to select these hypothesis sets, models, and measurement instruments. I would argue that this makes Aumann agreement essentially useless in “open world” scenarios.
b) Why should uniquely consistent aggregation methods exist at all? A long line of folks including Condorcet, Arrow, Sen and Parfit have pointed out that when you start aggregating beliefs, utility, or preferences, there do not exist methods that always give unambiguously “correct” answers.
Sure, but finding the set of coefficients for comparing different people’s utilities is a hard problem in AI alignment, or political economy generally. Not only are there tremendous normative uncertainties here (“how much inequality is too much?”) but the problem of combining utilities a minefield of paradoxes even if you are just summing or averaging.
Yeah. I was more trying to argue that, compared to Bayesian ideas, voting doesn’t win you all that much.