I think when several AIs with bounded utility functions decide to merge, they can reach any point on the Pareto frontier like this:
1) Allow linear combinations of utility functions. This lets you reach all “pointy” points.
2) Allow making a tuple of functions of type (1) whose values should be compared lexicographically (e.g. “maximize U+V, break ties by maximizing U”). This lets you reach some points on the edges of flat parts.
3) Allow the merging process to choose randomly which function of type (2) to give to the merged AI. This lets you reach the rest of the points on flat parts.
That’s a bit complicated, but I don’t think there’s a simpler way.
I don’t see why 2 is necessary given that any point on the Pareto frontier is a mixture of pointy points (intuition for this: any point on the face of a polyhedron is a mixture of that face’s corners). In any case, I agree with the basic mathematical point that you can get any Pareto optimal mixture of outcomes by mixing between non-negative linear combinations of utility functions.
Well, I was imagining a Pareto frontier that changes smoothly from flat to curved. Then we can’t quite get a pointy point exactly on the edge of the flat part. That’s what 2 is for, it gives us some of these points (though not all). But I guess that doesn’t matter if things are finite enough.
I think when several AIs with bounded utility functions decide to merge, they can reach any point on the Pareto frontier like this:
1) Allow linear combinations of utility functions. This lets you reach all “pointy” points.
2) Allow making a tuple of functions of type (1) whose values should be compared lexicographically (e.g. “maximize U+V, break ties by maximizing U”). This lets you reach some points on the edges of flat parts.
3) Allow the merging process to choose randomly which function of type (2) to give to the merged AI. This lets you reach the rest of the points on flat parts.
That’s a bit complicated, but I don’t think there’s a simpler way.
I don’t see why 2 is necessary given that any point on the Pareto frontier is a mixture of pointy points (intuition for this: any point on the face of a polyhedron is a mixture of that face’s corners). In any case, I agree with the basic mathematical point that you can get any Pareto optimal mixture of outcomes by mixing between non-negative linear combinations of utility functions.
Well, I was imagining a Pareto frontier that changes smoothly from flat to curved. Then we can’t quite get a pointy point exactly on the edge of the flat part. That’s what 2 is for, it gives us some of these points (though not all). But I guess that doesn’t matter if things are finite enough.
Ok, that seems right.