I’m skeptical that many of the problems with aggregation don’t both apply to actual individual human values once extrapolated, and generalize to AIs with closely related values, but I’d need to lay out the case for that more clearly. (I did discuss the difficulty of cooperation even given compatible goals a bit in this paper, but it’s nowhere near complete in addressing this issue.)
I’m skeptical that many of the problems with aggregation don’t both apply to actual individual human values once extrapolated, and generalize to AIs with closely related values, but I’d need to lay out the case for that more clearly. (I did discuss the difficulty of cooperation even given compatible goals a bit in this paper, but it’s nowhere near complete in addressing this issue.)
It’s worth you write up your point and post it—that tends to clarify the issue, for yourself as well as for others.