2b/2c. I think I would say that we should want a tyranny of the present to the extent that is in our values upon reflection. If, for example, Rome still existed and took over the world, their CEV should depend on their ethics and population. I think it would still be a very good utopia, but it may also have things we dislike.
Other considerations, like nearby Everett branches… well they don’t exist in this branch? I would endorse game theoretical cooperation with them, but I’m skeptical of any more automatic cooperation than what we already have. That is, this sort of fairness is a part of our values, and CEV (if not adversarially hacked) should represent those already?
I don’t think this would end up in a tyranny anything like the usual form of the word if we’re actually implementing CEV. We have values for people being able to change and adjust over time, and so those are in the CEV.
There may very well be limits to how far we want humanity to change in general, but that’s perfectly allowed to be in our values. Like, as a specific example, some have said that they think global status games will be vastly important in the far future and thus a zero-sum resource. I find it decently likely that an AGI implementing CEV would discourage such, because humans wouldn’t endorse it on reflection, even if it is a plausible default outcome.
Like, essentially my view is: Optimize our-branches’ humanity’s values as hard as possible, this contains desires for other people’s values to be satisfied, and thus they’re represented. Other forms of fairness to things we aren’t completely a fans of can be bargained for (locally, or acausally between branches/whatever).
So that’s my argument against the tyranny and Everett branches part. I’m less skeptical of considering whether to include the recently dead, but I also don’t have a great theory of how to weight them. Those about to be born wouldn’t have a notable effect on CEV, I’d believe.
The option you suggest in #3 is nice, though I think it runs some risks of being dominated or notably influenced by “humans in other very odd branches”, and so we’re outweighed by them despite them not locally existing. I think it is less that you want a human predicate, and more of a “human who has values compatible with this local branch”. This is part of why I advocate just bargaining between branches: if the humans in an AGI-made New Rome want us to instantiate their constructed friendly/restricted AGI Gods locally to proselytize, they can trade for it rather than that faction being automatically divvied out a star by our AGI’s CEV.
“Human who has values compatible with this local branch” feels weak as a definition, arbitrary, but I’m not sure we can do better than that. I imagine we’d even have weightings, because we likely legitimately value baby’s in special ways that don’t entail maxing out reward centers or boosting them to megaminds soon after birth, we have preferences about that. Then of course there’s minds that are sortof humanish, which is why you’d have a weighting.
(This is kinda rambly, but I do think a lot of this can be avoided with just plain CEV because I think most people on reflection would end up with “reevaluate whether the deal was fair with reflection and then adjust the deal and reference class based on that”.)
2b/2c. I think I would say that we should want a tyranny of the present to the extent that is in our values upon reflection. If, for example, Rome still existed and took over the world, their CEV should depend on their ethics and population. I think it would still be a very good utopia, but it may also have things we dislike.
Other considerations, like nearby Everett branches… well they don’t exist in this branch? I would endorse game theoretical cooperation with them, but I’m skeptical of any more automatic cooperation than what we already have. That is, this sort of fairness is a part of our values, and CEV (if not adversarially hacked) should represent those already? I don’t think this would end up in a tyranny anything like the usual form of the word if we’re actually implementing CEV. We have values for people being able to change and adjust over time, and so those are in the CEV. There may very well be limits to how far we want humanity to change in general, but that’s perfectly allowed to be in our values. Like, as a specific example, some have said that they think global status games will be vastly important in the far future and thus a zero-sum resource. I find it decently likely that an AGI implementing CEV would discourage such, because humans wouldn’t endorse it on reflection, even if it is a plausible default outcome.
Like, essentially my view is: Optimize our-branches’ humanity’s values as hard as possible, this contains desires for other people’s values to be satisfied, and thus they’re represented. Other forms of fairness to things we aren’t completely a fans of can be bargained for (locally, or acausally between branches/whatever).
So that’s my argument against the tyranny and Everett branches part. I’m less skeptical of considering whether to include the recently dead, but I also don’t have a great theory of how to weight them. Those about to be born wouldn’t have a notable effect on CEV, I’d believe.
The option you suggest in #3 is nice, though I think it runs some risks of being dominated or notably influenced by “humans in other very odd branches”, and so we’re outweighed by them despite them not locally existing. I think it is less that you want a human predicate, and more of a “human who has values compatible with this local branch”. This is part of why I advocate just bargaining between branches: if the humans in an AGI-made New Rome want us to instantiate their constructed friendly/restricted AGI Gods locally to proselytize, they can trade for it rather than that faction being automatically divvied out a star by our AGI’s CEV.
“Human who has values compatible with this local branch” feels weak as a definition, arbitrary, but I’m not sure we can do better than that. I imagine we’d even have weightings, because we likely legitimately value baby’s in special ways that don’t entail maxing out reward centers or boosting them to megaminds soon after birth, we have preferences about that. Then of course there’s minds that are sortof humanish, which is why you’d have a weighting.
(This is kinda rambly, but I do think a lot of this can be avoided with just plain CEV because I think most people on reflection would end up with “reevaluate whether the deal was fair with reflection and then adjust the deal and reference class based on that”.)