I wonder what this means in the context of FAI and what EY wrote on coherent extrapolated volition of humanity. That might be read to imply that the values of all humans count equally.
That’s an interesting question and one where I am wary of EY’s expressed beliefs on the subject. Eliezer has only ever talked about CEV as if it is something that applies to “humanity”, without specifying the aggregation mechanism but with the implication that once Extrapolation is applied the humans would in general have acceptable values. But this opens up enormous scope for the “What if most people are Dicks?” problem as well as the perverse incentive that people who breed more now can expect CEV to be more in favour of them. I’m not entirely confident that CEV wouldn’t result in a genocide or two and at best there are going to be more mild dystopic outcomes.
Optimising for CEV may sound egalitarian but on close inspection it is inherently less egalitarian than optimising for CEV. (This example should not be interpreted as an advocacy of creating an FAI with that value system.)
It could. But it needn’t. Just like we are not obliged to maximise our inclusive genetic fitness just because that was the incentives of our genes as we were evolving we need not implement a CEV based on social status just because currently human influence follows that pattern. There are two parts to this consideration: it applies both to determining the aggregation mechanism used by the CEV procedure itself and also, for a given aggregation procedure and given target CEV may compute a system based somewhat on status. This again makes us look at which group CEV is applied to. CEV will likely emphasise status to a different degree than CEV and so to the extent that those are different a group of nerds can be expected to be wary of CEV.
That’s an interesting question and one where I am wary of EY’s expressed beliefs on the subject. Eliezer has only ever talked about CEV as if it is something that applies to “humanity”, without specifying the aggregation mechanism but with the implication that once Extrapolation is applied the humans would in general have acceptable values. But this opens up enormous scope for the “What if most people are Dicks?” problem as well as the perverse incentive that people who breed more now can expect CEV to be more in favour of them. I’m not entirely confident that CEV wouldn’t result in a genocide or two and at best there are going to be more mild dystopic outcomes.
Optimising for CEV may sound egalitarian but on close inspection it is inherently less egalitarian than optimising for CEV. (This example should not be interpreted as an advocacy of creating an FAI with that value system.)
It could. But it needn’t. Just like we are not obliged to maximise our inclusive genetic fitness just because that was the incentives of our genes as we were evolving we need not implement a CEV based on social status just because currently human influence follows that pattern. There are two parts to this consideration: it applies both to determining the aggregation mechanism used by the CEV procedure itself and also, for a given aggregation procedure and given target CEV may compute a system based somewhat on status. This again makes us look at which group CEV is applied to. CEV will likely emphasise status to a different degree than CEV and so to the extent that those are different a group of nerds can be expected to be wary of CEV.