I’m completely confused now. I thought CEV was right by definition? If “80% of humanity turns out to be selfish bastards” then it will extrapolate on that.
No, CEV is right by definition. When CEV is used as shorthand for “the coherent extrapolated volitions of all of humanity” as is the case there then it is quite probably not right at all. Because many humans, to put it extremely politely, have preferences that are distinctly different to what I would call ‘right’.
If we start to cherry pick certain outcomes according to our current perception, why run CEV at all?
Yes, that would be pointless, it would be far better to compare the outcomes to CEV<group_I_identify_with_sufficiently> (then just use the latter!) The purpose of doing CEV at all is for signalling and cooperation.
Before or after extrapolation? If the former then why does that matter, if the latter then how do you know?
Former in as much as it allows inferences about the latter. I don’t need to know with any particular confidence for the purposes of the point. The point was to illustrate possible (and overwhelmingly obvious) failure modes.
Hoping that CEV is desirable rather than outright unfriendly isn’t a particularly good reason to consider it. It is going to result in outcomes that are worse from the perspective of whoever is running the GAI than CEV and CEV.
The purpose of doing CEV at all is for signalling and cooperation (or, possibly, outright confusion).
No, CEV is right by definition. When CEV is used as shorthand for “the coherent extrapolated volitions of all of humanity” as is the case there then it is quite probably not right at all. Because many humans, to put it extremely politely, have preferences that are distinctly different to what I would call ‘right’.
Yes, that would be pointless, it would be far better to compare the outcomes to CEV<group_I_identify_with_sufficiently> (then just use the latter!) The purpose of doing CEV at all is for signalling and cooperation.
Before or after extrapolation? If the former then why does that matter, if the latter then how do you know?
Former in as much as it allows inferences about the latter. I don’t need to know with any particular confidence for the purposes of the point. The point was to illustrate possible (and overwhelmingly obvious) failure modes.
Hoping that CEV is desirable rather than outright unfriendly isn’t a particularly good reason to consider it. It is going to result in outcomes that are worse from the perspective of whoever is running the GAI than CEV and CEV.
The purpose of doing CEV at all is for signalling and cooperation (or, possibly, outright confusion).
Do you mean it is simply an SIAI marketing strategy and that it is not what they are actually going to do?
Signalling and cooperation can include actual behavior.