I understand CEV. What I don’t understand is why the programmers would ask the AI for humanity’s CEV, rather than just their own CEV.
The only (sane) reason is for signalling—it’s hard to create FAI without someone else stopping you. Given a choice, however, CEV is strictly superior. If you actually do want to have FAI then FAI will be equivalent to it. But if you just think you want FAI but it turns out that, for example, FAI gets dominated by jerks in a way you didn’t expect then FAI will end up better than FAI… even from a purely altruistic perspective.
The only (sane) reason is for signalling—it’s hard to create FAI without someone else stopping you. Given a choice, however, CEV is strictly superior. If you actually do want to have FAI then FAI will be equivalent to it. But if you just think you want FAI but it turns out that, for example, FAI gets dominated by jerks in a way you didn’t expect then FAI will end up better than FAI… even from a purely altruistic perspective.