Human rationality and FAI design are both about producing real-world approximations to some mathematical ideal of what we mean by “rationality”. Puzzles in anthropics and decision theory suggest that our mathematical idealization of rationality is wrong or at least incomplete. Some people want to get this stuff sorted out so that we can make sure we’re not approximating the wrong thing.
Human rationality and FAI design are both about producing real-world approximations to some mathematical ideal of what we mean by “rationality”. Puzzles in anthropics and decision theory suggest that our mathematical idealization of rationality is wrong or at least incomplete. Some people want to get this stuff sorted out so that we can make sure we’re not approximating the wrong thing.