That FAI is good for you is a property of the term “FAI”.
“Friendly to their Creator AI”, choose an acronym. Perhaps FAI. Across the multiverse most civilizations that engage in successful AI efforts will produce an AI that is not friendly to me. AIs that are actually FAIs (which include by definition my own survival) are negligible.
Negligible, non-negligible, that’s what the word means. It talks about specifically working for your preference, because of what AI values and not because it needs to do so for trade. FAI could be impossible, for example, that doesn’t change the concept. BabyEater’s AI could be an UFAI, or it could be a FAI, depending on how well it serves your preference. It could turn out to be a FAI, if the sympathy aspect of their preference is strong enough to dole you a fair part of the world, more than you own by pure game-theoretic control.
FAI doesn’t imply full control given to your preference (for example, here on Earth we have many people with at least somewhat different preferences, and all control likely won’t be given to any single person). The term distinguishes AIs that optimize for you because of their own preference (and thus generate powerful control in the mathematical universe for your values, to a much more significant extent than you can do yourself), from AIs that optimize for you because of control pressure (in other terms, trade opportunity) from another AI (which is the case for “UFAI”).
(I’m trying to factor the discussion into the more independent topics to not lose track of the structure of the argument.)
Negligible, non-negligible, that’s what the word means. It talks about specifically working for your preference, because of what AI values and not because it needs to do so for trade. FAI could be impossible, for example, that doesn’t change the concept. BabyEater’s AI could be an UFAI, or it could be a FAI, depending on how well it serves your preference. It could turn out to be a FAI, if the sympathy aspect of their preference is strong enough to dole you a fair part of the world, more than you own by pure game-theoretic control.
FAI doesn’t imply full control given to your preference (for example, here on Earth we have many people with at least somewhat different preferences, and all control likely won’t be given to any single person). The term distinguishes AIs that optimize for you because of their own preference (and thus generate powerful control in the mathematical universe for your values, to a much more significant extent than you can do yourself), from AIs that optimize for you because of control pressure (in other terms, trade opportunity) from another AI (which is the case for “UFAI”).
(I’m trying to factor the discussion into the more independent topics to not lose track of the structure of the argument.)