I am more concerned that AI utility functions will serve a narrow group of humans.
I just talked about that in this comment, but it’s more relevant to this thread, so I’ll copy it here:
One could imagine an organization conspiring to create AGI that will optimize for the organization’s collective preferences rather than humanity’s collective preferences, but this won’t happen because: 1. No one will throw a fit and defect from an FAI project because they won’t be getting special treatment, but people will throw a fit if they perceive unfairness, so Friendly-to-humanity-AI will be a lot easier to get funding and community support for than friendly-to-exclusive-club-AI. 2. Our near mode reasoning cannot comprehend how much better a personalized AGI slave would be over FAI for us personally, so people will make that sort of decision in far mode, where idealistic values can outweigh greediness.
Finally, even if some exclusive club did somehow create an AGI that was friendly to them in particular, it wouldn’t be that bad. Even if people don’t care about each other very much, we do at least a little bit. Let’s say that an AGI optimizing an exclusive club’s CEV devotes .001% of its resources to things the rest of humanity would care about, and the rest to the things that just the club cares about. This is only worse than FAI by a factor of 10^5, which is negligible compared to the difference between FAI and UFAI.
I just talked about that in this comment, but it’s more relevant to this thread, so I’ll copy it here:
One could imagine an organization conspiring to create AGI that will optimize for the organization’s collective preferences rather than humanity’s collective preferences, but this won’t happen because: 1. No one will throw a fit and defect from an FAI project because they won’t be getting special treatment, but people will throw a fit if they perceive unfairness, so Friendly-to-humanity-AI will be a lot easier to get funding and community support for than friendly-to-exclusive-club-AI. 2. Our near mode reasoning cannot comprehend how much better a personalized AGI slave would be over FAI for us personally, so people will make that sort of decision in far mode, where idealistic values can outweigh greediness.
Finally, even if some exclusive club did somehow create an AGI that was friendly to them in particular, it wouldn’t be that bad. Even if people don’t care about each other very much, we do at least a little bit. Let’s say that an AGI optimizing an exclusive club’s CEV devotes .001% of its resources to things the rest of humanity would care about, and the rest to the things that just the club cares about. This is only worse than FAI by a factor of 10^5, which is negligible compared to the difference between FAI and UFAI.
Substitute “company” for “AI project” and look at what happens to the first argument.
Good point.