The problem is that it’s a utility maximiser. If the ontology crises causes the FAI’s goals to slide a bit in the wrong direction, it may end up optimising us out of existence (even if “happy humans with worthwhile and exciting lives” is still high in its preference ordering, it might not be at the top).
This is a uniform problem among all AIs. Avoiding it is very hard. That is why such a thing as the discipline of Friendly AI exists in the first place. You do, in fact, have to specify the preference ordering sufficiently well and keep it sufficiently stable.
Stepping down from maximization is also necessary just because actual maximization is undoable, but then that also has to be kept stable (satisficers may become maximizers, etc.) and if there’s something above eudaimonia in its preference ordering it might not take very much ‘work’ to bring it into existence.
The problem is that it’s a utility maximiser. If the ontology crises causes the FAI’s goals to slide a bit in the wrong direction, it may end up optimising us out of existence (even if “happy humans with worthwhile and exciting lives” is still high in its preference ordering, it might not be at the top).
This is a uniform problem among all AIs. Avoiding it is very hard. That is why such a thing as the discipline of Friendly AI exists in the first place. You do, in fact, have to specify the preference ordering sufficiently well and keep it sufficiently stable.
Stepping down from maximization is also necessary just because actual maximization is undoable, but then that also has to be kept stable (satisficers may become maximizers, etc.) and if there’s something above eudaimonia in its preference ordering it might not take very much ‘work’ to bring it into existence.