“humans don’t have actual preferences so the AI is just going to try to learn something adequate.”
Try something like: humans don’t have actual consistent preferences, so the AI is going to try and find a good approximation that covers all the contradictions and uncertainties in human preferences.
Try something like: humans don’t have actual consistent preferences, so the AI is going to try and find a good approximation that covers all the contradictions and uncertainties in human preferences.