I find “false positive” and “false negative” also a bit confusing, albeit less so than “type I” and “type II” errors. Perhaps because of a programming background, I usually interpret ‘false’ and ‘negative’ (and ‘0’) as the same thing. So is a ‘false positive’ something that is false but is mistaken as positive, or something that is positive (true), but that is mistaken as false (negative)? In other words, does ‘false’ apply to the postiveness (it is actually negative, but classified as positive), to being classified as positive (it is actually positive, but classified as positive)?
Perhaps we should call false positives “spurious” and false negatives “missed”.
The entity providing the goals for the AI wouldn’t have to be a human, it might instead be a corporation. A reasonable goal for such an AI might be to ‘maximize shareholder value’. The shareholders are not humans either, and what they value is only money.