My objection isn’t about defining niceness to the people programming the AI. My objection is about defining niceness (actually, defining “extending niceness” which isn’t the same) to the people determining whether the answer is correct. If we don’t know what it means to “extend niceness”, then we can’t know that any given answer is “extending niceness”, which means we have no way to know whether it’s actually an answer.
I don’t think that’s actually the case—I think we can extend niceness without knowing what that means in this sense. Working on a potential solution currently...
Well, you can do it without knowing whether you’re doing it, but how would you know if you’ve ever succeeded?
Furthermore, knowing that something is “extending niceness” is not the same as knowing if something is niceness. Let’s say you know what niceness is. What counts as an extension?
My objection isn’t about defining niceness to the people programming the AI. My objection is about defining niceness (actually, defining “extending niceness” which isn’t the same) to the people determining whether the answer is correct. If we don’t know what it means to “extend niceness”, then we can’t know that any given answer is “extending niceness”, which means we have no way to know whether it’s actually an answer.
I don’t think that’s actually the case—I think we can extend niceness without knowing what that means in this sense. Working on a potential solution currently...
Well, you can do it without knowing whether you’re doing it, but how would you know if you’ve ever succeeded?
Furthermore, knowing that something is “extending niceness” is not the same as knowing if something is niceness. Let’s say you know what niceness is. What counts as an extension?