I think a useful distinction can be made between different “Question-Answerers that cannot answer the ‘what ought we to do’ question”. Although all answers from any such system are unsafe, they can be unsafe in different ways: (1) unsafe in the sense in which any new human knowledge is unsafe, or (2) unsafe in specific UFAI-related risk sense, which is much more unsafe. A system that gives the first kind of answers can be thought of as a domain-specific intelligence enhancer for people.
I think a useful distinction can be made between different “Question-Answerers that cannot answer the ‘what ought we to do’ question”. Although all answers from any such system are unsafe, they can be unsafe in different ways:
(1) unsafe in the sense in which any new human knowledge is unsafe, or
(2) unsafe in specific UFAI-related risk sense, which is much more unsafe.
A system that gives the first kind of answers can be thought of as a domain-specific intelligence enhancer for people.