The problem is that the question “what would be the consequences” is too general to be answered exhaustively. We should at least have an idea about the general characteristics of the risk to ask more specifically; the Oracle doesn’t know what consequences are important for us unless it already comprehends human values an is thus already “friendly”.
But the oracle will be able to predict these consequences, and we’ll probably get into the habit of checking these.
The problem is that the question “what would be the consequences” is too general to be answered exhaustively. We should at least have an idea about the general characteristics of the risk to ask more specifically; the Oracle doesn’t know what consequences are important for us unless it already comprehends human values an is thus already “friendly”.