Even if the process of learning P_H is doing the work to turn it into a coherent probability distribution (removing irrationality and making things well-defined), the end result may find situations which the AI finds itself in too complex to be conceived.
I had trouble parsing the end of this sentence. Is the idea that the AI might get into situations that are too complex for the humans to understand?
I had trouble parsing the end of this sentence. Is the idea that the AI might get into situations that are too complex for the humans to understand?
Yeah. I’ve edited it a bit for clarity.