Eliezer thinks that what is inside the black box inexorably kills you when the black box is large enough, like how humans are cognitive daemons of natural selection (the outer optimization process operating on the black box of genes accidentally constructed a (sapient) inner optimization process inside the black box) and this is chicken-and-egg unavoidable whenever the black box is powerful enough to do something like predict complicated human judgments, since in this case the outer optimization was automatically powerful enough to consider and select among multiple hypotheses the size of humans, and the inner process is automatically as powerful as human intelligence.
It might be worth pointing out that evolution seems to be doing something different from the oracle in the Original Post.
Evolution:
building something piece by piece, and testing those pieces (in reality), and then building things from those
Oracle:
Wandering the space, adrift from that connection to reality, w/out the checking throughout.
It might be worth pointing out that evolution seems to be doing something different from the oracle in the Original Post.
Evolution:
building something piece by piece, and testing those pieces (in reality), and then building things from those
Oracle:
Wandering the space, adrift from that connection to reality, w/out the checking throughout.