Thinking about this some more, all high-bandwidth oracles (counterfactual or not) risk receiving messages crafted by future UFAI to take over the present.
Note that in the case of counterfactual oracle, this depends on UFAI “correctly” solving counterfactual mugging (i.e., the UFAI has to decide to pay some cost in its own world to take over a counterfactual world where the erasure event didn’t occur).
So such oracles are dangerous and shouldn’t be used.
This seems too categorical. Depending on the probabilities of various conditions, using such oracles might still be the best option in some circumstances.
Note that in the case of counterfactual oracle, this depends on UFAI “correctly” solving counterfactual mugging (i.e., the UFAI has to decide to pay some cost in its own world to take over a counterfactual world where the erasure event didn’t occur).
This seems too categorical. Depending on the probabilities of various conditions, using such oracles might still be the best option in some circumstances.
Some thoughts on that idea: https://www.lesswrong.com/posts/6WbLRLdmTL4JxxvCq/analysing-dangerous-messages-from-future-ufai-via-oracles
Yeah, agreed on both points.