Even if the oracle is myopic, there are still potential failure modes of the form “start outputting answer; [wait long enough for Oracle 2 to be built and take over the world]; finish outputting answer”, no?
(I suppose you can partially counter this by ensuring outputs are atomic, but relying on no-one inspecting a partial output to prevent an apocalypse seems failure-prone. Also, given that I thought of this failure mode immediately, I’d be worried that there are other more subtle failure modes still lurking.)
Yeah this seems right! :) I am assuming no one ever inspects a partial ouput. This does seem risky, and it’s likely there are a bunch more possible failure modes here.
(Btw, thanks for this exchange; just wanted to note that it was valuable for me and made me notice some mistakes in how I was thinking about oracles)
Even if the oracle is myopic, there are still potential failure modes of the form “start outputting answer; [wait long enough for Oracle 2 to be built and take over the world]; finish outputting answer”, no?
(I suppose you can partially counter this by ensuring outputs are atomic, but relying on no-one inspecting a partial output to prevent an apocalypse seems failure-prone. Also, given that I thought of this failure mode immediately, I’d be worried that there are other more subtle failure modes still lurking.)
Yeah this seems right! :) I am assuming no one ever inspects a partial ouput. This does seem risky, and it’s likely there are a bunch more possible failure modes here.
(Btw, thanks for this exchange; just wanted to note that it was valuable for me and made me notice some mistakes in how I was thinking about oracles)