I think that the concept of a reflexive oracle would be really useful to bring into this discussion if it hasn’t already been brought up. From the abstract:
In this paper, we introduce a “reflective” type of oracle, which is able to answer questions about the outputs of oracle machines with access to the same oracle. These oracles avoid diagonalization by answering some queries randomly. We show that machines with access to a reflective oracle can be used to define rational agents using causal decision theory. These agents model their environment as a probabilistic oracle machine, which may contain other agents as a non-distinguished part.
I think that the concept of a reflexive oracle would be really useful to bring into this discussion if it hasn’t already been brought up. From the abstract: