Why must the oracle continue to believe it’s messages weren’t read?
In the example you give I’m guessing the reason you’d want an oracle to believe with cold certainty that it’s messages won’t be read is to avoid it trying to influence the world with them but that doesn’t require that it continue to believe that later. As long as when it’s composing and ouputing the message it believes solidly that it will never be read and nothing can move that belief from zero then that’s fine. That does not preclude it being perfectly accepting that it’s past messages were in fact read and basing it’s beliefs about the world on that. That knowledge after all cannot shift the belief that this next message will never, ever ever be read unlike all the others.
Of course that brings up the question of why an oracle would even be designed as a goal based AI with any kind of utility function. Square peg, round hole and all that.
Why must the oracle continue to believe it’s messages weren’t read?
In the example you give I’m guessing the reason you’d want an oracle to believe with cold certainty that it’s messages won’t be read is to avoid it trying to influence the world with them but that doesn’t require that it continue to believe that later. As long as when it’s composing and ouputing the message it believes solidly that it will never be read and nothing can move that belief from zero then that’s fine. That does not preclude it being perfectly accepting that it’s past messages were in fact read and basing it’s beliefs about the world on that. That knowledge after all cannot shift the belief that this next message will never, ever ever be read unlike all the others.
Of course that brings up the question of why an oracle would even be designed as a goal based AI with any kind of utility function. Square peg, round hole and all that.
For Oracles, you can reset them after they’ve sent out their message. For autonomous AIs, this is more tricky.