The hypothesis that should interest an AI are not necessarily limited to those it can compute but to those it could test. A hypothesis is useless if it does not tell us something about how the world looks when it’s true as opposed to when it’s false. So if there is a way for the AI to interact with the world such that it expects different probabilities of outcomes depending on whether the (possibly uncomputable) hypothesis holds or not then it is something worth having a symbol for, even if the exact dynamics of this universe cannot be computed.
Let’s consider the case of our AI encountering a Turing Oracle. Two possible hypotheses of the AI could be A = This is in fact a Turing Oracle and for every program P it will output either the time until halting or 0 if no halting, and B = This is not a Turing Oracle but some computable machine Q. The AI could feed the supposed oracle a number of programs and if it was told any of them would halt it could try to run them for the specified number of steps to see if they did indeed halt. After each program had halted it would have to increase it’s probability that this was in fact a Turing Oracle using Bayes’ Theorem and estimates of the probabilities of guessing this right, or computationally deriving these numbers. If it did this for long enough and this was in fact a Turing Oracle it would gain higher and higher certainty of this fact.
What is it that the AI is doing? We can view the whole above process as a program which given one of a limited set of experimental outcomes outputs the probability that this experimental outcome would be the real one if H held. In the case of the Turing Oracle above the set of outcomes is the set of pairs (P,n) where P is a program and n a positive integer, and the program will output 1 if P halts after n steps and 0 otherwise. I think this captures in full generality all possibilities a computable agent would be able to recognise.
What if the AI later on gains some extra computational capacity which makes it non-computable? Say for example that it finds a Turing Oracle in like in the above example and integrates it into its main processor. But this is essentially everything that could happen: for the AI to become uncomputable, it would have to integrate an uncomputable physical process into its own processing. But for the AI to know it was actually uncomputable and not only incorporating the results of some computational process it didn’t recognise it would have to preform above test. So when it now preforms some uncomputable test on a new process we can see this simply as the composite of the tests of the original and the new process viewing all the message passing between the uncomputable processes as a part of the experimental setup rather than internal computation.
The hypothesis that should interest an AI are not necessarily limited to those it can compute but to those it could test. A hypothesis is useless if it does not tell us something about how the world looks when it’s true as opposed to when it’s false. So if there is a way for the AI to interact with the world such that it expects different probabilities of outcomes depending on whether the (possibly uncomputable) hypothesis holds or not then it is something worth having a symbol for, even if the exact dynamics of this universe cannot be computed.
Let’s consider the case of our AI encountering a Turing Oracle. Two possible hypotheses of the AI could be A = This is in fact a Turing Oracle and for every program P it will output either the time until halting or 0 if no halting, and B = This is not a Turing Oracle but some computable machine Q. The AI could feed the supposed oracle a number of programs and if it was told any of them would halt it could try to run them for the specified number of steps to see if they did indeed halt. After each program had halted it would have to increase it’s probability that this was in fact a Turing Oracle using Bayes’ Theorem and estimates of the probabilities of guessing this right, or computationally deriving these numbers. If it did this for long enough and this was in fact a Turing Oracle it would gain higher and higher certainty of this fact.
What is it that the AI is doing? We can view the whole above process as a program which given one of a limited set of experimental outcomes outputs the probability that this experimental outcome would be the real one if H held. In the case of the Turing Oracle above the set of outcomes is the set of pairs (P,n) where P is a program and n a positive integer, and the program will output 1 if P halts after n steps and 0 otherwise. I think this captures in full generality all possibilities a computable agent would be able to recognise.
What if the AI later on gains some extra computational capacity which makes it non-computable? Say for example that it finds a Turing Oracle in like in the above example and integrates it into its main processor. But this is essentially everything that could happen: for the AI to become uncomputable, it would have to integrate an uncomputable physical process into its own processing. But for the AI to know it was actually uncomputable and not only incorporating the results of some computational process it didn’t recognise it would have to preform above test. So when it now preforms some uncomputable test on a new process we can see this simply as the composite of the tests of the original and the new process viewing all the message passing between the uncomputable processes as a part of the experimental setup rather than internal computation.