It seems fair to worry about smuggling hypercomputation into optimal agents via assuming global optimization (due to the intractability of the halting problem). But there are plenty of domains where the global optima is just the asymptotic limit of local search.
For example suppose I have to predict the outcome of a biased coin flip, observing a bunch of actual coin flips and then computing p(heads) based off of that converges just fine to the right answer.
Even in Turing-complete domains, search sort of just works even though the Halting problem tells us proving we’ve found an optimal solution may be impossible.
It seems fair to worry about smuggling hypercomputation into optimal agents via assuming global optimization (due to the intractability of the halting problem). But there are plenty of domains where the global optima is just the asymptotic limit of local search.
For example suppose I have to predict the outcome of a biased coin flip, observing a bunch of actual coin flips and then computing p(heads) based off of that converges just fine to the right answer.
Even in Turing-complete domains, search sort of just works even though the Halting problem tells us proving we’ve found an optimal solution may be impossible.