In that reply he says the the meta-system would need to make an assumption that the meta system works, which isn’t much different from requiring that A1 proves A2 consistent before accepting it, which he mentioned in the first post briefly. That requirement of consistency is not what he originally asked for: that A1 proves that if A2 accepts P, then P, which Wei Dai does solve for the case of equivalent A1 and A2. It’s true he doesn’t show that such a modification is necessarily an improvement in all environments, but if Schmidhuber’s usefulness means “This modification has expected utility greater than or equal to all other modifications including not modifying”, as Eliezer used in his talk, then we don’t require that, any more than we would require a Bayesian to always make true predictions. The proof of usefulness, like any full expected utility calculation, would require considering the probabilities and values of consequences as they follow from the AIs actions across possible worlds consistent with the AIs observations. That is a huge computation, but I don’t see how it’s logically impossible.
However, getting back to the separate criterion of consistency, why? Where does consist(A2) ever enter into a calculation? What do we gain by it? If A2 is inconsistent, I can see how that would usually limit the expected utility of becoming A2, but that’s just not the same. “Good” shouldn’t be conditional on consistency, it’s just expectation of valuable consequences.
if Schmidhuber’s usefulness means “This modification has expected utility greater than or equal to all other modifications including not modifying”
Schmidhuber’s Goedel machine can handle decision rules other than the expected value, but apart from that, this is correct.
That is a huge computation, but I don’t see how it’s logically impossible.
It is required to compute its own expected actions. The naive way of doing this collapses into infinite recursion. We can imagine more sophisticated ways that usually work, but usually is not a proof; the Goedel machine only takes expectations over possible states of the environment, not over possible truth-values of mathematical statements. It is unclear how one could preform estimates over the latter; if you have an algorithm that can estimate the probability of, say, Goldbach’s conjecture, you have made quite a significant amount of progress in AGI.
Where does consist(A2) ever enter into a calculation? What do we gain by it?
Without either a proof of consist(A2) or some way of estimating the probability of consist(A2), we will not be able to calculate expected utilities.
In that reply he says the the meta-system would need to make an assumption that the meta system works, which isn’t much different from requiring that A1 proves A2 consistent before accepting it, which he mentioned in the first post briefly. That requirement of consistency is not what he originally asked for: that A1 proves that if A2 accepts P, then P, which Wei Dai does solve for the case of equivalent A1 and A2. It’s true he doesn’t show that such a modification is necessarily an improvement in all environments, but if Schmidhuber’s usefulness means “This modification has expected utility greater than or equal to all other modifications including not modifying”, as Eliezer used in his talk, then we don’t require that, any more than we would require a Bayesian to always make true predictions. The proof of usefulness, like any full expected utility calculation, would require considering the probabilities and values of consequences as they follow from the AIs actions across possible worlds consistent with the AIs observations. That is a huge computation, but I don’t see how it’s logically impossible.
However, getting back to the separate criterion of consistency, why? Where does consist(A2) ever enter into a calculation? What do we gain by it? If A2 is inconsistent, I can see how that would usually limit the expected utility of becoming A2, but that’s just not the same. “Good” shouldn’t be conditional on consistency, it’s just expectation of valuable consequences.
Schmidhuber’s Goedel machine can handle decision rules other than the expected value, but apart from that, this is correct.
It is required to compute its own expected actions. The naive way of doing this collapses into infinite recursion. We can imagine more sophisticated ways that usually work, but usually is not a proof; the Goedel machine only takes expectations over possible states of the environment, not over possible truth-values of mathematical statements. It is unclear how one could preform estimates over the latter; if you have an algorithm that can estimate the probability of, say, Goldbach’s conjecture, you have made quite a significant amount of progress in AGI.
Without either a proof of consist(A2) or some way of estimating the probability of consist(A2), we will not be able to calculate expected utilities.