Can you find any reason why this is not a counterexample?
Yes—the bit that violates de Blanc’s conditions is where you say “I choose the case where the set of possible worlds consistent with observation is …”. In fact, you have to assign positive probability to every computable function whose values at points in the set I agree with known values. (Every member of S_I must be a hypothesis.)
Personally, I think it’s very easy to ‘oversell’ de Blanc’s result. It’s an artifact of the weird constraint of being forced, for every n, to be able to compute a number representing “the probability of phi_n, if it turns out to be a meaningful hypothesis” even without knowing whether phi_n is a meaningful hypothesis. (E.g. “the first hypothesis whose utility (at k) is greater than [this thing which may not halt].”)
Yes—the bit that violates de Blanc’s conditions is where you say “I choose the case where the set of possible worlds consistent with observation is …”. In fact, you have to assign positive probability to every computable function whose values at points in the set I agree with known values. (Every member of S_I must be a hypothesis.)
What I am doing is choosing the members of S_I. So every member of S_I is a hypothesis.
If the theorem means what it says that it means—that “a computable utility function will have convergent expected utilities iff that function is bounded”—then this is true in any set of possible worlds, meaning any consistent choice of h and S_I. I have chosen one.
The only reason I would not have a free choice of S_I, is if I had already nailed down S_I in some other way (say, by choosing the set I), or if I were using a set S_I for which there was no possible set of observations I such that S_I was equal to the set of worlds consistent with I, or because each application of the theorem is meant to apply to all possible choices of S_I
The last of these would mean that the theorem is actually proving that, for any unbounded utility function U, there exists some set of possible worlds for which it does not converge. That would be a still-interesting, but much weaker result.
You’re not allowed to—de Blanc has already supplied a definition of S_I. One must either adopt his definition or be talking about something other than his result.
He supplied a definition, not a particular set. I am using his definition, and providing one possible instantiation that is compatible with that definition.
The set S is the set of all total recursive functions. This is set in stone for all time. Therefore, there is only one way that S_I can refer to different things:
Our stock of observational data may be different. In other words, the set I and the values of h(i) for i in I may be different.
But regardless of I and the values of h(i), it’s easy to see that one cannot restrict S_I in the way you’re attempting to do.
In fact, one can easily see that S_I = the set of functions of the form “if x is in I then h(x), otherwise f(x)” where f is an arbitrary recursive function.
That is, the whole “I” business is completely pointless, except (presumably) to help the reader assure themselves that the result does apply to AIXI.
So, we can rescue our utility function from the theorem if we are allowed to assign zero probability to arbitrary hypotheses that have no plausibility other than that they have not been absolutely ruled out. Such as the hypothesis that the laws of physics are valid at all times except on October 21, 2011.
Being allowed to do this would make the counterexample work.
Yes—the bit that violates de Blanc’s conditions is where you say “I choose the case where the set of possible worlds consistent with observation is …”. In fact, you have to assign positive probability to every computable function whose values at points in the set I agree with known values. (Every member of S_I must be a hypothesis.)
Personally, I think it’s very easy to ‘oversell’ de Blanc’s result. It’s an artifact of the weird constraint of being forced, for every n, to be able to compute a number representing “the probability of phi_n, if it turns out to be a meaningful hypothesis” even without knowing whether phi_n is a meaningful hypothesis. (E.g. “the first hypothesis whose utility (at k) is greater than [this thing which may not halt].”)
What I am doing is choosing the members of S_I. So every member of S_I is a hypothesis.
If the theorem means what it says that it means—that “a computable utility function will have convergent expected utilities iff that function is bounded”—then this is true in any set of possible worlds, meaning any consistent choice of h and S_I. I have chosen one.
The only reason I would not have a free choice of S_I, is if I had already nailed down S_I in some other way (say, by choosing the set I), or if I were using a set S_I for which there was no possible set of observations I such that S_I was equal to the set of worlds consistent with I, or because each application of the theorem is meant to apply to all possible choices of S_I
The last of these would mean that the theorem is actually proving that, for any unbounded utility function U, there exists some set of possible worlds for which it does not converge. That would be a still-interesting, but much weaker result.
You’re not allowed to—de Blanc has already supplied a definition of S_I. One must either adopt his definition or be talking about something other than his result.
He supplied a definition, not a particular set. I am using his definition, and providing one possible instantiation that is compatible with that definition.
The set S is the set of all total recursive functions. This is set in stone for all time. Therefore, there is only one way that S_I can refer to different things:
Our stock of observational data may be different. In other words, the set I and the values of h(i) for i in I may be different.
But regardless of I and the values of h(i), it’s easy to see that one cannot restrict S_I in the way you’re attempting to do.
In fact, one can easily see that S_I = the set of functions of the form “if x is in I then h(x), otherwise f(x)” where f is an arbitrary recursive function.
That is, the whole “I” business is completely pointless, except (presumably) to help the reader assure themselves that the result does apply to AIXI.
So, we can rescue our utility function from the theorem if we are allowed to assign zero probability to arbitrary hypotheses that have no plausibility other than that they have not been absolutely ruled out. Such as the hypothesis that the laws of physics are valid at all times except on October 21, 2011.
Being allowed to do this would make the counterexample work.