To clarify where my responses are coming from: I think what I’m saying is not that directly relevant to your specific point in the post. I’m more (1) interested in discussing the notion of only-X, broadly, and (2) reacting to the feature of your discussion (shared by much other discussion) that you (IIUC) consider only the extensional (input-output) behavior of programs, excluding from analysis the intensional properties. (Which is a reasonable approach, e.g. because the input-output behavior captures much of what we care about, and also because it’s maybe easier to analyze and already contains some of our problems / confusions.)
From where I’m sitting, when a program “makes an observation of the world”, that’s moving around in codespace. There’s of course useful stuff to say about the part that didn’t change. When we really understand how a cognitive algorithm works, it starts to look like a clear algorithm / data separation; e.g. in Bayesian updating, we have a clear picture of the code that’s fixed, and how it operates on the varying data. But before we understand the program in that way, we might be unable to usefully separate it out into a fixed part and a varying part. Then it’s natural to say things like “the child invented a strategy for picking up blocks; next time, they just use that strategy”, where the first clause is talking about a change in source code. We know for sure that such separations can be done, because for example we can say that the child is always operating in accordance with fixed physical law, and we might suspect there’s “fundamental brain algorithms” that are also basically fixed. Likewise, even though Solomonoff induction is always just Solomonoff induction plus data, it can be also useful to understand SI(some data) in terms of understanding those programs that are highly ranked by SI(some data), and it seems reasonable to call that “the algorithm changed to emphasize those programs”.
To clarify where my responses are coming from: I think what I’m saying is not that directly relevant to your specific point in the post. I’m more (1) interested in discussing the notion of only-X, broadly, and (2) reacting to the feature of your discussion (shared by much other discussion) that you (IIUC) consider only the extensional (input-output) behavior of programs, excluding from analysis the intensional properties. (Which is a reasonable approach, e.g. because the input-output behavior captures much of what we care about, and also because it’s maybe easier to analyze and already contains some of our problems / confusions.)
From where I’m sitting, when a program “makes an observation of the world”, that’s moving around in codespace. There’s of course useful stuff to say about the part that didn’t change. When we really understand how a cognitive algorithm works, it starts to look like a clear algorithm / data separation; e.g. in Bayesian updating, we have a clear picture of the code that’s fixed, and how it operates on the varying data. But before we understand the program in that way, we might be unable to usefully separate it out into a fixed part and a varying part. Then it’s natural to say things like “the child invented a strategy for picking up blocks; next time, they just use that strategy”, where the first clause is talking about a change in source code. We know for sure that such separations can be done, because for example we can say that the child is always operating in accordance with fixed physical law, and we might suspect there’s “fundamental brain algorithms” that are also basically fixed. Likewise, even though Solomonoff induction is always just Solomonoff induction plus data, it can be also useful to understand SI(some data) in terms of understanding those programs that are highly ranked by SI(some data), and it seems reasonable to call that “the algorithm changed to emphasize those programs”.