Normal agents won’t. Genuinely intelligent agents won’t.
I think those who are arguing that it will are imagining an agent with the Solomonoff prior totally wired into them in a manner that they can’t possibly unlearn.
But still, even if you have the Occamian prior (which I think is what’s meant by the Solomonoff prior), there is no need to unlearn it. You retain a prior on all hypotheses that decreases in weight exponentially with length, and it persists on top of any observations you’ve updated on. Those new observations, combined with the Occamian prior give you the optimal weights on (prefix) sensory bitstreams, discounting the ruled-out ones and favoring those closer to what you’ve actually observed.
Even then, it keeps updating in favor of the observations that match what an oracle gives (without having to explicitly represent that they’re from an oracle). No penalty from failure to unlearn.
The thing is, there is no one true razor. Different sources have different associated reference machines—some are more like Turing Machines, others are more like CA. If what you are looking at is barcodes, then short ones are pretty rare—and if you go into simulated worlds, sources can have practically any distribution you care to mention.
Yes, you can model these as “complier overhead” constants—which represent the “cost” of simulating one reference machine in another—but that is just another way of saying you have to unlearn the Solomonoff prior and use another one—which is more appropriate for your source.
You can still do that, whatever your reference machine is—provided it is computationally universal—and doesn’t have too much “faith”.
Normal agents won’t. Genuinely intelligent agents won’t.
I think those who are arguing that it will are imagining an agent with the Solomonoff prior totally wired into them in a manner that they can’t possibly unlearn.
But still, even if you have the Occamian prior (which I think is what’s meant by the Solomonoff prior), there is no need to unlearn it. You retain a prior on all hypotheses that decreases in weight exponentially with length, and it persists on top of any observations you’ve updated on. Those new observations, combined with the Occamian prior give you the optimal weights on (prefix) sensory bitstreams, discounting the ruled-out ones and favoring those closer to what you’ve actually observed.
Even then, it keeps updating in favor of the observations that match what an oracle gives (without having to explicitly represent that they’re from an oracle). No penalty from failure to unlearn.
The thing is, there is no one true razor. Different sources have different associated reference machines—some are more like Turing Machines, others are more like CA. If what you are looking at is barcodes, then short ones are pretty rare—and if you go into simulated worlds, sources can have practically any distribution you care to mention.
Yes, you can model these as “complier overhead” constants—which represent the “cost” of simulating one reference machine in another—but that is just another way of saying you have to unlearn the Solomonoff prior and use another one—which is more appropriate for your source.
You can still do that, whatever your reference machine is—provided it is computationally universal—and doesn’t have too much “faith”.