Intelligence requires observations. An intelligent agent needs to strike a delicate balance between the persistence of its internal structure and its susceptibility to new evidence. The optimal balance, a Bayesian update, has been explained many times before, and was shown to be optimal in keeping information about the world.
What’s optimal in terms of keeping information about the world is to store your entire sense data history—and everything you were born with. That is often extremely wasteful of memory (even if you compress) - but surely much more compact than keeping track of all the possible hypotheses that might explain your sense data, and their respective probabilities. I think it is hard to come up with a sense in which that is optimal.
You know, non-parametric methods along the lines of k-nearest neighbors basically do this. The cleverness in these methods is finding a log(n) access scheme for your data (a database problem, essentially, rather than a machine learning problem).
What’s optimal in terms of keeping information about the world is to store your entire sense data history—and everything you were born with. That is often extremely wasteful of memory (even if you compress) - but surely much more compact than keeping track of all the possible hypotheses that might explain your sense data, and their respective probabilities. I think it is hard to come up with a sense in which that is optimal.
You know, non-parametric methods along the lines of k-nearest neighbors basically do this. The cleverness in these methods is finding a log(n) access scheme for your data (a database problem, essentially, rather than a machine learning problem).