How do you learn Solomonoff Induction?
I read about a fascinating technique described on Wikipedia as a mathematically formalized combination of Occam’s razor and the Principle of Multiple Explanations. I want to add this to my toolbox. I’m dreaming of a concise set of actionable instructions for using Solomonoff induction. I realize this wish might be overly idealistic. I’m willing to peruse a much more convoluted tome and will consider making time for any background knowledge or prerequisites involved.
If anyone knows of a good book on this, or can tell me what set of information I need to acquire, please let me know. It would be much appreciated!
Solomonoff Induction is uncomputable, and implementing it will not be possible even in principle. It should be understood as an ideal which you should try to approximate, rather than something you can ever implement.
Solomonoff Induction is just bayesian epistemology with a prior determined by information theoretic complexity. As an imperfect agent trying to approximate it, you will get most of your value from simply grokking Bayesian epistemology. After you’ve done that, you may want to spend some time thinking about the philosophy of science of setting priors based on information theoretic complexity.
The classic textbook is Li and Vitanyi’s An Introduction to Kolmogorov Complexity and Its Applications.
Solomonoff induction is uncomputable, thus, as a direct consequence, it cannot be learned. Some approximations to it which are of practical interest: Occam learning and probably approximately correct learning. As a general matter, these questions are addressed by computational learning theory.
https://wiki.lesswrong.com/wiki/Solomonoff_induction http://lesswrong.com/lw/dhg/an_intuitive_explanation_of_solomonoff_induction/ should get you started.
Also Yudkowsky’s article on Occam’s Razor describes the Occam’s razor/ simplicity prior OP was interested in.
My book describes a philosophy of science based on large scale lossless data compression. It is not going to give you a toolbox for using SI; as others have observed, SI is of primarily theoretical importance, since it can’t be computed. However, different aspects of the book might help expand your worldview in this area.