To start, I propose a different frame to help you. Ask yourself not “How do I get intuition about information theory?” instead ask “How is information theory informing my intuitions?”
It looks to me like it’s more central than is Bayes’ Theorem, and that it provides essential context for why and how that theorem is relevant for rationality.
You’ve already noticed that this is “deep” and “widely applicable.” Another way of saying these things is “abstract,” and abstraction reflects generalizations over some domain of experience. These generalizations are the exact sort of things which form heuristics and intuitions to apply to more specific cases.
To the meat of question:
Step 1) grok the core technology (which you seem have already started)
Step 2) ask yourself the aforementioned question.
Step 3) try to apply it to as many domains as possible
Step 4) as you come into trouble with 3), restart from 1).
When you find yourself looking for more of 1) from where you are now, I recommend at least checking out Shannon’s original paper on information. I find his writing style to be much more approachable than average for what is a highly technical paper. Be careful when reading though, because his writing is very dense with each sentence carrying a lot of information ;)
To start, I propose a different frame to help you. Ask yourself not “How do I get intuition about information theory?” instead ask “How is information theory informing my intuitions?”
You’ve already noticed that this is “deep” and “widely applicable.” Another way of saying these things is “abstract,” and abstraction reflects generalizations over some domain of experience. These generalizations are the exact sort of things which form heuristics and intuitions to apply to more specific cases.
To the meat of question:
Step 1) grok the core technology (which you seem have already started)
Step 2) ask yourself the aforementioned question.
Step 3) try to apply it to as many domains as possible
Step 4) as you come into trouble with 3), restart from 1).
When you find yourself looking for more of 1) from where you are now, I recommend at least checking out Shannon’s original paper on information. I find his writing style to be much more approachable than average for what is a highly technical paper. Be careful when reading though, because his writing is very dense with each sentence carrying a lot of information ;)