You are using the wrong formula/using the formula wrong. Shannon actually uses markov models for the process you are describing and defines Entropy as the limit of ever greater n-grams. I’d agree that n-grams aren’t necessarily the best model of what compression looks like.
Wow, despite no longer being endorsed, this comment is actually extremely relevant to the upcoming posts! I have to admit I never went through the original paper in detail. It looks like Shannon was even more impressive than I realized! Thanks, I’ll definitely have to go through this slowly.
You are using the wrong formula/using the formula wrong. Shannon actually uses markov models for the process you are describing and defines Entropy as the limit of ever greater n-grams. I’d agree that n-grams aren’t necessarily the best model of what compression looks like.
Wow, despite no longer being endorsed, this comment is actually extremely relevant to the upcoming posts! I have to admit I never went through the original paper in detail. It looks like Shannon was even more impressive than I realized! Thanks, I’ll definitely have to go through this slowly.
Yeah it’s so relevant that I assumed I must have overlooked something.