You seem like as good a person to ask this as any: Is there a good introduction to information theory out there? How would one start digging into the field?
To be quite honest, I only really started to study it after reading Eliezer Yudkowsky’s Engines of Cognition, which connected it to what I know about thermodynamics. ( Twoblog posts inspired by it.) So, like you, I’m an autodidact on the topic.
Most people would recommend David MacKay’s downloadable book, which is written in a friendly, accessible tone. That helped a lot, but I also found it hard to follow at times. That may be due to not having a physical copy though. And it can’t be beat as a technical reference or in terms of depth.
Personally, my path to learning about it was to basically read the Wikipedia articles on Information Theory and Kullback-Leibler divergence, and every relevant, interesting link that branches off from those (on or off wikipedia).
ETA: Oh, and learning about statistical mechanics, especially the canonical ensemble was a big help for me too, esp. given the relation to the E. T. Jaynes articles on the maximum entropy formalism. But YMMV.
You seem like as good a person to ask this as any: Is there a good introduction to information theory out there? How would one start digging into the field?
To be quite honest, I only really started to study it after reading Eliezer Yudkowsky’s Engines of Cognition, which connected it to what I know about thermodynamics. ( Two blog posts inspired by it.) So, like you, I’m an autodidact on the topic.
Most people would recommend David MacKay’s downloadable book, which is written in a friendly, accessible tone. That helped a lot, but I also found it hard to follow at times. That may be due to not having a physical copy though. And it can’t be beat as a technical reference or in terms of depth.
Personally, my path to learning about it was to basically read the Wikipedia articles on Information Theory and Kullback-Leibler divergence, and every relevant, interesting link that branches off from those (on or off wikipedia).
ETA: Oh, and learning about statistical mechanics, especially the canonical ensemble was a big help for me too, esp. given the relation to the E. T. Jaynes articles on the maximum entropy formalism. But YMMV.