This doesn’t make sense to me. It seems that if you’re being strict about types, then “plain old probabilities” also require the correct type signature, and by using Shannon entropy you are still making an implicit assumption about the type signature.
Things that you can cast as a finite set. You can stretch this a bit by using limits to cover things that can be cast as compact metric spaces (and probably somewhat more than this), but this requires care and grounding in the finite set case in order to be unambiguously meaningful.
Ok, I see what you’re picturing now. That’s the picture we get if we approach probability through the Kolmogorov axioms. We get a different picture if we approach it through Cox’ theorem or logical inductors: these assign probabilities to sentences in a logic. That makes the things-over-which-we-have-a-probability-distribution extremely general—basically, we can assign probabilities to any statements we care to make about the universe, regardless of the type signature of the universe state.
Isn’t this just begging the question, though, by picking up an implicit type signature via the method by which probabilities are assigned? Like, if we lived in a different universe that followed different physics and had different math I’m not convinced it would all work out the same.
If the physics were different, information theory would definitely still be the same—it’s math, not physics. As for “different math”, I’m not even sure what that would mean or if the concept is coherent at all.
This doesn’t make sense to me. It seems that if you’re being strict about types, then “plain old probabilities” also require the correct type signature, and by using Shannon entropy you are still making an implicit assumption about the type signature.
What’s the type signature of the things over which we have a probability distribution?
Things that you can cast as a finite set. You can stretch this a bit by using limits to cover things that can be cast as compact metric spaces (and probably somewhat more than this), but this requires care and grounding in the finite set case in order to be unambiguously meaningful.
Ok, I see what you’re picturing now. That’s the picture we get if we approach probability through the Kolmogorov axioms. We get a different picture if we approach it through Cox’ theorem or logical inductors: these assign probabilities to sentences in a logic. That makes the things-over-which-we-have-a-probability-distribution extremely general—basically, we can assign probabilities to any statements we care to make about the universe, regardless of the type signature of the universe state.
Ah, that makes sense, thanks! I’d still say “sentences in a logic” is a specific type though.
Definitely, yes. The benefit is that it avoids directly specifying the type of the world-state.
Isn’t this just begging the question, though, by picking up an implicit type signature via the method by which probabilities are assigned? Like, if we lived in a different universe that followed different physics and had different math I’m not convinced it would all work out the same.
If the physics were different, information theory would definitely still be the same—it’s math, not physics. As for “different math”, I’m not even sure what that would mean or if the concept is coherent at all.