Something that bothers me about the Shannon entropy is that we know that it’s not the most fundamental type of entropy there is, since the von Neumann entropy is more fundamental.
A question I don’t have a great answer for: How could Shannon have noticed (a priori) that it was even possible that there was a more fundamental notion of entropy?
I don’t think I’d call von Neumann entropy “more fundamental”. After all, it only applies to quantum-mechanical universes, whereas Shannon applies to a much wider variety of universes. And to the extent that von Neumann entropy is itself interpretable as a Shannon entropy (which is how I usually think of it), Shannon also applies to this universe.
Shannon entropy is straightforwardly a special case of von Neumann entropy, so it applies to at least as many kinds of universes.
I still feel a bit confused about the “fundamentalness”, but in trying to formulate a response, I was convinced by Jaynes that von Neumann entropy has an adequate interpretation in terms of Shannon entropy.
Shannon entropy is straightforwardly a special case of von Neumann entropy, so it applies to at least as many kinds of universes.
Only if we’re already representing the universe in terms of quantum basis states. We can always take e.g. a deterministic universe and represent it in terms of pure states with zero entanglement, but that still involves a typecasting operation on our universe-state.
That’s the real issue here: von Neumann entropy comes with an assumption about the type signature of our universe-state, while Shannon entropy doesn’t—it’s just using plain old probability, and we can stick probability distributions on top of whatever we please.
This doesn’t make sense to me. It seems that if you’re being strict about types, then “plain old probabilities” also require the correct type signature, and by using Shannon entropy you are still making an implicit assumption about the type signature.
Things that you can cast as a finite set. You can stretch this a bit by using limits to cover things that can be cast as compact metric spaces (and probably somewhat more than this), but this requires care and grounding in the finite set case in order to be unambiguously meaningful.
Ok, I see what you’re picturing now. That’s the picture we get if we approach probability through the Kolmogorov axioms. We get a different picture if we approach it through Cox’ theorem or logical inductors: these assign probabilities to sentences in a logic. That makes the things-over-which-we-have-a-probability-distribution extremely general—basically, we can assign probabilities to any statements we care to make about the universe, regardless of the type signature of the universe state.
Isn’t this just begging the question, though, by picking up an implicit type signature via the method by which probabilities are assigned? Like, if we lived in a different universe that followed different physics and had different math I’m not convinced it would all work out the same.
If the physics were different, information theory would definitely still be the same—it’s math, not physics. As for “different math”, I’m not even sure what that would mean or if the concept is coherent at all.
I think the merit of Shannon was not to define entropy, but to understand the operational meaning of entropy in terms of coding a message with a minimal number of letters, leading to the notion of the capacity of a channel of communication, of error-correcting code and of “bit”.
Von Neumann’s entropy was introduced before Shannon’s entropy (1927, although the only reference I know is von Neumann’s book from 1932). It was also von Neumann how suggested the name “entropy” for the quantity that Shannon found. What Shannon could’ve noticed was that von Neumann’s entropy also has an operational meaning. But for that, he would’ve had to be interested in the transmission of quantum information by quantum channels, ideas that were not around at the time.
Something that bothers me about the Shannon entropy is that we know that it’s not the most fundamental type of entropy there is, since the von Neumann entropy is more fundamental.
A question I don’t have a great answer for: How could Shannon have noticed (a priori) that it was even possible that there was a more fundamental notion of entropy?
I don’t think I’d call von Neumann entropy “more fundamental”. After all, it only applies to quantum-mechanical universes, whereas Shannon applies to a much wider variety of universes. And to the extent that von Neumann entropy is itself interpretable as a Shannon entropy (which is how I usually think of it), Shannon also applies to this universe.
Shannon entropy is straightforwardly a special case of von Neumann entropy, so it applies to at least as many kinds of universes.
I still feel a bit confused about the “fundamentalness”, but in trying to formulate a response, I was convinced by Jaynes that von Neumann entropy has an adequate interpretation in terms of Shannon entropy.
Only if we’re already representing the universe in terms of quantum basis states. We can always take e.g. a deterministic universe and represent it in terms of pure states with zero entanglement, but that still involves a typecasting operation on our universe-state.
That’s the real issue here: von Neumann entropy comes with an assumption about the type signature of our universe-state, while Shannon entropy doesn’t—it’s just using plain old probability, and we can stick probability distributions on top of whatever we please.
This doesn’t make sense to me. It seems that if you’re being strict about types, then “plain old probabilities” also require the correct type signature, and by using Shannon entropy you are still making an implicit assumption about the type signature.
What’s the type signature of the things over which we have a probability distribution?
Things that you can cast as a finite set. You can stretch this a bit by using limits to cover things that can be cast as compact metric spaces (and probably somewhat more than this), but this requires care and grounding in the finite set case in order to be unambiguously meaningful.
Ok, I see what you’re picturing now. That’s the picture we get if we approach probability through the Kolmogorov axioms. We get a different picture if we approach it through Cox’ theorem or logical inductors: these assign probabilities to sentences in a logic. That makes the things-over-which-we-have-a-probability-distribution extremely general—basically, we can assign probabilities to any statements we care to make about the universe, regardless of the type signature of the universe state.
Ah, that makes sense, thanks! I’d still say “sentences in a logic” is a specific type though.
Definitely, yes. The benefit is that it avoids directly specifying the type of the world-state.
Isn’t this just begging the question, though, by picking up an implicit type signature via the method by which probabilities are assigned? Like, if we lived in a different universe that followed different physics and had different math I’m not convinced it would all work out the same.
If the physics were different, information theory would definitely still be the same—it’s math, not physics. As for “different math”, I’m not even sure what that would mean or if the concept is coherent at all.
I think the merit of Shannon was not to define entropy, but to understand the operational meaning of entropy in terms of coding a message with a minimal number of letters, leading to the notion of the capacity of a channel of communication, of error-correcting code and of “bit”.
Von Neumann’s entropy was introduced before Shannon’s entropy (1927, although the only reference I know is von Neumann’s book from 1932). It was also von Neumann how suggested the name “entropy” for the quantity that Shannon found. What Shannon could’ve noticed was that von Neumann’s entropy also has an operational meaning. But for that, he would’ve had to be interested in the transmission of quantum information by quantum channels, ideas that were not around at the time.