Given a network to analyze, have you considered training a GAN to generate some requested activations given some other activations? That should give straightforward estimates for the likes of mutual information between modules, and would be useful anyway to illustrate the function of a given module by generating variants of an input with the same activations in that module.
Not sure what to do about that mutual information often being infinite. Counting the dimension of the “mutual space” seems too discrete...
Given a network to analyze, have you considered training a GAN to generate some requested activations given some other activations? That should give straightforward estimates for the likes of mutual information between modules, and would be useful anyway to illustrate the function of a given module by generating variants of an input with the same activations in that module.
Not sure what to do about that mutual information often being infinite. Counting the dimension of the “mutual space” seems too discrete...