Thought: at one point, you talk about taking a general (not necessarily causal, infinite, or anything like that) distribution and applying the resampling process to this, leading to capturing the redundant information.
But does that necessarily work? I’d think that if e.g. your distribution was a multivariate Gaussian with nondeterministic correlations, you’d get regression to the mean, such that the limit of the resampling process just makes you end up with the mean. But this means that there’s no information left in the limiting variable.
I think what goes wrong is that if you start resampling the multivariate Gaussian, you end up combining two effects: blurring it (which is what you want, to abstract out nonlocal stuff), and dissipating it to the mean (which is what you don’t want). As long as you haven’t removed all the variance in the dissipation, the blur will still capture the information you want. But as you take the limit, the variance goes to zero and that prevents it from carrying any information.
In the Gaussian case specifically, you can probably solve that by just continually rescaling as you take the limit to keep the variance high, but I don’t know if there is a solution for e.g. discrete variables.
Wait, no, it’s resampling, not regression. So you introduce noise underway, which means if you only have a finite set of imperfectly correlated variables, the mutual information should drop to zero.
Thought: at one point, you talk about taking a general (not necessarily causal, infinite, or anything like that) distribution and applying the resampling process to this, leading to capturing the redundant information.
But does that necessarily work? I’d think that if e.g. your distribution was a multivariate Gaussian with nondeterministic correlations, you’d get regression to the mean, such that the limit of the resampling process just makes you end up with the mean. But this means that there’s no information left in the limiting variable.
I think what goes wrong is that if you start resampling the multivariate Gaussian, you end up combining two effects: blurring it (which is what you want, to abstract out nonlocal stuff), and dissipating it to the mean (which is what you don’t want). As long as you haven’t removed all the variance in the dissipation, the blur will still capture the information you want. But as you take the limit, the variance goes to zero and that prevents it from carrying any information.
In the Gaussian case specifically, you can probably solve that by just continually rescaling as you take the limit to keep the variance high, but I don’t know if there is a solution for e.g. discrete variables.
Wait, no, it’s resampling, not regression. So you introduce noise underway, which means if you only have a finite set of imperfectly correlated variables, the mutual information should drop to zero.