In mixture of expert problems, the experts are not independent, that’s the whole problem. They are all trying to correlate to some underlying reality, and thereby are correlated with each other.
But you also say “dozens of different things”. Are they trying to estimate the same things, different things, or different things that should all correlate to the same thing?
See my longer comment above for more details, but it’s sounding like you don’t want to evaluate over the whole data set, you just want to make some assumption about the statistics of your classifiers, and combine them via maximum entropy and those statistics.
In mixture of expert problems, the experts are not independent, that’s the whole problem. They are all trying to correlate to some underlying reality, and thereby are correlated with each other.
But you also say “dozens of different things”. Are they trying to estimate the same things, different things, or different things that should all correlate to the same thing?
See my longer comment above for more details, but it’s sounding like you don’t want to evaluate over the whole data set, you just want to make some assumption about the statistics of your classifiers, and combine them via maximum entropy and those statistics.