For clustering, one frame I’ve sometimes found useful is that if you don’t break stuff up into individual objects, you’ve got an extended space where for each location you’ve got the features that are present in that location. If you marginalize over location, you end up with a variable that is highly skewed, representing the fact that most locations are empty.
You could then condition on the variable being nonzero to return to something similar to your original clustering problem, but what I sometimes find useful is to think in the original highly skewed distribution.
If you do something like an SVD, you characterize the directions one can deviate from 0, which gives you something like a clustering, but in contrast to traditional clusterings it contains a built-in scale invariance element, since the magnitude of deviation from 0 is allowed to vary.
Thinking about the skewness is also neat for other reasons, e.g. it is part of what gives us the causal sparsity. (“Large” objects are harder to affect, and have more effect on other objects.)
Kind of tangential, but:
For clustering, one frame I’ve sometimes found useful is that if you don’t break stuff up into individual objects, you’ve got an extended space where for each location you’ve got the features that are present in that location. If you marginalize over location, you end up with a variable that is highly skewed, representing the fact that most locations are empty.
You could then condition on the variable being nonzero to return to something similar to your original clustering problem, but what I sometimes find useful is to think in the original highly skewed distribution.
If you do something like an SVD, you characterize the directions one can deviate from 0, which gives you something like a clustering, but in contrast to traditional clusterings it contains a built-in scale invariance element, since the magnitude of deviation from 0 is allowed to vary.
Thinking about the skewness is also neat for other reasons, e.g. it is part of what gives us the causal sparsity. (“Large” objects are harder to affect, and have more effect on other objects.)