A potential approach to tackle this could be to aim to discover features in smaller batches. After each batch of discovered features finishes learning we could freeze them and only calculate the orthogonality regularisation within the next batch, as well as between the next batch and the frozen features. Importantly we wouldn’t need to apply the regularisation within the already discovered features.
If n is the number of feature we’re trying to discover and m is the number of features in each batch, then I’m thinking the naive approach is O(n^2) while the batch approach would be O(m^2 + mn). Still quadratic in m, but we would have m<<n
Wouldn’t this still be quadratic?
If n is the number of feature we’re trying to discover and m is the number of features in each batch, then I’m thinking the naive approach is O(n^2) while the batch approach would be O(m^2 + mn). Still quadratic in m, but we would have m<<n
Isn’t that just one batch?
My bad! Yes since that’s just one batch it does indeed come out as quadratic overall. I’ll have a think about more efficient methods