One plausible possibility here might be that sets of feature vectors that are semantically related to each other in interesting ways (ways that favor certain forms of computation) tend to form (approximate) sub-spaces, of various types and dimensionalities facilitating those computations — which we could then learn a taxonomy of common types of. The days of the week circle would then be classified as an example of a two-dimensional subspace, specifically a circular modular arithmetic sub-space, more specifically modulo 7. Feature splitting would perhaps exist in rather higher-dimensional subspaces that usefully represented some form of tree structure of related concepts, possibly ones that made use of some combination of the cosine distances between features as some sort of similarity measure, and perhaps also of directions correlated with some related parts of the overall overcomplete basis (for example, perhaps in ways resembling word2vec’s king-man+woman~=queen structure).
One plausible possibility here might be that sets of feature vectors that are semantically related to each other in interesting ways (ways that favor certain forms of computation) tend to form (approximate) sub-spaces, of various types and dimensionalities facilitating those computations — which we could then learn a taxonomy of common types of. The days of the week circle would then be classified as an example of a two-dimensional subspace, specifically a circular modular arithmetic sub-space, more specifically modulo 7. Feature splitting would perhaps exist in rather higher-dimensional subspaces that usefully represented some form of tree structure of related concepts, possibly ones that made use of some combination of the cosine distances between features as some sort of similarity measure, and perhaps also of directions correlated with some related parts of the overall overcomplete basis (for example, perhaps in ways resembling word2vec’s king-man+woman~=queen structure).