Speaking of abstractions, recently I’ve been wondering about whether they can be supported by some sort of counting argument. Suppose you live in a universe with volume V, and you are trying to create a taxonomy for objects of size v. Intuitively, it seems like there should be exp(v) many conceivable such objects. But there’s only space for v/V objects, so if exp(v)>v/V, we can fall back to “allocate a new label for each unique object”, which still allows us to abstract relatively cheaply (at logV/v cost). (Which is a tool we regularly use for abstraction! Like for people, companies, etc., we do have some abstract taxonomies, but mostly we refer to them by unique labels. After conditioning on their class of course; clearly “person” contains a lot of information.)
Of course allocating a new label for each unique object is often not so satisfying. In practice we can usually do way better than that, so the counting argument must be missing something. But some of the things the counting argument misses seem to be things that are covered by previous theories of abstraction, so maybe by combining multiple approaches we can get a more comprehensive theory. In a way, if we think of traditional abstraction methods as looking at a low-rank approximation of stuff, then the new “allocate a new label for each unique object” algorithm gives you a sparse approximation of stuff, and we know that stuff is usually well-approximated by low-rank + sparse.
Speaking of abstractions, recently I’ve been wondering about whether they can be supported by some sort of counting argument. Suppose you live in a universe with volume V, and you are trying to create a taxonomy for objects of size v. Intuitively, it seems like there should be exp(v) many conceivable such objects. But there’s only space for v/V objects, so if exp(v)>v/V, we can fall back to “allocate a new label for each unique object”, which still allows us to abstract relatively cheaply (at logV/v cost). (Which is a tool we regularly use for abstraction! Like for people, companies, etc., we do have some abstract taxonomies, but mostly we refer to them by unique labels. After conditioning on their class of course; clearly “person” contains a lot of information.)
Of course allocating a new label for each unique object is often not so satisfying. In practice we can usually do way better than that, so the counting argument must be missing something. But some of the things the counting argument misses seem to be things that are covered by previous theories of abstraction, so maybe by combining multiple approaches we can get a more comprehensive theory. In a way, if we think of traditional abstraction methods as looking at a low-rank approximation of stuff, then the new “allocate a new label for each unique object” algorithm gives you a sparse approximation of stuff, and we know that stuff is usually well-approximated by low-rank + sparse.