On @TsviBT’s recommendation, I’m writing this up quickly here.
re: the famous graph from https://transformer-circuits.pub/2022/toy_model/index.html#geometry with all the colored bands, plotting “dimensions per feature in a model with superposition”, there look to be 3 obvious clusters outside of any colored band and between 2/5 and 1/2, the third of which is directly below the third inset image from the right. All three of these clusters are at 1/(1-S) ~ 4.
A picture of the plot, plus a summary of my thought processes for about the first 30 seconds of looking at it from the right perspective:
In particular, the clusters appear to correspond to dimensions-per-feature of about 0.44~0.45, that is, 4⁄9. Given the Thomson problem-ish nature of all the other geometric structures displayed, and being professionally dubious that there should be only such structures of subspace dimension 3 or lower, my immediate suspicion since last week when I first thought about this is that the uncolored clusters should be packing 9 vectors as far apart from each other as possible on the surface of a 3-sphere in some 4D subspace.
In particular, mathematicians have already found a 23-celled 4-tope with 9 vertices (which I have made some sketches of) where the angular separation between vertices is ~80.7° : http://neilsloane.com/packings/index.html#I . Roughly, the vertices are: the north pole of S^3; on a slice just (~9°) north of the equator, the vertices of a tetrahedron “pointing” in some direction; on a slice somewhat (~19°) north of the south pole, the vertices of a tetrahedron “pointing” dually to the previous tetrahedron. The edges are given by connecting vertices in each layer to the vertices in the adjacent layer or layers. Cross sections along the axis I described look like growing tetrahedra, briefly become various octahedra as we cross the first tetrahedon, and then resolve to the final tetrahedron before vanishing.
I therefore predict that we should see these clusters of 9 embedding vectors lying roughly in 4D subspaces taking on pretty much exactly the 23-cell shape mathematicians know about, to the same general precision as we’d find (say) pentagons or square antiprisms, within the model’s embedding vectors, when S ~ 3⁄4.
Potentially also there’s other 3/f, 4/f, and maybe 5/f; given professional experience I would not expect to see 6+/f sorts of features, because 6+ dimensions is high-dimensional and the clusters would (approximately) factor as products of lower-dimensional clusters already listed. There’s a few more clusters that I suspect might correspond to 3⁄7 (a pentagonal bipyramid?) or 5⁄12 (some terrifying 5-tope with 12 vertices, I guess), but I’m way less confident in those.
A hand-drawn rendition of the 23-cell in whiteboard marker:
I played with this with a colab notebook way back when. I can’t visualize things directly in 4 dimensions, but at the time I came up with the trick of visualizing the pairwise cosine similarity for each pair of features, which gives at least a local sense of what the angles are like.
Trying to squish 9 features into 4 dimensions looks to me like it either ends up with
4 antipodal pairs which are almost orthogonal to one another, and then one “orphan” direction squished into the largest remaining space
OR
3 almost orthogonal antipodal pairs plus a “Y” shape with the narrow angle being 72º and the wide angles being 144º
For reference this is what a square antiprism looks like in this type of diagram:
You maybe got stuck in some of the many local optima that Nurmela 1995 runs into. Genuinely, the best sphere code for 9 points in 4 dimensions is known to have a minimum angular separation of ~1.408 radians, for a worst-case cosine similarity of about 0.162.
You got a lot further than I did with my own initial attempts at random search, but you didn’t quite find it, either.
On @TsviBT’s recommendation, I’m writing this up quickly here.
re: the famous graph from https://transformer-circuits.pub/2022/toy_model/index.html#geometry with all the colored bands, plotting “dimensions per feature in a model with superposition”, there look to be 3 obvious clusters outside of any colored band and between 2/5 and 1/2, the third of which is directly below the third inset image from the right. All three of these clusters are at 1/(1-S) ~ 4.
A picture of the plot, plus a summary of my thought processes for about the first 30 seconds of looking at it from the right perspective:
In particular, the clusters appear to correspond to dimensions-per-feature of about 0.44~0.45, that is, 4⁄9. Given the Thomson problem-ish nature of all the other geometric structures displayed, and being professionally dubious that there should be only such structures of subspace dimension 3 or lower, my immediate suspicion since last week when I first thought about this is that the uncolored clusters should be packing 9 vectors as far apart from each other as possible on the surface of a 3-sphere in some 4D subspace.
In particular, mathematicians have already found a 23-celled 4-tope with 9 vertices (which I have made some sketches of) where the angular separation between vertices is ~80.7° : http://neilsloane.com/packings/index.html#I . Roughly, the vertices are: the north pole of S^3; on a slice just (~9°) north of the equator, the vertices of a tetrahedron “pointing” in some direction; on a slice somewhat (~19°) north of the south pole, the vertices of a tetrahedron “pointing” dually to the previous tetrahedron. The edges are given by connecting vertices in each layer to the vertices in the adjacent layer or layers. Cross sections along the axis I described look like growing tetrahedra, briefly become various octahedra as we cross the first tetrahedon, and then resolve to the final tetrahedron before vanishing.
I therefore predict that we should see these clusters of 9 embedding vectors lying roughly in 4D subspaces taking on pretty much exactly the 23-cell shape mathematicians know about, to the same general precision as we’d find (say) pentagons or square antiprisms, within the model’s embedding vectors, when S ~ 3⁄4.
Potentially also there’s other 3/f, 4/f, and maybe 5/f; given professional experience I would not expect to see 6+/f sorts of features, because 6+ dimensions is high-dimensional and the clusters would (approximately) factor as products of lower-dimensional clusters already listed. There’s a few more clusters that I suspect might correspond to 3⁄7 (a pentagonal bipyramid?) or 5⁄12 (some terrifying 5-tope with 12 vertices, I guess), but I’m way less confident in those.
A hand-drawn rendition of the 23-cell in whiteboard marker:
I played with this with a colab notebook way back when. I can’t visualize things directly in 4 dimensions, but at the time I came up with the trick of visualizing the pairwise cosine similarity for each pair of features, which gives at least a local sense of what the angles are like.
Trying to squish 9 features into 4 dimensions looks to me like it either ends up with
4 antipodal pairs which are almost orthogonal to one another, and then one “orphan” direction squished into the largest remaining space
OR
3 almost orthogonal antipodal pairs plus a “Y” shape with the narrow angle being 72º and the wide angles being 144º
For reference this is what a square antiprism looks like in this type of diagram:
You maybe got stuck in some of the many local optima that Nurmela 1995 runs into. Genuinely, the best sphere code for 9 points in 4 dimensions is known to have a minimum angular separation of ~1.408 radians, for a worst-case cosine similarity of about 0.162.
You got a lot further than I did with my own initial attempts at random search, but you didn’t quite find it, either.
EDIT: I and the person who first tried to render this SHAPE for me misunderstood its nature.