This is a neat observation, but I’m reminded of a story I was told about a math professor:
One day while in the middle of a long proof of an arcane theorem, the professor was stopped and questioned about a particular step by a student, who wondered what made that step true. The professor said “Its trivial!” then thought a bit more about the step, mumbled to himself “Wait, is it trivial?”, and excused himself to step out of the hall and think. Ten minutes later, he comes back into the hall and declares the step was indeed trivial, and proceeds along with the proof.
This feels similar to me. Neel and Tom figure out this algorithm 9.5 months ago, and now mathematicians have just realized that indeed the algorithm is obvious and simple, and indeed the only way to the operation described when matrix multiplications come easy.
Not to say the insight is wrong, but I would be far more impressed if you were able to predict the algorithm a network does in advance through similar reasoning rather than a 9.5 month later justification.
OK so just being completely honest, I don’t know if it’s just me but I’m getting a slightly weird or snarky vibe from this comment? I guess I will assume there is a good faith underlying point being made to which I can reply. So just to be clear:
I did not use any words such as “trivial”, “obvious” or “simple”. Stories like the one you recount are obviously making fun of mathematicians, some of whom do think its cool to say things are trivial/simple/obvious after they understand them. I often strongly disagree and generally dislike this behaviour and think there are many normal mathematicians who don’t engage in this sort of thing. In particular sometimes the most succinct insights are the hardest ones to come by (this isn’t a reference to my post; just a general point). And just because such insights are easily expressible once you have the right framing and the right abstractions, they should by no means be trivialized.
I deliberately emphasized the subjectivity of making the sorts of judgements that I am making. Again this kinda forms part of the joke of the story.
I have indeed been aware of the work since when it was first posted 10 months ago or so and have given it some thought on and off for a while (in the first sentence of the post I was just saying that I didn’t spend long writing the post, not that these thoughts were easily arrived-at).
I do not claim to have explained the entire algorithm, only to shed some light on why it might actually be a more natural thing to do than some people seem to have appreciated.
I think the original work is of a high quality and one might reasonably say ‘groundbreaking’.
In another one of my posts I discuss at more length the kind of thing you bring up in the last sentence of your comment, e.g.
it can feel like the role that serious mathematics has to play in interpretability is primarily reactive, i.e. consists mostly of activities like ‘adding’ rigour after the fact or building narrow models to explain specific already-observed phenomena.
....[but]… one of the most lauded aspects of mathematics is a certain inevitability with which our abstractions take on a life of their own and reward us later with insight, generalization, and the provision of predictions. Moreover—remarkably—often those abstractions are found in relatively mysterious, intuitive ways: i.e. not as the result of us just directly asking “What kind of thing seems most useful for understanding this object and making predictions?” but, at least in part, as a result of aesthetic judgement and a sense of mathematical taste.
And e.g. I talk about how this sort of thing has been the case in areas like mathematical physics for a long time. Part of the point is that (in my opinion, at least) there isn’t any neat shortcut to the kind of abstract thinking that lets you make the sort of predictions you are making reference to. It is very typical that you have to begin by reacting to existing empirical phenomena and using it as scaffolding. But I think, to me, it has come across as that you are being somewhat dismissive of this fact? As if, when B might well follow from A and someone actually starts to do A, you say “I would be far more impressed if B” instead of “maybe that’s progress towards B”?
(Also FWIW, Neel claims here that regarding the algorithm itself, another researcher he knows “roughly predicted this”.)
I don’t know if it’s just me but I’m getting a slightly weird or snarky vibe from this comment?
Sorry about that. On a re-read, I can see how the comment could be seen as snarky, but I was going more for critical via illustrative analogy. Oh the perils of the lack of inflection and facial expressions.
I think your criticisms of my thought in the above comment are right-on, and you’ve changed my mind on how useful your post was. I do think that lots of progress can be made in understanding stuff by just finding the right frame by which the result seems natural, and your post is doing this. Thanks!
My submission: when we teach modular arithmetic to people, we do it using the metaphor of clock arithmetic. Well, if you ignore the multiple frequencies and argmax weirdness, clock arithmetic is exactly what this network is doing! Find the coordinates of rotating the hour hand (on a 113-hour clock) x hours, then y hours, use trig identities to work out what it would be if you rotated x+y hours, then count how many steps back you have to rotate to get to 0 to tell where you ended up. In fairness, the final step is a little bit different than the usual imagined rule of “look at the hour mark where the hand ends up”, but not so different that clock arithmetic counts as a bad prediction IMO.
I agree a rotation matrix story would fit better, but I do think it’s a fair analogy: the numbers stored are just coses and sines, aka the x and y coordinates of the hour hand.
Like, the only reason we’re calling it a “Fourier basis” is that we’re looking at a few different speeds of rotation, in order to scramble the second-place answers that almost get you a cos of 1 at the end, while preserving the actual answer.
This is a neat observation, but I’m reminded of a story I was told about a math professor:
This feels similar to me. Neel and Tom figure out this algorithm 9.5 months ago, and now mathematicians have just realized that indeed the algorithm is obvious and simple, and indeed the only way to the operation described when matrix multiplications come easy.
Not to say the insight is wrong, but I would be far more impressed if you were able to predict the algorithm a network does in advance through similar reasoning rather than a 9.5 month later justification.
Hi Garrett,
OK so just being completely honest, I don’t know if it’s just me but I’m getting a slightly weird or snarky vibe from this comment? I guess I will assume there is a good faith underlying point being made to which I can reply. So just to be clear:
I did not use any words such as “trivial”, “obvious” or “simple”. Stories like the one you recount are obviously making fun of mathematicians, some of whom do think its cool to say things are trivial/simple/obvious after they understand them. I often strongly disagree and generally dislike this behaviour and think there are many normal mathematicians who don’t engage in this sort of thing. In particular sometimes the most succinct insights are the hardest ones to come by (this isn’t a reference to my post; just a general point). And just because such insights are easily expressible once you have the right framing and the right abstractions, they should by no means be trivialized.
I deliberately emphasized the subjectivity of making the sorts of judgements that I am making. Again this kinda forms part of the joke of the story.
I have indeed been aware of the work since when it was first posted 10 months ago or so and have given it some thought on and off for a while (in the first sentence of the post I was just saying that I didn’t spend long writing the post, not that these thoughts were easily arrived-at).
I do not claim to have explained the entire algorithm, only to shed some light on why it might actually be a more natural thing to do than some people seem to have appreciated.
I think the original work is of a high quality and one might reasonably say ‘groundbreaking’.
In another one of my posts I discuss at more length the kind of thing you bring up in the last sentence of your comment, e.g.
And e.g. I talk about how this sort of thing has been the case in areas like mathematical physics for a long time. Part of the point is that (in my opinion, at least) there isn’t any neat shortcut to the kind of abstract thinking that lets you make the sort of predictions you are making reference to. It is very typical that you have to begin by reacting to existing empirical phenomena and using it as scaffolding. But I think, to me, it has come across as that you are being somewhat dismissive of this fact? As if, when B might well follow from A and someone actually starts to do A, you say “I would be far more impressed if B” instead of “maybe that’s progress towards B”?
(Also FWIW, Neel claims here that regarding the algorithm itself, another researcher he knows “roughly predicted this”.)
Sorry about that. On a re-read, I can see how the comment could be seen as snarky, but I was going more for critical via illustrative analogy. Oh the perils of the lack of inflection and facial expressions.
I think your criticisms of my thought in the above comment are right-on, and you’ve changed my mind on how useful your post was. I do think that lots of progress can be made in understanding stuff by just finding the right frame by which the result seems natural, and your post is doing this. Thanks!
My submission: when we teach modular arithmetic to people, we do it using the metaphor of clock arithmetic. Well, if you ignore the multiple frequencies and argmax weirdness, clock arithmetic is exactly what this network is doing! Find the coordinates of rotating the hour hand (on a 113-hour clock) x hours, then y hours, use trig identities to work out what it would be if you rotated x+y hours, then count how many steps back you have to rotate to get to 0 to tell where you ended up. In fairness, the final step is a little bit different than the usual imagined rule of “look at the hour mark where the hand ends up”, but not so different that clock arithmetic counts as a bad prediction IMO.
Is this really an accurate analogy? I feel like clock arithmetic would be more like representing it as a rotation matrix, not a Fourier basis.
I agree a rotation matrix story would fit better, but I do think it’s a fair analogy: the numbers stored are just coses and sines, aka the x and y coordinates of the hour hand.
Like, the only reason we’re calling it a “Fourier basis” is that we’re looking at a few different speeds of rotation, in order to scramble the second-place answers that almost get you a cos of 1 at the end, while preserving the actual answer.
Here’s another way of looking at it which could be said to make it more trivial:
We can transform addition into multiplication by taking the exponential, i.e. x+y=z is equivalent to 10^x * 10^y = 10^z.
But if we unfold the digits into separate axes rather than as a single number, then 10^n is just a one-hot encoding of the integer n.
Taking the Fourier transform of the digits to do convolutions is a well-known fast multiplication algorithm.