yeah… not trying for a complete analysis here, but one thing which is missing is the all-important residual stream. It has been rather downplayed in the original “Attention is all you need” paper, and has been greatly emphasized in https://transformer-circuits.pub/2021/framework/index.html
but I have to admit that I’ve only started to feel that I more-or-less understand principal aspects of Transformer architecture after I’ve spent some quality time with the pedagogical implementation of GPT-2 by Andrej Karpathy, https://github.com/karpathy/minGPT, specifically with the https://github.com/karpathy/minGPT/blob/master/mingpt/model.py file. When I don’t understand something in a text, looking at a nice relatively simple-minded implementation allows me to see what exactly is going on
(People have also published some visualizations, some “illustrated Transformers”, and those are closer to the style of your sketches, but I don’t know which of them are good and which might be misleading. And, yes, at the end of the day, it takes time to get used to Transformers, one understands them gradually.)
Ah, it’s mostly your first figure which is counter-intuitive (when one looks at it, one gets the intuition of f(g(h... (x))), so it de-emphasizes the fact that each of these Transformer Block transformations is shaped like x=x+function(x))
Oh, yes you’re right...hmmm. Unsure what to do about that, I’d love a neat graphical solution. Wasn’t quite sure how to represent repeating units without just duplicating subdiagrams!
yeah… not trying for a complete analysis here, but one thing which is missing is the all-important residual stream. It has been rather downplayed in the original “Attention is all you need” paper, and has been greatly emphasized in https://transformer-circuits.pub/2021/framework/index.html
but I have to admit that I’ve only started to feel that I more-or-less understand principal aspects of Transformer architecture after I’ve spent some quality time with the pedagogical implementation of GPT-2 by Andrej Karpathy, https://github.com/karpathy/minGPT, specifically with the https://github.com/karpathy/minGPT/blob/master/mingpt/model.py file. When I don’t understand something in a text, looking at a nice relatively simple-minded implementation allows me to see what exactly is going on
(People have also published some visualizations, some “illustrated Transformers”, and those are closer to the style of your sketches, but I don’t know which of them are good and which might be misleading. And, yes, at the end of the day, it takes time to get used to Transformers, one understands them gradually.)
Many thanks! I have residuals going from before to after the MHA block, and from before to after MLP+FFNN. Should I have others in other places?
edit: Those links are super!
Ah, it’s mostly your first figure which is counter-intuitive (when one looks at it, one gets the intuition of
f(g(h... (x))), so it de-emphasizes the fact that each of these Transformer Block transformations is shaped likex=x+function(x))Oh, yes you’re right...hmmm. Unsure what to do about that, I’d love a neat graphical solution. Wasn’t quite sure how to represent repeating units without just duplicating subdiagrams!