MiguelDev comments on GPT-2′s positional embedding matrix is a helix

MiguelDev 22 Jul 2023 6:07 UTC
1 point
0
It remains unexplained why this helical structure is the most natural way for GPT-2 to express position.

Just an intuition but GPT2 uses GELU function which is normalized by pi.

Btw, what visualization softwared did you use here again?
- AdamYedidia 22 Jul 2023 6:19 UTC
  2 points
  0
  Parent
  Python (the matplotlib package).
  - MiguelDev 22 Jul 2023 7:01 UTC
    1 point
    0
    Parent
    Thank you. Will try it in the project im working on!
    - MiguelDev 23 Jul 2023 0:43 UTC
      1 point
      0
      Parent
      Also, what vectors are you using? is this the final output layer?
      
      I suggest trying the vectors in the encoder layers 0-48 in GPT2-xl. I am getting the impression that the visualization of those layers are more of a submerged iceberg rather than a helix...
      - AdamYedidia 26 Jul 2023 19:52 UTC
        2 points
        0
        Parent
        Nope, this is the pos_embed matrix! So before the first layer.
        MiguelDev 27 Jul 2023 2:37 UTC
        1 point
        0
        Parent
        I see. I’ll try this thanks!