AdamYedidia comments on GPT-2′s positional embedding matrix is a helix

[AdamYedidia]

Nope, this is the pos_embed matrix! So before the first layer.

[MiguelDev]

I see. I’ll try this thanks!