jacopo comments on GPT-2′s positional embedding matrix is a helix

[jacopo]

Is there something that would regularise the vectors towards constant norm? An helix would make a lot of sense in this case. Especially one with varying radius, like in some (not all) the images