GPT-NeoX-20B is a 20 billion parameter autoregressive language model whose weights will be made freely and openly available to the public through a permissive license. It is, to the best of our knowledge, the largest dense autoregressive model that has publicly available weights. In this paper, we describe the model architecture and training, evaluate its performance, and discuss the broader impacts of its release. We are open-sourcing the training and evaluation code, as well as the model weights, at https://github.com/EleutherAI/gpt-neox.
The alignment section of the paper would probably be especially of interest to this community.
EleutherAI’s GPT-NeoX-20B release
Link post
The alignment section of the paper would probably be especially of interest to this community.