nostalgebraist comments on [Link] Training Compute-Optimal Large Language Models

[nostalgebraist]

It’s strongly implied to be 2048, as in Gopher.