ryan_greenblatt comments on TurnTrout’s shortform feed

[ryan_greenblatt]

I think data ordering basically never matters for LLM pretraining. (As in, random is the best and trying to make the order more specific doesn’t help.)

[Daniel Kokotajlo]

That was my impression too.