Ronny Fernandez comments on Why do we assume there is a “real” shoggoth behind the LLM? Why not masks all the way down?

[Ronny Fernandez]

Yeah I’m totally with you that it definitely isn’t actually next token prediction, it’s some totally other goal drawn from the dist of goals you get when you sgd for minimizing next token prediction surprise.