Hmm, I think that’s a red herring though. Consider humans—most of them have read lots of text from an enormous variety of sources as well. Also while it’s true that current LLMs have only a little bit of fine-tuning applied after their pre-training, and so you can maybe argue that they are mostly just trained to predict text, this will be less and less true in the future.
How about “LLMs are like baby alien shoggoths, that instead of being raised in alien culture, we’ve adopted at birth and are trying to raise in human culture. By having them read the internet all day.”
(Come to think of it, I actually would feel noticeably more hopeful about our prospects for alignment success if we actually were “raising the AGI like we would a child.” If we had some interdisciplinary team of ML and neuroscience and child psychology experts that was carefully designing a curriculum for our near-future AGI agents, a curriculum inspired by thoughtful and careful analogies to human childhood, that wouldn’t change my overall view dramatically but it would make me noticeably more hopeful. Maybe brain architecture & instincts basically don’t matter that much and Blank Slate theory is true enough for our purposes that this will work to produce an agent with values that are in-distribution for the range of typical modern human values!)
(This doesn’t contradict anything you said, but it seems like we totally don’t know how to “raise an AGI like we would a child” with current ML. Like I don’t think it counts for very much if almost all of the training time is a massive amount of next-token prediction. Like a curriculum of data might work very differently on AI vs humans due to a vastly different amount of data and a different training objective.)
I’ve seen mixed data on how important curricula are for deep learning. One paper (on CIFAR) suggested that curricula only help if you have very few datapoints or the labels are noisy. But possibly that doesn’t generalize to LLMs.
I think data ordering basically never matters for LLM pretraining. (As in, random is the best and trying to make the order more specific doesn’t help.)
Hmm, I think that’s a red herring though. Consider humans—most of them have read lots of text from an enormous variety of sources as well. Also while it’s true that current LLMs have only a little bit of fine-tuning applied after their pre-training, and so you can maybe argue that they are mostly just trained to predict text, this will be less and less true in the future.
How about “LLMs are like baby alien shoggoths, that instead of being raised in alien culture, we’ve adopted at birth and are trying to raise in human culture. By having them read the internet all day.”
(Come to think of it, I actually would feel noticeably more hopeful about our prospects for alignment success if we actually were “raising the AGI like we would a child.” If we had some interdisciplinary team of ML and neuroscience and child psychology experts that was carefully designing a curriculum for our near-future AGI agents, a curriculum inspired by thoughtful and careful analogies to human childhood, that wouldn’t change my overall view dramatically but it would make me noticeably more hopeful. Maybe brain architecture & instincts basically don’t matter that much and Blank Slate theory is true enough for our purposes that this will work to produce an agent with values that are in-distribution for the range of typical modern human values!)
(This doesn’t contradict anything you said, but it seems like we totally don’t know how to “raise an AGI like we would a child” with current ML. Like I don’t think it counts for very much if almost all of the training time is a massive amount of next-token prediction. Like a curriculum of data might work very differently on AI vs humans due to a vastly different amount of data and a different training objective.)
I’ve seen mixed data on how important curricula are for deep learning. One paper (on CIFAR) suggested that curricula only help if you have very few datapoints or the labels are noisy. But possibly that doesn’t generalize to LLMs.
I think data ordering basically never matters for LLM pretraining. (As in, random is the best and trying to make the order more specific doesn’t help.)
That was my impression too.