And/or, that technique might be very useful for AIs generating/editing the synthetic data.
On the subject of cost, it’s also possible that we only need 10%, or 1%, or 0.1% of the dataset to illustrate the meaning of the <AI> tag, and the majority or great majority of it can be in human mode. I’m fairly sure both that more will be better, and that there will be diminishing returns from adding more, so if the alignbment tax of doing the entire dataset is too high, investigating how good results we can get with a smaller proprtion would be worth it. I believe the Pretraining Language Models with Human Preferences paper simply did the entire training set, but they were using processing that was a lot cheper to do than what I’m proposing.
Another possibility is that you’d do actually better with an expensive but very-high-quality 0.1% sample created by humans, rather than full coverage done by AI with some human input. My suspicion is that done right a human-AI combination is the way to go, but a small human dataset might be better than a large badly-AI-generated dataset.
And/or, that technique might be very useful for AIs generating/editing the synthetic data.
On the subject of cost, it’s also possible that we only need 10%, or 1%, or 0.1% of the dataset to illustrate the meaning of the
<AI>tag, and the majority or great majority of it can be in human mode. I’m fairly sure both that more will be better, and that there will be diminishing returns from adding more, so if the alignbment tax of doing the entire dataset is too high, investigating how good results we can get with a smaller proprtion would be worth it. I believe the Pretraining Language Models with Human Preferences paper simply did the entire training set, but they were using processing that was a lot cheper to do than what I’m proposing.Another possibility is that you’d do actually better with an expensive but very-high-quality 0.1% sample created by humans, rather than full coverage done by AI with some human input. My suspicion is that done right a human-AI combination is the way to go, but a small human dataset might be better than a large badly-AI-generated dataset.