I think a crux here is that I think that the synthetic data path is actually pretty helpful even from a capabilities perspective, because it lets you get much, much higher quality data than existing data, and most importantly in domains where you can abuse self-play like math or coding, you can get very, very high amounts of capability from synthetic data sources, so I think the synthetic data strategy has less capabilities taxes than a whole lot of alignment proposals on LW.
Importantly, we may well be able to automate the synthetic data alignment process in the near future, which would make it even less of a capabilities tax.
To be clear, just because it’s possible and solvable doesn’t mean it’s totally easy, we do still have our work cut out for us, it’s just that we’ve transformed it into a process where normal funding and science can actually solve the problem without further big breakthroughs/insights.
Then again, I do fear you might be right that they are under such competitive pressure, or at least value racing so highly that they will not slow down even a little, or at least not do any alignment work once superintelligence is reached.
I think a crux here is that I think that the synthetic data path is actually pretty helpful even from a capabilities perspective, because it lets you get much, much higher quality data than existing data, and most importantly in domains where you can abuse self-play like math or coding, you can get very, very high amounts of capability from synthetic data sources, so I think the synthetic data strategy has less capabilities taxes than a whole lot of alignment proposals on LW.
Importantly, we may well be able to automate the synthetic data alignment process in the near future, which would make it even less of a capabilities tax.
To be clear, just because it’s possible and solvable doesn’t mean it’s totally easy, we do still have our work cut out for us, it’s just that we’ve transformed it into a process where normal funding and science can actually solve the problem without further big breakthroughs/insights.
Then again, I do fear you might be right that they are under such competitive pressure, or at least value racing so highly that they will not slow down even a little, or at least not do any alignment work once superintelligence is reached.