Makes sense. FWIW, based on Jan’s comments I think the main/only thing the OpenAI alignment team is aiming for here is i, differentially speeding up alignment research. It doesn’t seem like Jan believes in this plan; personally I don’t believe in this plan.
4. We want to focus on aspects of research work that are differentially helpful to alignment. However, most of our day-to-day work looks like pretty normal ML work, so it might be that we’ll see limited alignment research acceleration before ML research automation happens.
I don’t know how to link to the specific comment, but here somewhere. Also:
We can focus on tasks differentially useful to alignment research
Your pessimism about iii still seems a bit off to me. I agree that if you were coordinating well between all the actors than yeah you could just hold off on AI assistants. But the actual decision the OpenAI alignment team is facing could be more like “use LLMs to help with alignment research or get left behind when ML research gets automated”. If facing such choices I might produce a plan like theirs, but notably I would be much more pessimistic about it. When the universe limits you to one option, you shouldn’t expect it to be particularly good. The option “everybody agrees to not build AI assistants and we can do alignment research first” is maybe not on the table, or at least it probably doesn’t feel like it is to the alignment team at OpenAI.
Oh, I think I agree—if the choice is to use AI assistants or not, then use them. If they need adapting to be useful for alignment, then do adapt them.
But suppose they only work kind-of-poorly—and using them for alignment requires making progress on them (which will also be useful for capabilities), and you will not be able to keep those results internal. And that you can either do this work or do literally nothing. (Which is unrealistic.) Then I would say doing literally nothing is better. (Though it certainly feels bad, and probably costs you your job. So I guess some third option would be preferable.)
Makes sense. FWIW, based on Jan’s comments I think the main/only thing the OpenAI alignment team is aiming for here is i, differentially speeding up alignment research. It doesn’t seem like Jan believes in this plan; personally I don’t believe in this plan.
I don’t know how to link to the specific comment, but here somewhere. Also:
Your pessimism about iii still seems a bit off to me. I agree that if you were coordinating well between all the actors than yeah you could just hold off on AI assistants. But the actual decision the OpenAI alignment team is facing could be more like “use LLMs to help with alignment research or get left behind when ML research gets automated”. If facing such choices I might produce a plan like theirs, but notably I would be much more pessimistic about it. When the universe limits you to one option, you shouldn’t expect it to be particularly good. The option “everybody agrees to not build AI assistants and we can do alignment research first” is maybe not on the table, or at least it probably doesn’t feel like it is to the alignment team at OpenAI.
Oh, I think I agree—if the choice is to use AI assistants or not, then use them. If they need adapting to be useful for alignment, then do adapt them.
But suppose they only work kind-of-poorly—and using them for alignment requires making progress on them (which will also be useful for capabilities), and you will not be able to keep those results internal. And that you can either do this work or do literally nothing. (Which is unrealistic.) Then I would say doing literally nothing is better. (Though it certainly feels bad, and probably costs you your job. So I guess some third option would be preferable.)