I’m very glad to see this post! Academic understanding of deep learning theory has improved quite a lot recently, and I think alignment research should be more aware of such results.
Some related results you may be interested in:
The actual NTKs of more realistic neural networks continuously change throughout their training process, which impacts learnability / generalization
I’m very glad to see this post! Academic understanding of deep learning theory has improved quite a lot recently, and I think alignment research should be more aware of such results.
Some related results you may be interested in:
The actual NTKs of more realistic neural networks continuously change throughout their training process, which impacts learnability / generalization
https://arxiv.org/abs/2008.00938
https://arxiv.org/abs/2106.06770
Discrete gradient descent leads to different training dynamics from the small step size / gradient flow regime
https://arxiv.org/abs/2009.11162
http://arxiv-export-lb.library.cornell.edu/abs/2107.09133