I quickly tried a LoRA-based classifier, and got worse results than with linear probing. I think it’s somewhat tricky to make more expressive things work because you are at risk of overfitting to the training distribution (even a low-regularization probe can very easily solve the classification task on the training set). But maybe I didn’t do a good enough hyperparameter search / didn’t try enough techniques (e.g. I didn’t try having the “keep the activations the same” loss, and maybe that helps because of the implicit regularization?).
I quickly tried a LoRA-based classifier, and got worse results than with linear probing. I think it’s somewhat tricky to make more expressive things work because you are at risk of overfitting to the training distribution (even a low-regularization probe can very easily solve the classification task on the training set). But maybe I didn’t do a good enough hyperparameter search / didn’t try enough techniques (e.g. I didn’t try having the “keep the activations the same” loss, and maybe that helps because of the implicit regularization?).