TurnTrout comments on Steering GPT-2-XL by adding an activation vector

[TurnTrout]

Thanks for the feedback. Some related work was “hidden” in footnotes because, in an earlier version of the post, the related work was in the body and I wanted to decrease the time it took a reader to get to our results. The related work section is now basically consolidated into the appendix.

I also added another paragraph:

Lastly, Editing Models with Task Arithmetic explored a “dual” version of our activation additions. That work took vectors between weights before and after finetuning on a new task, and then added or subtracted task-specific weight-difference vectors. While this seems interesting, task arithmetic requires finetuning. In Activation additions have advantages over (RL/​supervised) finetuning, we explain the advantages our approach has over finetuning.