In the experiments I ran with GPT-2, RLACE and INLP are both used with a rank-1 projection. So RLACE could have “more impact” if it removed a more important direction, which I think it does.
I know it’s not the intended use of INLP, but I got my inspiration from this technique, and that’s why I write INLP (Ravfogel, 2020) (the original technique removes multiple directions to obtain a measurable effect)
[Edit] Tell me if you prefer that I avoid calling the “linear classifier method” INLP (it isn’t actually iterated in the experiments I ran, but it is where I discovered the idea of using a linear classifier to project data to remove information)!
In the experiments I ran with GPT-2, RLACE and INLP are both used with a rank-1 projection. So RLACE could have “more impact” if it removed a more important direction, which I think it does.
I know it’s not the intended use of INLP, but I got my inspiration from this technique, and that’s why I write INLP (Ravfogel, 2020) (the original technique removes multiple directions to obtain a measurable effect)
[Edit] Tell me if you prefer that I avoid calling the “linear classifier method” INLP (it isn’t actually iterated in the experiments I ran, but it is where I discovered the idea of using a linear classifier to project data to remove information)!