Shouldn’t diagonal elements in the perplexity table all be equal to the baseline (since the addition should be 0)?
I’m a bit confused about the use of perplexity here. The added vector introduces bias (away from one digit and towards another). It shouldn’t be surprising that perplexity increases? Eyeballing the visualizations they do all seem to shift mass away from b and towards a.
You’re right, however the results from the Steering GPT-2-XL post showed that in GPT-2-XL, similar modifications had very little effect on model perplexity. The patched model also doesn’t only shift weight from b to a. It also has wonky effects on other digits. For example, in the 3-1 patch for input 4, the weight given to 9 very much increased. More interestingly, it is not too uncommon to find examples which cause seemingly random digits to suddenly become the most likely. The 1-8 patch for input 9 is an example:
I have a couple of basic questions:
Shouldn’t diagonal elements in the perplexity table all be equal to the baseline (since the addition should be 0)?
I’m a bit confused about the use of perplexity here. The added vector introduces bias (away from one digit and towards another). It shouldn’t be surprising that perplexity increases? Eyeballing the visualizations they do all seem to shift mass away from
band towardsa.Yup. You should be able to see this in the chart.
You’re right, however the results from the Steering GPT-2-XL post showed that in GPT-2-XL, similar modifications had very little effect on model perplexity. The patched model also doesn’t only shift weight from
btoa. It also has wonky effects on other digits. For example, in the 3-1 patch for input 4, the weight given to 9 very much increased. More interestingly, it is not too uncommon to find examples which cause seemingly random digits to suddenly become the most likely. The 1-8 patch for input 9 is an example: