Clément Dumas comments on What’s up with LLMs representing XORs of arbitrary features?

[Clément Dumas]

You can get ~75% just by computing the or. But we found that only at the last layer and step16000 of Pythia-70m training it achieves better than 75%, see this video