At the time of my original comment, I had not looked at it.
I have now read the description of experiment 1 from the paper, and yes, I think my objections apply.
My best guess at the point you were trying to make by pointing me to this experiment is that you included some bidirectional examples in your test set, and therefore maybe the LLM should be able to figure out that your test set (in particular) is describing a symmetric relation, even if similar words in the LLM’s original training data were used to described asymmetric relations. Is that your implied argument?
Did you look at the design for our Experiment 1 in the paper? Do you think your objections to apply to that design?
At the time of my original comment, I had not looked at it.
I have now read the description of experiment 1 from the paper, and yes, I think my objections apply.
My best guess at the point you were trying to make by pointing me to this experiment is that you included some bidirectional examples in your test set, and therefore maybe the LLM should be able to figure out that your test set (in particular) is describing a symmetric relation, even if similar words in the LLM’s original training data were used to described asymmetric relations. Is that your implied argument?