Ofer comments on Implications of GPT-2

[Ofer]

And this sounds like goal post moving:

I’m failing to see a goal-post-moving between me writing:

It’s a cool language model but can it do even modest logic-related stuff without similar examples in the training data?

and then later writing (in reply to your comment quoting that sentence):

unless a very similar problem appears in the training data—e.g. the exact same problem but with different labels

If I’m missing something I’d be grateful for a further explanation.

[gwern]

In what sense is being able to do addition or subtraction with different numbers, for example, which is what it means to learn addition or subtraction, not ‘the exact same problem but with different labels’?

[Ofer]

Thank you for clarifying!

FWIW, when I wrote “the exact same problem but with different labels” I meant “the exact same problem but with different arbitrary names for entities”.

For example, I would consider the following two problems to be “the exact same problem but with different labels”:

“X+1=2 therefore X=”

“Y+1=2 therefore Y=”

But NOT the following two problems:

“1+1=”

“1+2=”