I take a critical tone here in an effort to cut that hype off at the pass.
Maybe this is just my AI safety focus, or something, but I find myself annoyed by ‘hype management’ more often than not; I think the underlying root cause of the frustration is that it’s easier to reach agreement on object-level details than interpretations, which are themselves easier than interpretations of interpretations.
Like, when I heard “GPT-3”, I thought “like GPT-2, except one more,” and from what I can tell that expectation is roughly accurate. The post agrees, and notes that since “one” doesn’t correspond to anything here, the main thing this tells you is that this transformer paper came from people who feel like they own the GPT name instead of people who don’t feel that. It sounds like you expected “GPT” to mean something more like “paradigm-breaker” and so you were disappointed, but this feels like a ding on your expectations more than a ding on the paper.
But under the hype management goal, the question of whether we should celebrate it as “as predicted, larger models continue to perform better, and astoundingly 175B parameters for the amount of training we did still hasn’t converged” or criticize it as “oh, it is a mere confirmation of a prediction widely suspected” isn’t a question of what’s in the paper (as neither disagree), or even your personal take, but what you expect the social distribution of takes is, so that your statement is the right pull on the group beliefs.
---
Maybe putting this another way, when I view this as “nostalgebraist the NLP expert who is following and sharing his own research taste”, I like the post, as expert taste is useful even if you the reader disagree; and when I view it as “nostalgebraist the person who has goals for social epistemology around NLP” I like it less.
I agree with you about hype management in general, I think. The following does seem like a point of concrete disagreement:
It sounds like you expected “GPT” to mean something more like “paradigm-breaker” and so you were disappointed, but this feels like a ding on your expectations more than a ding on the paper.
If the paper had not done few-shot learning, and had just reviewed LM task performance / generation quality / zero-shot (note that zero-shot scales up well too!), I would agree with you.
However, as I read the paper, it touts few-shot as this new, exciting capability that only properly emerges at the new scale. I expected that, if any given person found the paper impressive, it would be for this purported newness and not only “LM scaling continues,” and this does seem to be the case (e.g. gwern, dxu). So there is a real, object-level dispute over the extent to which this is a qualitative jump.
I’m not sure I have concrete social epistemology goals except “fewer false beliefs” -- that is, I am concerned with group beliefs, but only because they point to which truths will be most impactful to voice. I predicted people would be overly impressed with few-shot, and I wanted to counter that. Arguably I should have concentrated less on “does this deserve the title GPT-3?” and more heavily on few-shot, as I’ve done more recently.
Maybe this is just my AI safety focus, or something, but I find myself annoyed by ‘hype management’ more often than not; I think the underlying root cause of the frustration is that it’s easier to reach agreement on object-level details than interpretations, which are themselves easier than interpretations of interpretations.
Like, when I heard “GPT-3”, I thought “like GPT-2, except one more,” and from what I can tell that expectation is roughly accurate. The post agrees, and notes that since “one” doesn’t correspond to anything here, the main thing this tells you is that this transformer paper came from people who feel like they own the GPT name instead of people who don’t feel that. It sounds like you expected “GPT” to mean something more like “paradigm-breaker” and so you were disappointed, but this feels like a ding on your expectations more than a ding on the paper.
But under the hype management goal, the question of whether we should celebrate it as “as predicted, larger models continue to perform better, and astoundingly 175B parameters for the amount of training we did still hasn’t converged” or criticize it as “oh, it is a mere confirmation of a prediction widely suspected” isn’t a question of what’s in the paper (as neither disagree), or even your personal take, but what you expect the social distribution of takes is, so that your statement is the right pull on the group beliefs.
---
Maybe putting this another way, when I view this as “nostalgebraist the NLP expert who is following and sharing his own research taste”, I like the post, as expert taste is useful even if you the reader disagree; and when I view it as “nostalgebraist the person who has goals for social epistemology around NLP” I like it less.
I agree with you about hype management in general, I think. The following does seem like a point of concrete disagreement:
If the paper had not done few-shot learning, and had just reviewed LM task performance / generation quality / zero-shot (note that zero-shot scales up well too!), I would agree with you.
However, as I read the paper, it touts few-shot as this new, exciting capability that only properly emerges at the new scale. I expected that, if any given person found the paper impressive, it would be for this purported newness and not only “LM scaling continues,” and this does seem to be the case (e.g. gwern, dxu). So there is a real, object-level dispute over the extent to which this is a qualitative jump.
I’m not sure I have concrete social epistemology goals except “fewer false beliefs” -- that is, I am concerned with group beliefs, but only because they point to which truths will be most impactful to voice. I predicted people would be overly impressed with few-shot, and I wanted to counter that. Arguably I should have concentrated less on “does this deserve the title GPT-3?” and more heavily on few-shot, as I’ve done more recently.