On the other hand, why did news reports[1] suggest that Google was caught flat-footed by ChatGPT and re-oriented to rush Bard to market?
My sense is that Google/DeepMind’s lethargy in the area of language models is due to a combination of a few factors:
They’ve diversified their bets to include things like protein folding, fusion plasma control, etc. which are more application-driven and not on an AGI path.
They’ve focused more on fundamental research and less on productizing and scaling.
Their language model experts might have a somewhat high annual attrition rate.
I just looked up the authors on Google Brain’s Attention is All You Need, and all but one have left Google after 5.25 years, many for startups, and one for OpenAI. That works out to an annual attrition of 33%.
For DeepMind’s Chinchilla paper, 6 of 22 researchers have been lost in 1 year: 4 to OpenAI and 2 to startups. That’s 27% annual attrition.
By contrast, 16 or 17 of the 30 authors on the GPT-3 paper seem to still be at OpenAI, 2.75 years later, which works out to 20% annual attrition. Notably, of those who have left, not a one has left for Google or DeepMind, though interestingly, 8 have left for Anthropic. (Admittedly, this somewhat reflects the relative newness and growth rates of Google/DeepMind, OpenAI, and Anthropic, since a priori we expect more migration from slow-growing orgs to fast-growing orgs than vice versa.)
It’s broadly reported that Google as an organization struggles with stifling bureaucracy and a lack of urgency. (This was also my observation working there more than ten years ago, and I expect it’s gotten worse since.)
OpenAI seems to also have been caught flat-footed by ChatGPT, or more specifically by the success it got. It seems like the success came largely from the chat interface that made it intuitive for people on the street to use—and none of the LLM techies at any company realized what a difference that would make.
Yes, although the chat interface was necessary but insufficient. They also needed a capable language model behind it, which OpenAI already had, and Google still lacks months later.
I think talking about Google/DeepMind as a unitary entity is a mistake. I’m gonna guess that Peter agrees, and that’s why he specified DeepMind. Google’s publications identify at least two internal language models superior to Lambda, so their release of Bard based on Lambda doesn’t tell us much. They are certainly behind in commercializing chatbots, but is that a weak claim. How DeepMind compares to OpenAI is difficult. Four people going to OpenAI is damning, though.
A somewhat reliable source has told me that they don’t have the compute infrastructure to support making a more advanced model available to users.
That might also reflect limited engineering efforts to optimize state-of-the-art models for real world usage (think of the performance gains from GPT-3.5 Turbo) as opposed to hitting benchmarks for a paper to be published.
I agree that those are possibilities.
On the other hand, why did news reports[1] suggest that Google was caught flat-footed by ChatGPT and re-oriented to rush Bard to market?
My sense is that Google/DeepMind’s lethargy in the area of language models is due to a combination of a few factors:
They’ve diversified their bets to include things like protein folding, fusion plasma control, etc. which are more application-driven and not on an AGI path.
They’ve focused more on fundamental research and less on productizing and scaling.
Their language model experts might have a somewhat high annual attrition rate.
I just looked up the authors on Google Brain’s Attention is All You Need, and all but one have left Google after 5.25 years, many for startups, and one for OpenAI. That works out to an annual attrition of 33%.
For DeepMind’s Chinchilla paper, 6 of 22 researchers have been lost in 1 year: 4 to OpenAI and 2 to startups. That’s 27% annual attrition.
By contrast, 16 or 17 of the 30 authors on the GPT-3 paper seem to still be at OpenAI, 2.75 years later, which works out to 20% annual attrition. Notably, of those who have left, not a one has left for Google or DeepMind, though interestingly, 8 have left for Anthropic. (Admittedly, this somewhat reflects the relative newness and growth rates of Google/DeepMind, OpenAI, and Anthropic, since a priori we expect more migration from slow-growing orgs to fast-growing orgs than vice versa.)
It’s broadly reported that Google as an organization struggles with stifling bureaucracy and a lack of urgency. (This was also my observation working there more than ten years ago, and I expect it’s gotten worse since.)
e.g. this from the New York Times
OpenAI seems to also have been caught flat-footed by ChatGPT, or more specifically by the success it got. It seems like the success came largely from the chat interface that made it intuitive for people on the street to use—and none of the LLM techies at any company realized what a difference that would make.
Yes, although the chat interface was necessary but insufficient. They also needed a capable language model behind it, which OpenAI already had, and Google still lacks months later.
I think talking about Google/DeepMind as a unitary entity is a mistake. I’m gonna guess that Peter agrees, and that’s why he specified DeepMind. Google’s publications identify at least two internal language models superior to Lambda, so their release of Bard based on Lambda doesn’t tell us much. They are certainly behind in commercializing chatbots, but is that a weak claim. How DeepMind compares to OpenAI is difficult. Four people going to OpenAI is damning, though.
A somewhat reliable source has told me that they don’t have the compute infrastructure to support making a more advanced model available to users.
That might also reflect limited engineering efforts to optimize state-of-the-art models for real world usage (think of the performance gains from GPT-3.5 Turbo) as opposed to hitting benchmarks for a paper to be published.