But depending on what you count: we had scaling laws for deep learning back in 2017, or at least 2020. I know people who were really paying attention; who really saw it; who really bet.
Interestingly, I feel like the evidence we got from public info about scaling laws at the time was consistent with long timelines. In roughly 2018-2021, I remember at least a few people making approximately the following argument:
(1) OpenAI came out with a blog post in 2018 claiming that training compute was doubling every 3.4 months.
(2) Extrapolating this trend indicates that training runs will start costing around $1 trillion dollars by 2025.
(3) Therefore, this trend cannot be sustained beyond 2025. Unless AGI arrives before 2025, we will soon enter an AI winter.
However, it turned out that OpenAI was likely wrong about the compute trend, and training compute was doubling roughly every 6-10 months, not every 3.4 months. And moreover the Kaplan et al. scaling laws turned out to be inaccurate too. This was a big update within the scaling hypothesis paradigm, since it demonstrated that we were getting better returns to compute than we thought.
Interestingly, I feel like the evidence we got from public info about scaling laws at the time was consistent with long timelines. In roughly 2018-2021, I remember at least a few people making approximately the following argument:
(1) OpenAI came out with a blog post in 2018 claiming that training compute was doubling every 3.4 months.
(2) Extrapolating this trend indicates that training runs will start costing around $1 trillion dollars by 2025.
(3) Therefore, this trend cannot be sustained beyond 2025. Unless AGI arrives before 2025, we will soon enter an AI winter.
However, it turned out that OpenAI was likely wrong about the compute trend, and training compute was doubling roughly every 6-10 months, not every 3.4 months. And moreover the Kaplan et al. scaling laws turned out to be inaccurate too. This was a big update within the scaling hypothesis paradigm, since it demonstrated that we were getting better returns to compute than we thought.
It was indeed a good point, at the time, but still, the people who predicted short timelines were right and deserve credit.
Definitely. I don’t think it makes much sense to give people credit for being wrong for legible reasons.