I personally was mostly ignoring Transformers until GPT-3 came along. (That was mostly because I believed that “true AI” had to be a recurrent machine, whereas Transformers were feed-forward, and I did not understand that Transformers emulated RNNs when used in the autoregressive mode (the paper explaining that was published shortly after GPT-3).) So I thought, Transformers were remarkable, but the “true route to AGI” was elsewhere.
Then GPT-3 achieved two “Holy Grails” widely believed to be “impossible”/”long in the future”. Namely, few shot learning (so like a human it could learn a pattern from a single exposure without long training) and semi-competent program synthesis (which was considered to be next to impossible because of fragility of computer code to noise such as one-letter changes, with this supposed near-impossibility of program synthesis being the key obstacle to recursive self-improvement and foom).
These two breakthroughs were the key reason why I updated, rather than general linguistic competence (which was indeed quite impressive in 2019 models already).
ChatGPT was released on November 30 2022, so it’s only been around 7 months. The older ones were GPT-2 and GPT-3 which got attention among AI-followers but were relatively unknown to the public—and again, it wasn’t obvious then when or if ordinary people would come to know or care about these advances.
Someone else said similar about the basement possibility, which I did not know.
Interesting questions raised though: Even if it wasn’t clear until GPT, wouldn’t that still have left something like 2-3 years?
Granted that is not 10-20 years.
It seems we all, collectively, did not update nearly enough on ChatGPT-2?
I personally was mostly ignoring Transformers until GPT-3 came along. (That was mostly because I believed that “true AI” had to be a recurrent machine, whereas Transformers were feed-forward, and I did not understand that Transformers emulated RNNs when used in the autoregressive mode (the paper explaining that was published shortly after GPT-3).) So I thought, Transformers were remarkable, but the “true route to AGI” was elsewhere.
Then GPT-3 achieved two “Holy Grails” widely believed to be “impossible”/”long in the future”. Namely, few shot learning (so like a human it could learn a pattern from a single exposure without long training) and semi-competent program synthesis (which was considered to be next to impossible because of fragility of computer code to noise such as one-letter changes, with this supposed near-impossibility of program synthesis being the key obstacle to recursive self-improvement and foom).
These two breakthroughs were the key reason why I updated, rather than general linguistic competence (which was indeed quite impressive in 2019 models already).
ChatGPT was released on November 30 2022, so it’s only been around 7 months. The older ones were GPT-2 and GPT-3 which got attention among AI-followers but were relatively unknown to the public—and again, it wasn’t obvious then when or if ordinary people would come to know or care about these advances.