Not commenting on this whole thread, which I do have a lot of takes about that I am still processing, but a quick comment on this line:
The hardware advances are about to peter out, and scaling up the spend on supercomputer training by another 100x from ~$1B is not really an option.
I don’t see any reason for why we wouldn’t see a $100B training run within the next few years. $100B is not that much (it’s roughly a third of Google’s annual revenue, so if they really see competition in this domain as an existential threat, they alone might be able to fund a training run like this).
It might have to involve some collaboration of multiple tech companies, or some government involvement, but I currently expect that if scaling continues to work, we are going to see a $100B training run (though like, this stuff is super hard to forecast, so I am more like 60% on this, and also wouldn’t be surprised if it didn’t happen).
In retrospect I actually somewhat agree with you so I edited that line and denoted with a strike-through. Yes a $100B training run is an option in theory, but it is unlikely to translate to a 100x increase in training compute due to datacenter scaling difficulties, and this is also greater than OpenAI’s estimated market cap. (I also added a note with a quick fermi estimate showing that a training run of that size would require massively increasing nvidia’s GPU output by at least an OOM) For various reasons I expect even those with pockets that deep to instead invest more in a number of GPT4 size runs exploring alternate training paths.
Not commenting on this whole thread, which I do have a lot of takes about that I am still processing, but a quick comment on this line:
I don’t see any reason for why we wouldn’t see a $100B training run within the next few years. $100B is not that much (it’s roughly a third of Google’s annual revenue, so if they really see competition in this domain as an existential threat, they alone might be able to fund a training run like this).
It might have to involve some collaboration of multiple tech companies, or some government involvement, but I currently expect that if scaling continues to work, we are going to see a $100B training run (though like, this stuff is super hard to forecast, so I am more like 60% on this, and also wouldn’t be surprised if it didn’t happen).
In retrospect I actually somewhat agree with you so I edited that line and denoted with a strike-through. Yes a $100B training run is an option in theory, but it is unlikely to translate to a 100x increase in training compute due to datacenter scaling difficulties, and this is also greater than OpenAI’s estimated market cap. (I also added a note with a quick fermi estimate showing that a training run of that size would require massively increasing nvidia’s GPU output by at least an OOM) For various reasons I expect even those with pockets that deep to instead invest more in a number of GPT4 size runs exploring alternate training paths.