That’s a good question. I can see both the scenario of price increasing both more or less than that.
The compute needed for the training is in this example the only significant factor in price, and that’s the one that scales at 10x cost for 5x size. (However, sadly I can’t find the source where I read it, so again, please feel free to share if someone has a better method for estimation.)
Building the infrastructure for training a model of several trillion parameters could easily create a chip shortage, drastically increasing costs of AI chips, and thus leading to training costing way more than the estimate.
However, it might be possible that building a huge infrastructure would have many benefits of scale. For example Google might build a TPU “gigafactory” and because of the high volume of produced TPUs, the price per TPU would decrease significantly.
Building the infrastructure for training a model of several trillion parameters could easily create a chip shortage, drastically increasing costs of AI chips, and thus leading to training costing way more than the estimate.
Doing this slowly might be the sort of increased demand that would increase investment, and decrease price.
That’s a good question. I can see both the scenario of price increasing both more or less than that.
The compute needed for the training is in this example the only significant factor in price, and that’s the one that scales at 10x cost for 5x size. (However, sadly I can’t find the source where I read it, so again, please feel free to share if someone has a better method for estimation.)
Building the infrastructure for training a model of several trillion parameters could easily create a chip shortage, drastically increasing costs of AI chips, and thus leading to training costing way more than the estimate.
However, it might be possible that building a huge infrastructure would have many benefits of scale. For example Google might build a TPU “gigafactory” and because of the high volume of produced TPUs, the price per TPU would decrease significantly.
Doing this slowly might be the sort of increased demand that would increase investment, and decrease price.