Economic cost-benefit analysis of training SOTA model seems entirely wrong to me.
If training a new SOTA model enabled a company to gain just a fraction of the global search market
This is admirably concrete, so I will use this, but the point generalizes. This assumes Microsoft can gain (and keep) 1% of the global search market by spending $1B USD and training a new SOTA model, which is obviously false? After training a new SOTA model, they need to deploy it for inference, and inference cost dominates training cost. The analysis seems to assume inference cost is negligible, but that’s just not true for search engine case which requires wide deployment. The analysis should either give an example of economic gain that does not require wide deployment (things like stock picking comes to mind), or should analyze inference cost, at the very least it should not assume inference cost is approximately zero.
It was an oversight to not include inference costs, but I need to highlight that this is a fermi estimate and from what I can see it isn’t enough of a difference to actually challenge the conclusion.
Do you happen to know what the inference costs are? I’ve only been able to find figures for revenue (page 29).
Do you think that number is high enough to undermine the general conclusion that there is billions of dollars of profit to be made from training the next SOTA model?
I also am not sure it is enough to change the conclusion, but I am pretty sure “put ChatGPT to Bing” doesn’t work as a business strategy due to inference cost. You seem to think otherwise, so I am interested in a discussion.
Dylan Patel’s SemiAnalysis is a well respected publication on business analysis of semiconductor industry. In The Inference Cost Of Search Disruption, he estimates the cost per query at 0.36 cents. He also wrote a sequel on cost structure of search business, which I recommend. Dylan also points out simply serving ChatGPT for every query at Google would require $100B in capital investment, which clearly dominates other expenditures. I think Dylan is broadly right, and if you think he is wrong, I am interested in your opinions where.
You’ve convinced me that it’s either too difficult to tell or (more likely) just completely incorrect. Thanks for the links and the comments.
Initially it was intended just to put the earlier estimate in perspective and check it wasn’t too crazy, but I see I “overextended” in making the claims about search.
Economic cost-benefit analysis of training SOTA model seems entirely wrong to me.
This is admirably concrete, so I will use this, but the point generalizes. This assumes Microsoft can gain (and keep) 1% of the global search market by spending $1B USD and training a new SOTA model, which is obviously false? After training a new SOTA model, they need to deploy it for inference, and inference cost dominates training cost. The analysis seems to assume inference cost is negligible, but that’s just not true for search engine case which requires wide deployment. The analysis should either give an example of economic gain that does not require wide deployment (things like stock picking comes to mind), or should analyze inference cost, at the very least it should not assume inference cost is approximately zero.
It was an oversight to not include inference costs, but I need to highlight that this is a fermi estimate and from what I can see it isn’t enough of a difference to actually challenge the conclusion.
Do you happen to know what the inference costs are? I’ve only been able to find figures for revenue (page 29).
Do you think that number is high enough to undermine the general conclusion that there is billions of dollars of profit to be made from training the next SOTA model?
I also am not sure it is enough to change the conclusion, but I am pretty sure “put ChatGPT to Bing” doesn’t work as a business strategy due to inference cost. You seem to think otherwise, so I am interested in a discussion.
Inference cost is secret. The primary sources are OpenAI pricing table (ChatGPT 3.5 is 0.2 cents per 1000 tokens, GPT-4 is 30x more expensive, GPT-4 with long context is 60x more expensive), Twitter conversation between Elon Musk and Sam Altman on cost (“single-digits cents per chat” as of December 2022), and OpenAI’s claim of 90% cost reduction since December. From this I conclude OpenAI is selling API calls at cost or at loss, almost certainly not at profit.
Dylan Patel’s SemiAnalysis is a well respected publication on business analysis of semiconductor industry. In The Inference Cost Of Search Disruption, he estimates the cost per query at 0.36 cents. He also wrote a sequel on cost structure of search business, which I recommend. Dylan also points out simply serving ChatGPT for every query at Google would require $100B in capital investment, which clearly dominates other expenditures. I think Dylan is broadly right, and if you think he is wrong, I am interested in your opinions where.
You’ve convinced me that it’s either too difficult to tell or (more likely) just completely incorrect. Thanks for the links and the comments.
Initially it was intended just to put the earlier estimate in perspective and check it wasn’t too crazy, but I see I “overextended” in making the claims about search.