It was an oversight to not include inference costs, but I need to highlight that this is a fermi estimate and from what I can see it isn’t enough of a difference to actually challenge the conclusion.
Do you happen to know what the inference costs are? I’ve only been able to find figures for revenue (page 29).
Do you think that number is high enough to undermine the general conclusion that there is billions of dollars of profit to be made from training the next SOTA model?
I also am not sure it is enough to change the conclusion, but I am pretty sure “put ChatGPT to Bing” doesn’t work as a business strategy due to inference cost. You seem to think otherwise, so I am interested in a discussion.
Dylan Patel’s SemiAnalysis is a well respected publication on business analysis of semiconductor industry. In The Inference Cost Of Search Disruption, he estimates the cost per query at 0.36 cents. He also wrote a sequel on cost structure of search business, which I recommend. Dylan also points out simply serving ChatGPT for every query at Google would require $100B in capital investment, which clearly dominates other expenditures. I think Dylan is broadly right, and if you think he is wrong, I am interested in your opinions where.
You’ve convinced me that it’s either too difficult to tell or (more likely) just completely incorrect. Thanks for the links and the comments.
Initially it was intended just to put the earlier estimate in perspective and check it wasn’t too crazy, but I see I “overextended” in making the claims about search.
It was an oversight to not include inference costs, but I need to highlight that this is a fermi estimate and from what I can see it isn’t enough of a difference to actually challenge the conclusion.
Do you happen to know what the inference costs are? I’ve only been able to find figures for revenue (page 29).
Do you think that number is high enough to undermine the general conclusion that there is billions of dollars of profit to be made from training the next SOTA model?
I also am not sure it is enough to change the conclusion, but I am pretty sure “put ChatGPT to Bing” doesn’t work as a business strategy due to inference cost. You seem to think otherwise, so I am interested in a discussion.
Inference cost is secret. The primary sources are OpenAI pricing table (ChatGPT 3.5 is 0.2 cents per 1000 tokens, GPT-4 is 30x more expensive, GPT-4 with long context is 60x more expensive), Twitter conversation between Elon Musk and Sam Altman on cost (“single-digits cents per chat” as of December 2022), and OpenAI’s claim of 90% cost reduction since December. From this I conclude OpenAI is selling API calls at cost or at loss, almost certainly not at profit.
Dylan Patel’s SemiAnalysis is a well respected publication on business analysis of semiconductor industry. In The Inference Cost Of Search Disruption, he estimates the cost per query at 0.36 cents. He also wrote a sequel on cost structure of search business, which I recommend. Dylan also points out simply serving ChatGPT for every query at Google would require $100B in capital investment, which clearly dominates other expenditures. I think Dylan is broadly right, and if you think he is wrong, I am interested in your opinions where.
You’ve convinced me that it’s either too difficult to tell or (more likely) just completely incorrect. Thanks for the links and the comments.
Initially it was intended just to put the earlier estimate in perspective and check it wasn’t too crazy, but I see I “overextended” in making the claims about search.