I also am not sure it is enough to change the conclusion, but I am pretty sure “put ChatGPT to Bing” doesn’t work as a business strategy due to inference cost. You seem to think otherwise, so I am interested in a discussion.
Dylan Patel’s SemiAnalysis is a well respected publication on business analysis of semiconductor industry. In The Inference Cost Of Search Disruption, he estimates the cost per query at 0.36 cents. He also wrote a sequel on cost structure of search business, which I recommend. Dylan also points out simply serving ChatGPT for every query at Google would require $100B in capital investment, which clearly dominates other expenditures. I think Dylan is broadly right, and if you think he is wrong, I am interested in your opinions where.
You’ve convinced me that it’s either too difficult to tell or (more likely) just completely incorrect. Thanks for the links and the comments.
Initially it was intended just to put the earlier estimate in perspective and check it wasn’t too crazy, but I see I “overextended” in making the claims about search.
I also am not sure it is enough to change the conclusion, but I am pretty sure “put ChatGPT to Bing” doesn’t work as a business strategy due to inference cost. You seem to think otherwise, so I am interested in a discussion.
Inference cost is secret. The primary sources are OpenAI pricing table (ChatGPT 3.5 is 0.2 cents per 1000 tokens, GPT-4 is 30x more expensive, GPT-4 with long context is 60x more expensive), Twitter conversation between Elon Musk and Sam Altman on cost (“single-digits cents per chat” as of December 2022), and OpenAI’s claim of 90% cost reduction since December. From this I conclude OpenAI is selling API calls at cost or at loss, almost certainly not at profit.
Dylan Patel’s SemiAnalysis is a well respected publication on business analysis of semiconductor industry. In The Inference Cost Of Search Disruption, he estimates the cost per query at 0.36 cents. He also wrote a sequel on cost structure of search business, which I recommend. Dylan also points out simply serving ChatGPT for every query at Google would require $100B in capital investment, which clearly dominates other expenditures. I think Dylan is broadly right, and if you think he is wrong, I am interested in your opinions where.
You’ve convinced me that it’s either too difficult to tell or (more likely) just completely incorrect. Thanks for the links and the comments.
Initially it was intended just to put the earlier estimate in perspective and check it wasn’t too crazy, but I see I “overextended” in making the claims about search.