I think this discussion is too narrow and focused on just Sama and Microsoft.
The global market “wants” AGI, ASI, human obsolescence*.
The consequences of this event accelerate that:
Case 1: Microsoft bureaucracy drags Sama’s teams productivity down to zero. In this case, OpenAI doesn’t develop a GPT-5, and Microsoft doesn’t release a better model either. This opens up the market niche for the next competitor at a productive startup to develop the model, obviously assisted by former openAI employees who bring all the IP with them, and all the money and business flows to the startup. Brief delay in AGI, new frontrunner could be a firm with no EA influence and no safety culture beyond the legal minimum.
Case 2: Microsoft pours money in Sama’s group, Microsoft releases increasingly powerful models, bing and windows gain the market share they lost. Models become powerful enough that some real world incidents start to happen.
Conclusion: both outcomes benefit acceleration.
I think OAIs board understands the absolute requirement to remain with the best model in order to stay relevant, hence the panic merger proposal with anthropic.
*As an emergent property of the rules, aka Moloch. Note that chaos makes optimizing emergent outcomes more probable. See RNA experiments where thermal noise will cause RNA to self organize I to replicators.
While you have a point, I think this model might have too much faith in the Efficient Market Hypothesis.
It’s true that the market “wants” human obsolescence, in the sense that companies that sell it would earn a ton of money. But if DeepMind and Anthropic went bankrupt tomorrow, it’s not obvious that anyone would actually step up to fill the niche left by them.
Market failures are common, and the major AI Labs arguably sprung up because of ideological motives; not profit motives. Because of people buying into EA-style AI risk and deciding to go about solving it in a galaxy-brained manner.
They’re something like high-risk high-reward startups. And if a speculative startup fails, it’s not at all obvious that a replacement startup will spring up to fill its niche the very next day, even if the idea were solid. (Or, another analogy, they’re something like institutions concerned with fundamental theoretical research, and those famously need to be funded by governments, not the market.)
On the contrary, the reference point for how AGI-research-based-on-profit-motives goes is FAIR, and FAIR is lagging behind. The market, left to its own devices, will push for finding LLM applications or scaling them in the dumbest manner possible or such; not for racing for AGI.
I don’t have so little faith in the market as to completely discount your model, but I wouldn’t assign it 100% either.
Assume openAI was collecting a 10 percent profit margin (the other 90 percent paid for compute). Allegedly they burned about 540 million in 2022 when gpt-4 was developed. Call it 1 billion total cost for a gpt-4 (compute + staff compensation).
Then 130 million annual net revenue on a 1 billion investment is 13 percent ROI. In terms of “monetary free energy” that’s net output. Large businesses exist that run on less margin.
I am not specced in economics or finance but that looks like a sustainable business, and it’s obviously self amplifying. Assuming the nascent general ai* industry has easy short term potential growth (as in, companies can rent access to a good model and collect many more billions) then it’s possibly self sustaining. Even without outside investment some of those profits would get invested into the next model, and so on.
You can also think of a second form of “revenue” as investor hype. Fresh hype is created with each major improvement in the model that investors can perceive, and they bring more money each round.
While yes the EMH is imperfect,* investors clearly see enormous profit in the short term from general ai. This is the main thing that will drive the general ai industry to whatever is technically achievable : tens of billions in outside money. And yes, blowing up openAI slows this down...but the investors who were willing to give openAI tens of billions in a few weeks still have their money. Where’s it going to go next?
By general ai I mean a model that is general purpose with many customers without needing expensive modifications. Different from AGI, which means also human level capabilities. General ai appears, from the data above, to be financially self sustaining even if it is not human level.
*As other lesswrong posts point out, downside risks like nuclear wars or markets ceasing to exist because a rampant ASI ate everything would not be “priced in”.
I am not specced in economics or finance but that looks like a sustainable business, and it’s obviously self amplifying
“Make a giant LLM and deploy it” is a self-sustaining business, yes, and if all major AI Labs died tomorrow a plethora of companies filling the niche of “make a giant LLM and deploy it” would spring up, yes.
“Re-invest revenue into making an even larger LLM” is a sensible company policy, as well.
But is “have a roadmap to AGI and invest into research that brings your models closer to it, even if that doesn’t immediately translate into revenue” a self-sustaining business model? I’m much less confident on that. It’s possible that the ideas of “AGI = money” have already propagated enough that profit-oriented non-imaginative business people would decide to spend their revenue on that. But that’s not obvious to me.
I expect the non-ideologically-motivated replacements for the current major AI labs to just have no idea what “racing to AGI” even means, in terms of “what research directions to pursue” as opposed to “what utterances to emit”. The current AI industry as a whole is pretty bad at it as-is, but the major AI labs explicitly have some vision of what it physically means. I don’t expect the replacements for them that the market would generate on its own to have even that much of a sense of direction.
Again, it’s possible that it’s no longer true, that the ideas propagated enough that some of the “native” replacement companies would competently race as well. But it’s not an open-and-shut case, I think.
So your model is that people can make big llms, and the innovation from openAI and from open source will eventually all be in one large model. Aka “gpt 4.1”. But that each llm shop, while free of encumbrances and free to seek maximum profit, would not have the necessary concentration of money and talent in one place to develop AGI.
Instead they would simply keep making smaller delta’s to their product, something a less talented and GPU poorer crew could do, and capabilities would be stuck in a local minimum.
So you believe that either this would push back AGI several years (eventually the staff at these smaller shops would skill up from experience and as compute gets cheaper they would eventually have what 100B of compute will buy in 2024) or possibly longer if there is no smooth path of small incremental steps from gpt-4.1 to AGI.
I will add one comment to this : it’s not actually a threshold of “gpt4.1 to AGI”. Assuming you believe RSI will work, you need “a good enough seed model plus sufficient compute to train and benchmark thousands of automatically generated AGI candidates”.
Gpt4.1 plus a reinforcement learning element might be enough for the “seed AI”.
Giant LLMs are as useful as they are agentic (with ability to remain aware of a specific large body of data and keep usefully chipping away at a task), which doesn’t seem particularly different from AGI as a direction (at least while it hasn’t yet been walked far enough to tell the difference). The distinction is in AGI being a particular crucial threshold of capability that local pursuit of better agentic LLMs will ignore until it’s crossed.
Assume openAI was collecting a 10 percent profit margin (the other 90 percent paid for compute).
Wait, is OpenAI net positive profit on marginal users? I had assumed probably not, although it’s not particularly central to any of my models right at the moment.
As Zvi mentioned in one of the roundups, the conventional wisdom for entering a new monopolistic tech niche is to grow as fast as possible.
So it’s likely that OpenAI loses money per user. GitHub copilot allegedly costs $40 in compute per $20 a month subscriber.
So yes, you are right, but no, it doesn’t matter. This is because there’s other variables. The cost of compute is driven up by outside investment. If somehow dynamiting openAI causes all the outside investors to go invest somewhere else—sort of like the hype cycles for nft or crypto—the cost of compute would drop.
For example Nvidia is estimated to pay $3000 to build each H100. If Nvidia charges $5000 a card, and stops charging a 20 percent software license fee, that essentially cuts the compute cost by more than half*, making current AI models at current prices more than profitable.
Nvidia would do this in the hypothetical world of “investors get bored and another ai winter begins”. This neglects Nvidia reducing their costs and developing a cheaper to build card per unit of LLM performance, which they obviously are doing.
*Quick and dirty sanity check: assuming 50 percent utilization (GPU is bounded by memory I/o then it would use $33,000 in electricity over 5 years and currently costs $50,000 at current prices, 25k is list price, 25k is license fee. Were Nvidia to simply charge a more modest margin the all in cost would drop from 83k to 38k. Data center electricity is probably cheaper than 13 cents but there are costs for backup power and other systems)
Conclusion: what’s different now is general ai is bringing in enough revenue to be a self sustaining business. It’s not an industry that can fold and go dormant like failed tech startups that folded and the product or service they developed ceased to be available anywhere.
The time to blow up openAI was prior to the release of chatGPT.
I think this discussion is too narrow and focused on just Sama and Microsoft.
The global market “wants” AGI, ASI, human obsolescence*.
The consequences of this event accelerate that:
Case 1: Microsoft bureaucracy drags Sama’s teams productivity down to zero. In this case, OpenAI doesn’t develop a GPT-5, and Microsoft doesn’t release a better model either. This opens up the market niche for the next competitor at a productive startup to develop the model, obviously assisted by former openAI employees who bring all the IP with them, and all the money and business flows to the startup. Brief delay in AGI, new frontrunner could be a firm with no EA influence and no safety culture beyond the legal minimum.
Case 2: Microsoft pours money in Sama’s group, Microsoft releases increasingly powerful models, bing and windows gain the market share they lost. Models become powerful enough that some real world incidents start to happen.
Conclusion: both outcomes benefit acceleration.
I think OAIs board understands the absolute requirement to remain with the best model in order to stay relevant, hence the panic merger proposal with anthropic.
*As an emergent property of the rules, aka Moloch. Note that chaos makes optimizing emergent outcomes more probable. See RNA experiments where thermal noise will cause RNA to self organize I to replicators.
While you have a point, I think this model might have too much faith in the Efficient Market Hypothesis.
It’s true that the market “wants” human obsolescence, in the sense that companies that sell it would earn a ton of money. But if DeepMind and Anthropic went bankrupt tomorrow, it’s not obvious that anyone would actually step up to fill the niche left by them.
Market failures are common, and the major AI Labs arguably sprung up because of ideological motives; not profit motives. Because of people buying into EA-style AI risk and deciding to go about solving it in a galaxy-brained manner.
They’re something like high-risk high-reward startups. And if a speculative startup fails, it’s not at all obvious that a replacement startup will spring up to fill its niche the very next day, even if the idea were solid. (Or, another analogy, they’re something like institutions concerned with fundamental theoretical research, and those famously need to be funded by governments, not the market.)
On the contrary, the reference point for how AGI-research-based-on-profit-motives goes is FAIR, and FAIR is lagging behind. The market, left to its own devices, will push for finding LLM applications or scaling them in the dumbest manner possible or such; not for racing for AGI.
I don’t have so little faith in the market as to completely discount your model, but I wouldn’t assign it 100% either.
Were the ai startups high risk and dependent on investor largess to exist, then absolutely I would agree with your model.
But : https://sacra.com/c/openai/#:~:text=OpenAI has surpassed %241.3B,at the end of 2022.
Assume openAI was collecting a 10 percent profit margin (the other 90 percent paid for compute). Allegedly they burned about 540 million in 2022 when gpt-4 was developed. Call it 1 billion total cost for a gpt-4 (compute + staff compensation).
Then 130 million annual net revenue on a 1 billion investment is 13 percent ROI. In terms of “monetary free energy” that’s net output. Large businesses exist that run on less margin.
I am not specced in economics or finance but that looks like a sustainable business, and it’s obviously self amplifying. Assuming the nascent general ai* industry has easy short term potential growth (as in, companies can rent access to a good model and collect many more billions) then it’s possibly self sustaining. Even without outside investment some of those profits would get invested into the next model, and so on.
You can also think of a second form of “revenue” as investor hype. Fresh hype is created with each major improvement in the model that investors can perceive, and they bring more money each round.
While yes the EMH is imperfect,* investors clearly see enormous profit in the short term from general ai. This is the main thing that will drive the general ai industry to whatever is technically achievable : tens of billions in outside money. And yes, blowing up openAI slows this down...but the investors who were willing to give openAI tens of billions in a few weeks still have their money. Where’s it going to go next?
By general ai I mean a model that is general purpose with many customers without needing expensive modifications. Different from AGI, which means also human level capabilities. General ai appears, from the data above, to be financially self sustaining even if it is not human level.
*As other lesswrong posts point out, downside risks like nuclear wars or markets ceasing to exist because a rampant ASI ate everything would not be “priced in”.
“Make a giant LLM and deploy it” is a self-sustaining business, yes, and if all major AI Labs died tomorrow a plethora of companies filling the niche of “make a giant LLM and deploy it” would spring up, yes.
“Re-invest revenue into making an even larger LLM” is a sensible company policy, as well.
But is “have a roadmap to AGI and invest into research that brings your models closer to it, even if that doesn’t immediately translate into revenue” a self-sustaining business model? I’m much less confident on that. It’s possible that the ideas of “AGI = money” have already propagated enough that profit-oriented non-imaginative business people would decide to spend their revenue on that. But that’s not obvious to me.
I expect the non-ideologically-motivated replacements for the current major AI labs to just have no idea what “racing to AGI” even means, in terms of “what research directions to pursue” as opposed to “what utterances to emit”. The current AI industry as a whole is pretty bad at it as-is, but the major AI labs explicitly have some vision of what it physically means. I don’t expect the replacements for them that the market would generate on its own to have even that much of a sense of direction.
Again, it’s possible that it’s no longer true, that the ideas propagated enough that some of the “native” replacement companies would competently race as well. But it’s not an open-and-shut case, I think.
So your model is that people can make big llms, and the innovation from openAI and from open source will eventually all be in one large model. Aka “gpt 4.1”. But that each llm shop, while free of encumbrances and free to seek maximum profit, would not have the necessary concentration of money and talent in one place to develop AGI.
Instead they would simply keep making smaller delta’s to their product, something a less talented and GPU poorer crew could do, and capabilities would be stuck in a local minimum.
So you believe that either this would push back AGI several years (eventually the staff at these smaller shops would skill up from experience and as compute gets cheaper they would eventually have what 100B of compute will buy in 2024) or possibly longer if there is no smooth path of small incremental steps from gpt-4.1 to AGI.
I will add one comment to this : it’s not actually a threshold of “gpt4.1 to AGI”. Assuming you believe RSI will work, you need “a good enough seed model plus sufficient compute to train and benchmark thousands of automatically generated AGI candidates”.
Gpt4.1 plus a reinforcement learning element might be enough for the “seed AI”.
That summary sounds right, yep!
Except that. It might, but I don’t think that’s particularly likely.
Giant LLMs are as useful as they are agentic (with ability to remain aware of a specific large body of data and keep usefully chipping away at a task), which doesn’t seem particularly different from AGI as a direction (at least while it hasn’t yet been walked far enough to tell the difference). The distinction is in AGI being a particular crucial threshold of capability that local pursuit of better agentic LLMs will ignore until it’s crossed.
Wait, is OpenAI net positive profit on marginal users? I had assumed probably not, although it’s not particularly central to any of my models right at the moment.
As Zvi mentioned in one of the roundups, the conventional wisdom for entering a new monopolistic tech niche is to grow as fast as possible.
So it’s likely that OpenAI loses money per user. GitHub copilot allegedly costs $40 in compute per $20 a month subscriber.
So yes, you are right, but no, it doesn’t matter. This is because there’s other variables. The cost of compute is driven up by outside investment. If somehow dynamiting openAI causes all the outside investors to go invest somewhere else—sort of like the hype cycles for nft or crypto—the cost of compute would drop.
For example Nvidia is estimated to pay $3000 to build each H100. If Nvidia charges $5000 a card, and stops charging a 20 percent software license fee, that essentially cuts the compute cost by more than half*, making current AI models at current prices more than profitable.
Nvidia would do this in the hypothetical world of “investors get bored and another ai winter begins”. This neglects Nvidia reducing their costs and developing a cheaper to build card per unit of LLM performance, which they obviously are doing.
*Quick and dirty sanity check: assuming 50 percent utilization (GPU is bounded by memory I/o then it would use $33,000 in electricity over 5 years and currently costs $50,000 at current prices, 25k is list price, 25k is license fee. Were Nvidia to simply charge a more modest margin the all in cost would drop from 83k to 38k. Data center electricity is probably cheaper than 13 cents but there are costs for backup power and other systems)
Conclusion: what’s different now is general ai is bringing in enough revenue to be a self sustaining business. It’s not an industry that can fold and go dormant like failed tech startups that folded and the product or service they developed ceased to be available anywhere.
The time to blow up openAI was prior to the release of chatGPT.