I’m kinda opposite on the timelines thing? This is probably a timeline delayer even if I thought LLM’s scaled to AGI, which I don’t but let’s play along.
If a Pharma company could look at another company’s product and copy it and release it for free with no consequences, but the product they release itself could only be marginally improved without massive investment, what does that do to the landscape?
It kills the entire industry. This HURTS anyone trying to fundraise, reckless billions will have a harder time finding their way into the hands of developers because many investors will not be happy with the possibility (already demonstrated at least twice) that someone could just read the outputs of your API available model and eat your lunch, and releases so that people have a less restricted, more customizable and cheaper alternative they can run on their own hardware. Expanding that view, how many services will hose this model for cheaper than Openai will host gpt?
Want proof? Openai has problems running a profit on its services yet has effectively cut prices (or otherwise given away more for less money) twice since deepseek came out. Is Openai so grateful for the free research that deepseek produced that they would rather have that than (probably) billions of dollars in lost revenue, added cost and thinner investment?
Being more speculative, the way in which models have converged on being basically interchangeable should be a red flag that the real growth is plateaued. Goods competing mostly via price is a sign that there’s uniformity in quality, that they’re mostly interchangeable.
Real model growth seems to have been discarded in favor of finding ways to make a model stare at a problem for hours at a time and come up with answers that are… Maybe human passable if the problem is easy and it’s accustomed to it? All it’ll cost is a million dollars per run. If that sounds like it’s just brute forcing the problem, it’s because it is.
Where is the real advancement? The only real advancement is inference time scaling and it doesn’t look like this last reach has gotten us close to AGI. The reasoning models are less flexible, not more, the opposite of what you would think if they were actually reasoning, best case is that the reasoning is an excuse to summon a magic token or remove a toxic token.
Am I crazy? Why would this accelerate your timeline?
Basically, the mistake in your analogy is that demand for the drug is limited and quite inelastic while the demand for AI (or basically most kinds of software) is quite elastic and potentially unlimited.
I absolutely agree with the comparison of o3 at ARC-AGI/FrontierMath to brute forcing, but with algorithmic efficiency improvements that million dollar per run is expected to gradually decrease, first becoming competitive with highly skilled human labor and then perhaps even overcompeting it. The timelines depend a lot on when (if ever) these improvements plateau. The industry doesn’t expect it to happen soon, cf. D. Amodei’s comments on their speed actually accelerating https://www.lesswrong.com/posts/BkzeJZCuCyrQrEMAi/dario-amodei-on-deepseek-and-export-controls
I feel like this comes down a lot to intuition, all I can say is gesture at the thinning distance between marginal cost and prices, wave my hand in the direction of discount rates and the valuation of Openai and ask… Are you sure?
The demand curve on this seems textbook inelastic at current margins. slashing the price of milk by 10x would have us cleaning our driveways with it, slashing the price of eggs would have us using crushed eggshells as low grade building material. A 10x decrease in the price per token of AI is barely even noticed, in fact in some markets outside of programming the consumer interest is down during that same window. This an example of a low margin good with little variation in quality descending into a price war. Maybe LLM’s have a long ways left to grow and can scale to agi (maybe, maybe not) but if we’re looking just at the market this doesn’t look like something Jevon’s paradox applies to at all, people are just saying words and if you switched out Jevon for piglet they’d make as much sense imo
The proposal just seems ridiculous to me, right? Who right now is standing on the sidelines with a killer AI app that could rip up the market if only tokens were a bit cheaper? There isn’t, the bottleneck is and always has been quality, the ability for LLM’s to be less-wrong-so-dang-always. Jevon’s paradox seems to be filling the role of a magic word in these conversations, it’s involved despite being out of place.
Sorry if this is invective at all, you’re mostly explaining a point of view so I’m not frustrated in your direction, but people are making little sense to me right now.
This is actually a good use case, which fits with what gpt does well, where very cheap tokens help!
Pending some time for people to pick at it to test it’s limits, this might be really good. My instinct is legal research, case law etc. will be the test of how good it is, if it does well this might be it’s foothold into real commercial use that actually generates profit.
My prediction is that we will be glad this exists. It will not be “phd level”, a phrase which defaces all who utter it, but it will save some people a lot of time and effort
Where I think we disagree:
This will likely not elicit a Jevon’s-paradox scenario where we will collectively spend much more money on LLM tokens despite their decreased cost, Killer app this is not.
My prediction is that low level users will use this infrequently because Google (or vanilla chatGPT) is sufficient, what they are looking for is not a report but a webpage and one likely at the top of their search already. Even if it would save them time, they will never use it so often that their first instinct would be deep research and not Google, they will not recognize where deep research would be better and won’t change their habits even if they do. On the far end, some grad students will use this to get them started but it will not do the work of actually doing the research. Besides pay walls disrupting things and limits to important physical media, there is a high likelihood that this won’t replace any of the actual research grad students (or lawyers/paralegals etc) will have to do. The number of hours they spend won’t be much effected, the range of users who will find much value will be few and they probably won’t use it every day.
I expect that, by token usage, deep research will not be a big part of what people use chatGPT for. If I’m wrong I predict it’s because law professions found a use for it.
I will see everyone in 1 year (if we’re alive) to see if this pans out!
The single factor prime causative factor driving the explosive growth in AI demand/revenue is and always has been the exponential reduction in $/flop via moore’s law, which simply is jevon’s paradox manifested. With more compute everything is increasingly easy and obvious; even idiots can create AGI with enough compute.
I think there’s some miscommunication here, on top of a fundamental disagreement on whether more compute takes us to AGI.
On miscommunication, we’re not talking about the lowering cost per flop, we’re talking about a world where openai either does or does not have a price war eating it’s margins.
On fundamental disagreement, I assume you don’t take very seriously the idea that AI labs are seeing a breakdown of scaling laws? No problem if so, reality should resolve that disagreement relatively soon!
Also, Amodei needs to cool it. There’s a reading of the things he’s been saying lately that could be taken as sane but a plausible reading that makes him look like a buffoon. Credibility is a scarce resource
It’s a bit separate topic and not what was discussed in this thread previously but I will try to answer.
I assume because Nvidia’s moat is in CUDA and chips with high RAM bandwidth optimized specifically for training while competition in inference (where the weights are static) software and hardware is already higher, and going to be even higher still by the time DeepSeek’s optimizations become a de-facto industry standard and induce some additional demand
I’m kinda opposite on the timelines thing? This is probably a timeline delayer even if I thought LLM’s scaled to AGI, which I don’t but let’s play along.
If a Pharma company could look at another company’s product and copy it and release it for free with no consequences, but the product they release itself could only be marginally improved without massive investment, what does that do to the landscape?
It kills the entire industry. This HURTS anyone trying to fundraise, reckless billions will have a harder time finding their way into the hands of developers because many investors will not be happy with the possibility (already demonstrated at least twice) that someone could just read the outputs of your API available model and eat your lunch, and releases so that people have a less restricted, more customizable and cheaper alternative they can run on their own hardware. Expanding that view, how many services will hose this model for cheaper than Openai will host gpt?
Want proof? Openai has problems running a profit on its services yet has effectively cut prices (or otherwise given away more for less money) twice since deepseek came out. Is Openai so grateful for the free research that deepseek produced that they would rather have that than (probably) billions of dollars in lost revenue, added cost and thinner investment?
Being more speculative, the way in which models have converged on being basically interchangeable should be a red flag that the real growth is plateaued. Goods competing mostly via price is a sign that there’s uniformity in quality, that they’re mostly interchangeable.
Real model growth seems to have been discarded in favor of finding ways to make a model stare at a problem for hours at a time and come up with answers that are… Maybe human passable if the problem is easy and it’s accustomed to it? All it’ll cost is a million dollars per run. If that sounds like it’s just brute forcing the problem, it’s because it is.
Where is the real advancement? The only real advancement is inference time scaling and it doesn’t look like this last reach has gotten us close to AGI. The reasoning models are less flexible, not more, the opposite of what you would think if they were actually reasoning, best case is that the reasoning is an excuse to summon a magic token or remove a toxic token.
Am I crazy? Why would this accelerate your timeline?
This seems to be the line of thinking behind the market reaction which has puzzled many people in the ML space. Everyone’s favorite response to this thesis has been to invoke the Jevons paradox https://www.lesswrong.com/posts/HBcWPz82NLfHPot2y/jevon-s-paradox-and-economic-intuitions. You can check https://www.lesswrong.com/posts/hRxGrJJq6ifL4jRGa/deepseek-panic-at-the-app-store or listen to this less technical explanation from Bloomberg:
Basically, the mistake in your analogy is that demand for the drug is limited and quite inelastic while the demand for AI (or basically most kinds of software) is quite elastic and potentially unlimited.
I absolutely agree with the comparison of o3 at ARC-AGI/FrontierMath to brute forcing, but with algorithmic efficiency improvements that million dollar per run is expected to gradually decrease, first becoming competitive with highly skilled human labor and then perhaps even overcompeting it. The timelines depend a lot on when (if ever) these improvements plateau. The industry doesn’t expect it to happen soon, cf. D. Amodei’s comments on their speed actually accelerating https://www.lesswrong.com/posts/BkzeJZCuCyrQrEMAi/dario-amodei-on-deepseek-and-export-controls
I feel like this comes down a lot to intuition, all I can say is gesture at the thinning distance between marginal cost and prices, wave my hand in the direction of discount rates and the valuation of Openai and ask… Are you sure?
The demand curve on this seems textbook inelastic at current margins. slashing the price of milk by 10x would have us cleaning our driveways with it, slashing the price of eggs would have us using crushed eggshells as low grade building material. A 10x decrease in the price per token of AI is barely even noticed, in fact in some markets outside of programming the consumer interest is down during that same window. This an example of a low margin good with little variation in quality descending into a price war. Maybe LLM’s have a long ways left to grow and can scale to agi (maybe, maybe not) but if we’re looking just at the market this doesn’t look like something Jevon’s paradox applies to at all, people are just saying words and if you switched out Jevon for piglet they’d make as much sense imo
The proposal just seems ridiculous to me, right? Who right now is standing on the sidelines with a killer AI app that could rip up the market if only tokens were a bit cheaper? There isn’t, the bottleneck is and always has been quality, the ability for LLM’s to be less-wrong-so-dang-always. Jevon’s paradox seems to be filling the role of a magic word in these conversations, it’s involved despite being out of place.
Sorry if this is invective at all, you’re mostly explaining a point of view so I’m not frustrated in your direction, but people are making little sense to me right now.
OpenAI’s Deep Research is looking like something that could be big and they were standing on the sidelines in part because the tokens weren’t cheap.
This is actually a good use case, which fits with what gpt does well, where very cheap tokens help!
Pending some time for people to pick at it to test it’s limits, this might be really good. My instinct is legal research, case law etc. will be the test of how good it is, if it does well this might be it’s foothold into real commercial use that actually generates profit.
My prediction is that we will be glad this exists. It will not be “phd level”, a phrase which defaces all who utter it, but it will save some people a lot of time and effort
Where I think we disagree: This will likely not elicit a Jevon’s-paradox scenario where we will collectively spend much more money on LLM tokens despite their decreased cost, Killer app this is not.
My prediction is that low level users will use this infrequently because Google (or vanilla chatGPT) is sufficient, what they are looking for is not a report but a webpage and one likely at the top of their search already. Even if it would save them time, they will never use it so often that their first instinct would be deep research and not Google, they will not recognize where deep research would be better and won’t change their habits even if they do. On the far end, some grad students will use this to get them started but it will not do the work of actually doing the research. Besides pay walls disrupting things and limits to important physical media, there is a high likelihood that this won’t replace any of the actual research grad students (or lawyers/paralegals etc) will have to do. The number of hours they spend won’t be much effected, the range of users who will find much value will be few and they probably won’t use it every day.
I expect that, by token usage, deep research will not be a big part of what people use chatGPT for. If I’m wrong I predict it’s because law professions found a use for it.
I will see everyone in 1 year (if we’re alive) to see if this pans out!
The single factor prime causative factor driving the explosive growth in AI demand/revenue is and always has been the exponential reduction in $/flop via moore’s law, which simply is jevon’s paradox manifested. With more compute everything is increasingly easy and obvious; even idiots can create AGI with enough compute.
I think there’s some miscommunication here, on top of a fundamental disagreement on whether more compute takes us to AGI.
On miscommunication, we’re not talking about the lowering cost per flop, we’re talking about a world where openai either does or does not have a price war eating it’s margins.
On fundamental disagreement, I assume you don’t take very seriously the idea that AI labs are seeing a breakdown of scaling laws? No problem if so, reality should resolve that disagreement relatively soon!
Also, Amodei needs to cool it. There’s a reading of the things he’s been saying lately that could be taken as sane but a plausible reading that makes him look like a buffoon. Credibility is a scarce resource
I don’t get it. Nvidia chips were still used to train deepseek. Why would nvidia take a hit?
It’s a bit separate topic and not what was discussed in this thread previously but I will try to answer.
I assume because Nvidia’s moat is in CUDA and chips with high RAM bandwidth optimized specifically for training while competition in inference (where the weights are static) software and hardware is already higher, and going to be even higher still by the time DeepSeek’s optimizations become a de-facto industry standard and induce some additional demand