Awesome! This is exactly the sort of thing I was hoping to inspire with this and this. In what follows, I’ll list a bunch of thoughts & critiques:
1. It would be great if you could pepper your story with dates, so that we can construct a timeline and judge for ourselves whether we think things are happening too quickly or not.
2. Auto-generated articles and auto-generated videos being so popular that they crowd out most human content creators… this happens at the beginning of the story? I think already this is somewhat implausible and also very interesting and deserves elaboration. Like, how are you imagining it: we take a pre-trained language model, fine-tune it on our article style, and then let it loose using RL from human feedback (clicks, ad revenue) to learn online? And it just works? I guess I don’t have any arguments yet for why that shouldn’t work, but it seems intuitively to me that this would only work once we are getting pretty close to HLAGI / APS-AI. How big are these models in your story? Presumably bigger than GPT-3, right, since even a fine-tuned GPT-3 wouldn’t be able to outperform human content creators (right?). And currently video generation tech lags behind text generation tech.
3. “Not long after, Google rocks the tech industry with a major announcement at I/O. They’ve succeeded in training a deep learning model to completely auto-generate simple SaaS software from a natural-language description. ” Is this just like Codex but better? Maybe I don’t what SaaS software is.
4. “At first, the public is astonished. But after nothing more is heard about this breakthrough for several months, most eventually dismiss it as a publicity stunt. But one year later, Google launches an improved version of the model in a new Search widget called “synthetic SaaS”.”—I didn’t successfully read between the lines here, what happened in that quiet year?
5. “The S&P 500 doubles that year, driven by explosive growth in the big-cap tech stocks. Unemployment claims reach levels not seen since the beginning of the Covid crisis.” Why is unemployment so high? So far it seems like basic programming jobs have been automated away, and lots of writing and video generation jobs. But how many jobs are those? Is it enough to increase unemployment by a few percent? I did some googling and it seems like there are between 0.5 and 1 million jobs in the USA that are like this, though I’m not at all confident. (there are 0.25M programmer jobs) More than a hundred million total employed, though. So to make unemployment go up by a couple percent a bunch of other stuff would need to be automated away besides the stuff you’ve mentioned, right?
6. “At the end of that year, the stock market once again delivers astronomical gains. Yet, curiously, the publicly disclosed performance of hedge funds — particularly of the market-neutral funds that trade most frequently — consists almost entirely of losses.” I take it this is because several tech companies are secretly using AI to trade? Is that legal? How would they be able to keep this secret?
7. You have a section on autonomous drones. Why is it relevant? Is the implication that they are going to be used by the AI to take over? The last section makes it seem like the AI would have succeeded in taking over anyway, drones or no. Ditto for the USA’s self-improving cyberwar software.
8. “Codex 4 is expected to cost nearly a billion dollars in compute alone.” This suggests that all the AIs made so far cost less than that? Which means it’s, like, not even 2025 yet according to Ajeya’s projection?
9. “After a rigorous internal debate, it’s also decided to give Codex 4 the ability to suggest changes to its own codebase during training, in an attempt to maximize performance via architectural improvements in the model.” I thought part of the story here was that more complex architectures do worse? Are you imagining that Codex 4 discovers simpler architectures? By the way, I don’t think that’s a plausible part of the story—I think even if the scaling hypothesis and bitter lesson are true, it’s still the case that more complex, fiddly architectures help. It’s just that they don’t help much compared to scaling up compute.
10. “This slows down the work to a crawl and multiplies the expense by an order of magnitude, but safety is absolutely paramount.” Why is Microsoft willing to pay these costs? They don’t seem particularly concerned about AI risk now, are you imagining this changes in the next 4 years? How does it change? Is it because people are impressed by all the AI progress and start to listen to AI safety people?
11. Also, if it’s slowing the work to a crawl and multiplying the expense, shouldn’t Microsoft/OpenAI be beaten to the punch by some other company that isn’t bothering with those precautions? Or is the “market” extremely inefficient, so to speak?
12. “Not long after this, the world ends.” Aaaaagh tell me more! What exactly went wrong? Why did the safety techniques fail? (To be clear, I totally expect that the techniques you describe would fail. But I’m interested to hear your version of the story.)
13. Who is Jessica? Is she someone important? If she’s not important, then it wouldn’t be worth a millisecond delay to increase success probability for killing her.
14. It sounds like you are imagining some sort of intelligence explosion happening in between the Codex 4 section and the Jessica section. Is this right or a misinterpretation?
Anyhow, thanks a bunch for doing this! If you have critiques of my own story I’d love to hear them.
Hey Daniel — thanks so much for taking the time to write this thoughtful feedback. I really appreciate you doing this, and very much enjoyed your “2026” post as well. I apologize for the delay and lengthy comment here, but wanted to make sure I addressed all your great points.
1. It would be great if you could pepper your story with dates, so that we can construct a timeline and judge for ourselves whether we think things are happening too quickly or not.
I’ve intentionally avoided referring to absolute dates, other than by indirect implication (e.g. “iOS 19”). In writing this, I was more interested in exploring how a plausible technical development model might interact with the cultural and economic contexts of these companies. As a result I decided to focus on a chain of events instead of a timeline.
But another reason is that I don’t feel I know enough to have a strong view on dates. I do suspect we have been in an overhang of sorts for the past year or so, and that the key constraints on broad-based development of scaled models up to this point have been institutional frictions. It takes a long time to round up the internal buy-in you need for an investment at this scale, even in an org that has a technical culture, and even if you have a committed internal champion. And that means the pace of development immediately post-GPT3 is unusually dependent on random factors like the whims of decision-makers, and therefore has been/will be especially hard to predict.
(E.g., how big will Google Pathways be, in terms of scale/compute? How much capex committed? Nobody knows yet, as far as I can tell. As a wild guess, Jeff Dean could probably get a $1B allocation for this if he wanted to. Does he want $1B? Does he want $10B? Could he get $10B if he really pushed for it? Does the exec team “get it” yet? When you’re thinking in terms of ROI for something like this, a wide range of outcomes is on the table.)
2. Auto-generated articles and auto-generated videos being so popular that they crowd out most human content creators… this happens at the beginning of the story? I think already this is somewhat implausible and also very interesting and deserves elaboration. Like, how are you imagining it: we take a pre-trained language model, fine-tune it on our article style, and then let it loose using RL from human feedback (clicks, ad revenue) to learn online? And it just works? I guess I don’t have any arguments yet for why that shouldn’t work, but it seems intuitively to me that this would only work once we are getting pretty close to HLAGI / APS-AI. How big are these models in your story? Presumably bigger than GPT-3, right, since even a fine-tuned GPT-3 wouldn’t be able to outperform human content creators (right?). And currently video generation tech lags behind text generation tech.
The beginning of the story still lies in our future, so to be clear, this isn’t a development I’d necessarily expect immediately. I am definitely imagining an LM bigger than GPT-3, but it doesn’t seem at all implausible that ByteDance would build such an LM on, say, a 24-month timeframe from today. They certainly have the capital for it, and the company has a history of favoring algorithmic recommendations and AI over user-driven virality — so particularly in Toutiao’s case, this would be a natural extension of their existing content strategy. And apart from pure scale, the major technical hurdle for auto-generated articles seems like it’s probably the size of the attention window, which people have been making notable progress on this recently.
I’d say the “it just works” characterization is not quite right — I explicitly say that this system takes some time to fine tune even after it’s first deployed in production. To elaborate a bit, I wouldn’t expect any training based on human feedback at first, but rather something more like manual screening/editing of auto-generated articles by internal content teams. That last part is not something I said explicitly in the text; maybe I should?
I think your point about video is a great critique though. It’s true that video has lagged behind text. My thinking here was that the Douyin/TikTok form factor is an especially viable setting to build early video gen models: the videos are short, and they already have a reliable reward model available in the form of the existing rec algorithm. But even though this might be the world’s best corpus to train on, I do agree with you that there is more fundamental uncertainty around video models. I’d be interested in an further thoughts you might have on this point.
One question on this part: what do you mean by “APS-AI”?
3. “Not long after, Google rocks the tech industry with a major announcement at I/O. They’ve succeeded in training a deep learning model to completely auto-generate simple SaaS software from a natural-language description. ” Is this just like Codex but better? Maybe I don’t what SaaS software is.
Yes, pretty much just Codex but better. One quick-and-dirty way to think of SaaS use cases is: “any business workflow that touches a spreadsheet”. There are many, many, many such use cases.
4. “At first, the public is astonished. But after nothing more is heard about this breakthrough for several months, most eventually dismiss it as a publicity stunt. But one year later, Google launches an improved version of the model in a new Search widget called “synthetic SaaS”.”—I didn’t successfully read between the lines here, what happened in that quiet year?
Ah this wasn’t meant to be subtle or anything, just that it takes time to go from “prototype demo” to “Google-scale production rollout”. Sorry if that wasn’t clear.
5. “The S&P 500 doubles that year, driven by explosive growth in the big-cap tech stocks. Unemployment claims reach levels not seen since the beginning of the Covid crisis.” Why is unemployment so high? So far it seems like basic programming jobs have been automated away, and lots of writing and video generation jobs. But how many jobs are those? Is it enough to increase unemployment by a few percent? I did some googling and it seems like there are between 0.5 and 1 million jobs in the USA that are like this, though I’m not at all confident. (there are 0.25M programmer jobs) More than a hundred million total employed, though. So to make unemployment go up by a couple percent a bunch of other stuff would need to be automated away besides the stuff you’ve mentioned, right?
You’re absolutely right. I was imagining some additional things happening here which I didn’t put into the story and therefore didn’t think through in enough detail. I’d expect unemployment to increase, but not necessarily to this extent or on these timescales. Will delete this sentence — thanks!
6. “At the end of that year, the stock market once again delivers astronomical gains. Yet, curiously, the publicly disclosed performance of hedge funds — particularly of the market-neutral funds that trade most frequently — consists almost entirely of losses.” I take it this is because several tech companies are secretly using AI to trade? Is that legal? How would they be able to keep this secret?
Good question. I don’t actually expect that any tech companies would do this. While it could strictly speaking be done in a legal way, I can’t imagine the returns would justify the regulatory and business-relationship risk. More to the point, big tech cos already own money machines that work, and that have even better returns on capital than market trading from an unleveraged balance sheet would.
My implication here is rather that other hedge funds enter the market and begin trading using sophisticated AIs. Hedge funds aren’t required to disclose public returns, so I’m imagining that one or more of these funds have entered the market without disclosure.
7. You have a section on autonomous drones. Why is it relevant? Is the implication that they are going to be used by the AI to take over? The last section makes it seem like the AI would have succeeded in taking over anyway, drones or no. Ditto for the USA’s self-improving cyberwar software.
Great observation. I was debating whether to cut this part, actually. I kept it because 1) it motivated the plot later, when OpenAI debates whether to build in an explicit self-improvement mechanism; and 2) it felt like I should tell some kind of story about military applications. But given how I’m actually thinking about self-improvement and the risk model (see 9 and 12, below) I think this can be cut with little loss.
8. “Codex 4 is expected to cost nearly a billion dollars in compute alone.” This suggests that all the AIs made so far cost less than that? Which means it’s, like, not even 2025 yet according to Ajeya’s projection?
Oh yeah, you’re totally right and this is a major error on my part. This should be more like $10B+. Will edit!
9. “After a rigorous internal debate, it’s also decided to give Codex 4 the ability to suggest changes to its own codebase during training, in an attempt to maximize performance via architectural improvements in the model.” I thought part of the story here was that more complex architectures do worse? Are you imagining that Codex 4 discovers simpler architectures? By the way, I don’t think that’s a plausible part of the story—I think even if the scaling hypothesis and bitter lesson are true, it’s still the case that more complex, fiddly architectures help. It’s just that they don’t help much compared to scaling up compute.
I agree that the bitter lesson is not as straightforward as “complex architectures do worse”, and I also agree with you that fiddly architectures can do better than simple ones. But I don’t really believe the kinds of fiddly architectures humans will design are likely to perform better than our simplest architectures at scale. Roughly speaking, I do not believe we are smart enough to approach this sort of work with the right assumptions to design good architectures, and under those conditions, the fewer assumptions we embed in our architectures, the better.
I do believe that the systems we build will be better at designing such architectures than we are, though. And that means there is indeed something to be gained from fiddly architectures — just not from “human-fiddly” ones. In fact, you can argue that this is what meta-learning does: a system that meta-learns is one that redesigns its own architecture, in some sense. And actually, articulating it that way suggests that this kind of self-improvement is really just the limit case of meta-learning — which in turn makes the explicit self-improvement scheme in my story redundant! So yep, I think this gets cut too. :)
10. “This slows down the work to a crawl and multiplies the expense by an order of magnitude, but safety is absolutely paramount.” Why is Microsoft willing to pay these costs? They don’t seem particularly concerned about AI risk now, are you imagining this changes in the next 4 years? How does it change? Is it because people are impressed by all the AI progress and start to listen to AI safety people?
There is no “canon” reason why they are doing this — I’m taking some liberties in this direction because I don’t expect the kinds of safety precautions they are taking to matter much. However I do expect that alignment will soon become an obvious limiting factor in getting big models to do what we want, and it doesn’t seem too unreasonable to expect this might be absorbed as a more general lesson.
11. Also, if it’s slowing the work to a crawl and multiplying the expense, shouldn’t Microsoft/OpenAI be beaten to the punch by some other company that isn’t bothering with those precautions? Or is the “market” extremely inefficient, so to speak?
The story as written is intentionally consistent with OpenAI being beaten to the punch by a less cautious company. In fact, I consider that the more plausible failure scenario (see next point) even though the text strongly implies otherwise.
Still, it’s marginally plausible that nobody was yet willing to commit funds on that scale at the time of the project — and in the world of this story, that’s indeed what happened. Relatively few organizations have the means for something like this, so that does make the market less efficient than it would be if it had more viable participants.
12. “Not long after this, the world ends.” Aaaaagh tell me more! What exactly went wrong? Why did the safety techniques fail? (To be clear, I totally expect that the techniques you describe would fail. But I’m interested to hear your version of the story.)
Yeah, I left this deliberately ambiguous. The reason is that I’m working from a risk model that I’m a bit reluctant to publicize too widely, since it feels like there is some chance that the publication itself might be slightly risky. (I have shared it privately with a couple of folks though, and would be happy to follow up with you on this by DM — please let me know if you’re interested.) As a result, while I didn’t want to write a story that was directly inconsistent with my real risk model, I did end up writing a story that strongly implies an endgame scenario which I don’t actually believe is very likely (i.e., “OpenAI carefully tries to train an aligned AI but it blows up”).
Honestly I wasn’t 100% sure how to work around this problem — hence the ambiguity and the frankly kludgy feel of the OpenAI bit at the end. But I figured the story itself was worth posting at least for its early development model (predicated on a radical version of connectionism) and economic deployment scenario (predicated on earliest rollouts in environments with fastest feedback cycles). I’d be especially interested in your thoughts on how to handle this, actually.
13. Who is Jessica? Is she someone important? If she’s not important, then it wouldn’t be worth a millisecond delay to increase success probability for killing her.
Jessica is an average person. The AI didn’t delay anything to kill her; it doesn’t care about her. Rather I’m intending to imply that whatever safety precautions were in place to keep the AI from breaking out merely had the effect of causing a very small time delay.
14. It sounds like you are imagining some sort of intelligence explosion happening in between the Codex 4 section and the Jessica section. Is this right or a misinterpretation?
Yes that is basically right.
Thanks again Daniel!
UPDATE: Made several changes to the post based on this feedback.
Thanks, this was a load of helpful clarification and justification!
APS-AI means Advanced, Planning, Strategically-Aware AI. Advanced means superhuman at some set of tasks (such as persuasion, strategy, etc.) that combines to enable the acquisition of power and resources, at least in today’s world. The term & concept is due to Joe Carlsmith (see his draft report on power-seeking AI, he blogged about it a while ago).
3. “Not long after, Google rocks the tech industry with a major announcement at I/O. They’ve succeeded in training a deep learning model to completely auto-generate simple SaaS software from a natural-language description. ” Is this just like Codex but better? Maybe I don’t what SaaS software is.
Yes, pretty much just Codex but better. One quick-and-dirty way to think of SaaS use cases is: “any business workflow that touches a spreadsheet”. There are many, many, many such use cases.
Adding to this — as I understand, Codex can only write a single function at a time. While an SaaS product would be composed of many functions (and a database schema, and an AWS / Azure / GCP cloud services configuration, and a front-end web / phone app...).
It’s like the difference between 10 lines of code and the entirety of Gmail.
I interpreted the Medallion stuff as a hint that AGI was already loose and sucking up resources (money) to buy more compute for itself. But I’m not sure that actually makes sense, now that I think about it.
See my response to point 6 of Daniel’s comment — it’s rather that I’m imagining competing hedge funds (run by humans) beginning to enter the market with this sort of technology.
Awesome! This is exactly the sort of thing I was hoping to inspire with this and this. In what follows, I’ll list a bunch of thoughts & critiques:
1. It would be great if you could pepper your story with dates, so that we can construct a timeline and judge for ourselves whether we think things are happening too quickly or not.
2. Auto-generated articles and auto-generated videos being so popular that they crowd out most human content creators… this happens at the beginning of the story? I think already this is somewhat implausible and also very interesting and deserves elaboration. Like, how are you imagining it: we take a pre-trained language model, fine-tune it on our article style, and then let it loose using RL from human feedback (clicks, ad revenue) to learn online? And it just works? I guess I don’t have any arguments yet for why that shouldn’t work, but it seems intuitively to me that this would only work once we are getting pretty close to HLAGI / APS-AI. How big are these models in your story? Presumably bigger than GPT-3, right, since even a fine-tuned GPT-3 wouldn’t be able to outperform human content creators (right?). And currently video generation tech lags behind text generation tech.
3. “Not long after, Google rocks the tech industry with a major announcement at I/O. They’ve succeeded in training a deep learning model to completely auto-generate simple SaaS software from a natural-language description. ” Is this just like Codex but better? Maybe I don’t what SaaS software is.
4. “At first, the public is astonished. But after nothing more is heard about this breakthrough for several months, most eventually dismiss it as a publicity stunt. But one year later, Google launches an improved version of the model in a new Search widget called “synthetic SaaS”.”—I didn’t successfully read between the lines here, what happened in that quiet year?
5. “The S&P 500 doubles that year, driven by explosive growth in the big-cap tech stocks. Unemployment claims reach levels not seen since the beginning of the Covid crisis.” Why is unemployment so high? So far it seems like basic programming jobs have been automated away, and lots of writing and video generation jobs. But how many jobs are those? Is it enough to increase unemployment by a few percent? I did some googling and it seems like there are between 0.5 and 1 million jobs in the USA that are like this, though I’m not at all confident. (there are 0.25M programmer jobs) More than a hundred million total employed, though. So to make unemployment go up by a couple percent a bunch of other stuff would need to be automated away besides the stuff you’ve mentioned, right?
6. “At the end of that year, the stock market once again delivers astronomical gains. Yet, curiously, the publicly disclosed performance of hedge funds — particularly of the market-neutral funds that trade most frequently — consists almost entirely of losses.” I take it this is because several tech companies are secretly using AI to trade? Is that legal? How would they be able to keep this secret?
7. You have a section on autonomous drones. Why is it relevant? Is the implication that they are going to be used by the AI to take over? The last section makes it seem like the AI would have succeeded in taking over anyway, drones or no. Ditto for the USA’s self-improving cyberwar software.
8. “Codex 4 is expected to cost nearly a billion dollars in compute alone.” This suggests that all the AIs made so far cost less than that? Which means it’s, like, not even 2025 yet according to Ajeya’s projection?
9. “After a rigorous internal debate, it’s also decided to give Codex 4 the ability to suggest changes to its own codebase during training, in an attempt to maximize performance via architectural improvements in the model.” I thought part of the story here was that more complex architectures do worse? Are you imagining that Codex 4 discovers simpler architectures? By the way, I don’t think that’s a plausible part of the story—I think even if the scaling hypothesis and bitter lesson are true, it’s still the case that more complex, fiddly architectures help. It’s just that they don’t help much compared to scaling up compute.
10. “This slows down the work to a crawl and multiplies the expense by an order of magnitude, but safety is absolutely paramount.” Why is Microsoft willing to pay these costs? They don’t seem particularly concerned about AI risk now, are you imagining this changes in the next 4 years? How does it change? Is it because people are impressed by all the AI progress and start to listen to AI safety people?
11. Also, if it’s slowing the work to a crawl and multiplying the expense, shouldn’t Microsoft/OpenAI be beaten to the punch by some other company that isn’t bothering with those precautions? Or is the “market” extremely inefficient, so to speak?
12. “Not long after this, the world ends.” Aaaaagh tell me more! What exactly went wrong? Why did the safety techniques fail? (To be clear, I totally expect that the techniques you describe would fail. But I’m interested to hear your version of the story.)
13. Who is Jessica? Is she someone important? If she’s not important, then it wouldn’t be worth a millisecond delay to increase success probability for killing her.
14. It sounds like you are imagining some sort of intelligence explosion happening in between the Codex 4 section and the Jessica section. Is this right or a misinterpretation?
Anyhow, thanks a bunch for doing this! If you have critiques of my own story I’d love to hear them.
Hey Daniel — thanks so much for taking the time to write this thoughtful feedback. I really appreciate you doing this, and very much enjoyed your “2026” post as well. I apologize for the delay and lengthy comment here, but wanted to make sure I addressed all your great points.
I’ve intentionally avoided referring to absolute dates, other than by indirect implication (e.g. “iOS 19”). In writing this, I was more interested in exploring how a plausible technical development model might interact with the cultural and economic contexts of these companies. As a result I decided to focus on a chain of events instead of a timeline.
But another reason is that I don’t feel I know enough to have a strong view on dates. I do suspect we have been in an overhang of sorts for the past year or so, and that the key constraints on broad-based development of scaled models up to this point have been institutional frictions. It takes a long time to round up the internal buy-in you need for an investment at this scale, even in an org that has a technical culture, and even if you have a committed internal champion. And that means the pace of development immediately post-GPT3 is unusually dependent on random factors like the whims of decision-makers, and therefore has been/will be especially hard to predict.
(E.g., how big will Google Pathways be, in terms of scale/compute? How much capex committed? Nobody knows yet, as far as I can tell. As a wild guess, Jeff Dean could probably get a $1B allocation for this if he wanted to. Does he want $1B? Does he want $10B? Could he get $10B if he really pushed for it? Does the exec team “get it” yet? When you’re thinking in terms of ROI for something like this, a wide range of outcomes is on the table.)
The beginning of the story still lies in our future, so to be clear, this isn’t a development I’d necessarily expect immediately. I am definitely imagining an LM bigger than GPT-3, but it doesn’t seem at all implausible that ByteDance would build such an LM on, say, a 24-month timeframe from today. They certainly have the capital for it, and the company has a history of favoring algorithmic recommendations and AI over user-driven virality — so particularly in Toutiao’s case, this would be a natural extension of their existing content strategy. And apart from pure scale, the major technical hurdle for auto-generated articles seems like it’s probably the size of the attention window, which people have been making notable progress on this recently.
I’d say the “it just works” characterization is not quite right — I explicitly say that this system takes some time to fine tune even after it’s first deployed in production. To elaborate a bit, I wouldn’t expect any training based on human feedback at first, but rather something more like manual screening/editing of auto-generated articles by internal content teams. That last part is not something I said explicitly in the text; maybe I should?
I think your point about video is a great critique though. It’s true that video has lagged behind text. My thinking here was that the Douyin/TikTok form factor is an especially viable setting to build early video gen models: the videos are short, and they already have a reliable reward model available in the form of the existing rec algorithm. But even though this might be the world’s best corpus to train on, I do agree with you that there is more fundamental uncertainty around video models. I’d be interested in an further thoughts you might have on this point.
One question on this part: what do you mean by “APS-AI”?
Yes, pretty much just Codex but better. One quick-and-dirty way to think of SaaS use cases is: “any business workflow that touches a spreadsheet”. There are many, many, many such use cases.
Ah this wasn’t meant to be subtle or anything, just that it takes time to go from “prototype demo” to “Google-scale production rollout”. Sorry if that wasn’t clear.
You’re absolutely right. I was imagining some additional things happening here which I didn’t put into the story and therefore didn’t think through in enough detail. I’d expect unemployment to increase, but not necessarily to this extent or on these timescales. Will delete this sentence — thanks!
Good question. I don’t actually expect that any tech companies would do this. While it could strictly speaking be done in a legal way, I can’t imagine the returns would justify the regulatory and business-relationship risk. More to the point, big tech cos already own money machines that work, and that have even better returns on capital than market trading from an unleveraged balance sheet would.
My implication here is rather that other hedge funds enter the market and begin trading using sophisticated AIs. Hedge funds aren’t required to disclose public returns, so I’m imagining that one or more of these funds have entered the market without disclosure.
Great observation. I was debating whether to cut this part, actually. I kept it because 1) it motivated the plot later, when OpenAI debates whether to build in an explicit self-improvement mechanism; and 2) it felt like I should tell some kind of story about military applications. But given how I’m actually thinking about self-improvement and the risk model (see 9 and 12, below) I think this can be cut with little loss.
Oh yeah, you’re totally right and this is a major error on my part. This should be more like $10B+. Will edit!
I agree that the bitter lesson is not as straightforward as “complex architectures do worse”, and I also agree with you that fiddly architectures can do better than simple ones. But I don’t really believe the kinds of fiddly architectures humans will design are likely to perform better than our simplest architectures at scale. Roughly speaking, I do not believe we are smart enough to approach this sort of work with the right assumptions to design good architectures, and under those conditions, the fewer assumptions we embed in our architectures, the better.
I do believe that the systems we build will be better at designing such architectures than we are, though. And that means there is indeed something to be gained from fiddly architectures — just not from “human-fiddly” ones. In fact, you can argue that this is what meta-learning does: a system that meta-learns is one that redesigns its own architecture, in some sense. And actually, articulating it that way suggests that this kind of self-improvement is really just the limit case of meta-learning — which in turn makes the explicit self-improvement scheme in my story redundant! So yep, I think this gets cut too. :)
There is no “canon” reason why they are doing this — I’m taking some liberties in this direction because I don’t expect the kinds of safety precautions they are taking to matter much. However I do expect that alignment will soon become an obvious limiting factor in getting big models to do what we want, and it doesn’t seem too unreasonable to expect this might be absorbed as a more general lesson.
The story as written is intentionally consistent with OpenAI being beaten to the punch by a less cautious company. In fact, I consider that the more plausible failure scenario (see next point) even though the text strongly implies otherwise.
Still, it’s marginally plausible that nobody was yet willing to commit funds on that scale at the time of the project — and in the world of this story, that’s indeed what happened. Relatively few organizations have the means for something like this, so that does make the market less efficient than it would be if it had more viable participants.
Yeah, I left this deliberately ambiguous. The reason is that I’m working from a risk model that I’m a bit reluctant to publicize too widely, since it feels like there is some chance that the publication itself might be slightly risky. (I have shared it privately with a couple of folks though, and would be happy to follow up with you on this by DM — please let me know if you’re interested.) As a result, while I didn’t want to write a story that was directly inconsistent with my real risk model, I did end up writing a story that strongly implies an endgame scenario which I don’t actually believe is very likely (i.e., “OpenAI carefully tries to train an aligned AI but it blows up”).
Honestly I wasn’t 100% sure how to work around this problem — hence the ambiguity and the frankly kludgy feel of the OpenAI bit at the end. But I figured the story itself was worth posting at least for its early development model (predicated on a radical version of connectionism) and economic deployment scenario (predicated on earliest rollouts in environments with fastest feedback cycles). I’d be especially interested in your thoughts on how to handle this, actually.
Jessica is an average person. The AI didn’t delay anything to kill her; it doesn’t care about her. Rather I’m intending to imply that whatever safety precautions were in place to keep the AI from breaking out merely had the effect of causing a very small time delay.
Yes that is basically right.
Thanks again Daniel!
UPDATE: Made several changes to the post based on this feedback.
Thanks, this was a load of helpful clarification and justification!
APS-AI means Advanced, Planning, Strategically-Aware AI. Advanced means superhuman at some set of tasks (such as persuasion, strategy, etc.) that combines to enable the acquisition of power and resources, at least in today’s world. The term & concept is due to Joe Carlsmith (see his draft report on power-seeking AI, he blogged about it a while ago).
No problem, glad it was helpful!
And thanks for the APS-AI definition, I wasn’t aware of the term.
Adding to this — as I understand, Codex can only write a single function at a time. While an SaaS product would be composed of many functions (and a database schema, and an AWS / Azure / GCP cloud services configuration, and a front-end web / phone app...).
It’s like the difference between 10 lines of code and the entirety of Gmail.
I interpreted the Medallion stuff as a hint that AGI was already loose and sucking up resources (money) to buy more compute for itself. But I’m not sure that actually makes sense, now that I think about it.
See my response to point 6 of Daniel’s comment — it’s rather that I’m imagining competing hedge funds (run by humans) beginning to enter the market with this sort of technology.