@L Rudolf L Thank you for spending the time to write up this timeline! imho, we need more people to really think deeply about how these things could plausibly play out over the next few years or so. And, actually spending the time to share (at least their mainline expectations) as well. So yes, thank you for providing this excellent meal of “food for thought”!
That said… the statement below (imho) stands out as a potential “blocker” for your later arguments here? And, it also hits at the core of a lot of my own suspicions about the near-term future of AI and cybersecurity.
“2026… it remains true that existing code can now be much more easily attacked since all you need is an o6 or Claude subscription.”
If you suspect that any existing software will be “hackable” by something like OpenAI’s o6 or Anthropic’s Claude, doesn’t this implicitly assume a level of software maintenance that seems… almost utopian?
This would imply, that all functional software in the world is patched to remove any bugs that any “PhD+ level reasoning/coding AI agent” could find and exploit. That alone strikes me as a task of monumental proportions.
Simply finding the bugs/exploits does seem like the kind of task that millions of “PhD+-level reasoning/coding agents” might be able to accomplish, given enough compute and time. E.g. - If each bug/exploit takes one H100 GFX card 10mins to find and fix... - With 100k H100 GFX cards, they could fix 1M bugs in… 2 hours? - (1,000,000 bugs * 10 mins/bug) / 100,000 H100 GFX Cards = 100 minutes
But, we’re not just talking about patching all the world’s software—imagine the millions of repositories on GitHub alone—with updated versions that contain no bugs or exploits that a “PhD+-level reasoning/coding model” could find and exploit. It goes much further than that. Finding and fixing 1 million bugs/exploits in active repos in GitHub would only be the first step. Then, starting millions (?) of PRs in Github, to be reviewed and merged? Could this be done by humans? Not likely. Or fully automatically, by other “PhD+ reasoning/coding agent AIs”? Possible, but that seems… scary?
Any bottleneck involving human review, approval, or manual intervention would likely make this an insurmountable challenge, given the sheer scale of the problem.
And then, it requires restarting and upgrading every running version of every application that’s accessible on the internet? Think about all the legacy systems, the embedded devices, the mobile devices, the on-site Oracle servers, Microsoft SQL servers, all Banks Mainframe servers (Walmart, etc.), the vast amounts of (closed source) Pascal, Cobol and Fortran running on them. Will these Banks and Corporations allow PhD level LLMs to operate mostly autonomously to patch hundreds or thousands of newly discovered security flaws in all of their existing systems? And, if not, does that leave them open to being attacked by “PhD level reasoning/coding AI agents” from their competitors, or adversarial nations?
The speed at which these models are improving, and the potential for them to find zero-day exploits in even relatively well-maintained code, makes this a truly daunting prospect. Are we really prepared for a world where any vulnerability, no matter how obscure, is potentially discoverable and exploitable by anyone with a subscription to a powerful LLM?
What are your thoughts on the feasibility of actually achieving this level of global software hygiene, and what are the implications if humans can’t? Would all these future software updates be applied by LLMs fully autonomously?
Imho, I just don’t see how the Internet could remain functional if/when there are millions of “PhD+ Reasoning/Coding AI agents”, with access to potentially millions of software bugs and exploits, on the same internet as any of the worlds existing software stacks?
It feels like we’re on the precipice of a massive, possibly insurmountable, cybersecurity crisis, even before we get to the more existential risks of true ASI?
@L Rudolf L Thank you for spending the time to write up this timeline! imho, we need more people to really think deeply about how these things could plausibly play out over the next few years or so. And, actually spending the time to share (at least their mainline expectations) as well. So yes, thank you for providing this excellent meal of “food for thought”!
That said… the statement below (imho) stands out as a potential “blocker” for your later arguments here? And, it also hits at the core of a lot of my own suspicions about the near-term future of AI and cybersecurity.
If you suspect that any existing software will be “hackable” by something like OpenAI’s o6 or Anthropic’s Claude, doesn’t this implicitly assume a level of software maintenance that seems… almost utopian?
This would imply, that all functional software in the world is patched to remove any bugs that any “PhD+ level reasoning/coding AI agent” could find and exploit. That alone strikes me as a task of monumental proportions.
Simply finding the bugs/exploits does seem like the kind of task that millions of “PhD+-level reasoning/coding agents” might be able to accomplish, given enough compute and time. E.g.
- If each bug/exploit takes one H100 GFX card 10mins to find and fix...
- With 100k H100 GFX cards, they could fix 1M bugs in… 2 hours?
- (1,000,000 bugs * 10 mins/bug) / 100,000 H100 GFX Cards = 100 minutes
But, we’re not just talking about patching all the world’s software—imagine the millions of repositories on GitHub alone—with updated versions that contain no bugs or exploits that a “PhD+-level reasoning/coding model” could find and exploit. It goes much further than that. Finding and fixing 1 million bugs/exploits in active repos in GitHub would only be the first step. Then, starting millions (?) of PRs in Github, to be reviewed and merged? Could this be done by humans? Not likely. Or fully automatically, by other “PhD+ reasoning/coding agent AIs”? Possible, but that seems… scary?
Any bottleneck involving human review, approval, or manual intervention would likely make this an insurmountable challenge, given the sheer scale of the problem.
And then, it requires restarting and upgrading every running version of every application that’s accessible on the internet? Think about all the legacy systems, the embedded devices, the mobile devices, the on-site Oracle servers, Microsoft SQL servers, all Banks Mainframe servers (Walmart, etc.), the vast amounts of (closed source) Pascal, Cobol and Fortran running on them. Will these Banks and Corporations allow PhD level LLMs to operate mostly autonomously to patch hundreds or thousands of newly discovered security flaws in all of their existing systems? And, if not, does that leave them open to being attacked by “PhD level reasoning/coding AI agents” from their competitors, or adversarial nations?
The speed at which these models are improving, and the potential for them to find zero-day exploits in even relatively well-maintained code, makes this a truly daunting prospect. Are we really prepared for a world where any vulnerability, no matter how obscure, is potentially discoverable and exploitable by anyone with a subscription to a powerful LLM?
What are your thoughts on the feasibility of actually achieving this level of global software hygiene, and what are the implications if humans can’t? Would all these future software updates be applied by LLMs fully autonomously?
Imho, I just don’t see how the Internet could remain functional if/when there are millions of “PhD+ Reasoning/Coding AI agents”, with access to potentially millions of software bugs and exploits, on the same internet as any of the worlds existing software stacks?
It feels like we’re on the precipice of a massive, possibly insurmountable, cybersecurity crisis, even before we get to the more existential risks of true ASI?
Edit: See also: An Alternate History of the Future, 2025-2040