By 2027, evaluations are showing that frontier models—including open-source models—could meaningfully help in engineering pandemics, if bad actors so chose. There’s a messy but moderately effective effort by AI safety organisations and several agencies within governments to have some sort of misuse mitigation measures in place, in particular for API-accessible models. However, in the absence of a major incident, governments don’t care enough, and open-source models seem hard to contain. Also, some bioterrorism continues being blocked by wet lab skills and just continuing good luck regarding the absence of a motivated bioterrorist.
And also later:
“However, by 2030 there are open-source dark web models that will do whatever you want including designing candidate pandemic agents that are unnaturally lethal and virulent, and there is no quick way to pandemic-proof the world against bioterrorism.”
Thank you for thinking through possible near future history in such detail.
For these issues, I fear your timelines may be too long.
I’ve now been digging away at one particular coal face, trying to assess how helpful frontier models are for engineering pandemics, for some time now. That work will eventually see print. But given that your survey is comprehensive, it might be worth sharing some things learned about this one issue.
Right now, March 2025, frontier models are in use, multiple times daily, by many researchers in mainly biological labs, because the knowledge and guidance they provide help move the work forward. In particular, current models give outstanding, infinitely granular, inexhaustibly patient advice on how to carry out particular molecular biological and other manipulations; including all of those manipulations generally known to the LessWrong community as “wet lab skills”.
More important, because frontier models appear to be trained on the entirety of available biological literature, the models are cognizant of that literature. For molecular, cell biological, and whole-organism work in metazoans, a large subset of that relevant literature studies, or was born from study, of infectious disease and pathogens. And a much larger portion of the research literature exploits hundreds of different molecular elements and systems from microorganisms, including viruses, a multitude of different viral proteins, promoters, terminators, splice sites… For this reason, the guidance provided by the frontier models of early 2025, infinitely patient and deeply wise, is already helpful for providing many of the capabilities that would contribute to engineering effective pandemic mayhem. Moreover, because those models’ knowledge of pathogen-relevant biology is so deeply interwoven with their general biological knowledge, current and proposed technical means to generate models that cannot be used to make weapons may not be effective. Perhaps there will be no need to wait until 2030 nor need to shop for open weight models on the dark web.
For these reasons and others, the systematic evaluations carried out by the large AI labs may not completely capture present risks. And one might reasonably imagine that the better-trained and better-reasoning models of 2027 and 2030 might be yet more creative and helpful in abetting bad outcomes.
It’s an unfortunate truth that the same organisms are a) the most information-dense, b) have the most engineering literature, and c) are the most dangerous if misused (intentionally or accidentally). It’s perhaps the most direct capability-safety tradeoff. I did imagine a genomic LLM just trained on higher eukaryotes which would be safer but would stop many “typical” biotechnological benefits.
And also later:
“However, by 2030 there are open-source dark web models that will do whatever you want including designing candidate pandemic agents that are unnaturally lethal and virulent, and there is no quick way to pandemic-proof the world against bioterrorism.”
Thank you for thinking through possible near future history in such detail.
For these issues, I fear your timelines may be too long.
I’ve now been digging away at one particular coal face, trying to assess how helpful frontier models are for engineering pandemics, for some time now. That work will eventually see print. But given that your survey is comprehensive, it might be worth sharing some things learned about this one issue.
Right now, March 2025, frontier models are in use, multiple times daily, by many researchers in mainly biological labs, because the knowledge and guidance they provide help move the work forward. In particular, current models give outstanding, infinitely granular, inexhaustibly patient advice on how to carry out particular molecular biological and other manipulations; including all of those manipulations generally known to the LessWrong community as “wet lab skills”.
More important, because frontier models appear to be trained on the entirety of available biological literature, the models are cognizant of that literature. For molecular, cell biological, and whole-organism work in metazoans, a large subset of that relevant literature studies, or was born from study, of infectious disease and pathogens. And a much larger portion of the research literature exploits hundreds of different molecular elements and systems from microorganisms, including viruses, a multitude of different viral proteins, promoters, terminators, splice sites… For this reason, the guidance provided by the frontier models of early 2025, infinitely patient and deeply wise, is already helpful for providing many of the capabilities that would contribute to engineering effective pandemic mayhem. Moreover, because those models’ knowledge of pathogen-relevant biology is so deeply interwoven with their general biological knowledge, current and proposed technical means to generate models that cannot be used to make weapons may not be effective. Perhaps there will be no need to wait until 2030 nor need to shop for open weight models on the dark web.
For these reasons and others, the systematic evaluations carried out by the large AI labs may not completely capture present risks. And one might reasonably imagine that the better-trained and better-reasoning models of 2027 and 2030 might be yet more creative and helpful in abetting bad outcomes.
It’s an unfortunate truth that the same organisms are a) the most information-dense, b) have the most engineering literature, and c) are the most dangerous if misused (intentionally or accidentally). It’s perhaps the most direct capability-safety tradeoff. I did imagine a genomic LLM just trained on higher eukaryotes which would be safer but would stop many “typical” biotechnological benefits.