Preventing the next bioweapon
As artificial intelligence advances, biotech leaders team up to boost DNA synthesis screening software security
About two years ago, Dr. Jake Beal ’00, PhD ’07 was testing his company’s DNA synthesis screening software when he spotted a terrifying security vulnerability. Made and distributed by his team at RTX BBN Technologies, the software serves as the last line of defense against the online ordering of nucleic acid sequences that could produce dangerous proteins. When Beal inputted 10,000 versions of a toxin modified by artificial intelligence (AI), the mutants sailed through the screening checks “like a hot knife through butter.” Almost nothing was caught.
“That’s the moment where my heart froze up and I was like, ‘Oh my God,’” Beal recalled.
In a paper published in Science on Oct. 2, researchers, including Beal, detailed their findings on this now-patched security vulnerability. The paper was one of the first to offer “really clear evidence” that AI-assisted protein design can create proteins undetectable to current screening methods, said Jon Arizti Sanz PhD ’24, a biosecurity researcher at the Broad Institute who wasn’t involved in the study. In addition, the paper established a procedure for mitigating biosecurity risks in a field where AI is playing an increasingly large role.
Staying ahead of the ‘crashing wave’
It sounds like something out of a movie: an angry scientist or violent organization deploys a toxin so dangerous that millions die. But to Chief Scientific Officer at Microsoft Eric Horvitz MD, PhD, it’s more than just science fiction.
While preparing for a workshop at the University of Washington Institute for Protein Design’s October 2023 AI safety summit, Horvitz teamed up with Microsoft Senior Applied Scientist Dr. Bruce Wittmann and 2024 Nobel Prize-winning University of Washington Professor David Baker to generate alternative nucleic acid codes for the toxin ricin using AI. When they sent the sequences to four DNA synthesis companies, including one using RTX BBN’s software, virtually all of the potentially harmful sequences passed the screening.
At the conference, the trio met Dr. Greg McKelvey Jr. MD, then the biosecurity lead of the National Security Council. “He came to the workshop, and he held the paper in his hand, and he said, ‘You know what? You probably don't want to distribute this,’” Horvitz said.
Meanwhile at RTX BBN, Beal had identified why his software was failing: it had been trained to flag long stretches of DNA that matched toxin sequences, but AI-assisted protein design tools were changing so many codons that no long matches were present. Fortunately, the design tools did preserve the sequences corresponding to the structurally important parts of the toxin, making those regions a better target for the screening software.
While the companies fixed their code, Horvitz and Wittman developed tests to ensure that the new defenses could better identify the toxin sequences. They used a technique known in the AI safety world as “red-teaming,” pretending to be malicious actors and generating tricky sequences for each company to characterize. The goal of this process, Horvitz explained, was to develop a protocol to “stay ahead of that crashing wave of good” AI development while paying attention to the “rough edges we don’t understand very well.”
The resulting security patches successfully identified almost all of the potentially hazardous sequences. Since none of the sequences were synthesized in a lab, no one knows how many of them would actually be toxic.
Spreading the word to the right people
Once the patches were deployed, researchers had to decide how to alert experts of the potential risks of AI-assisted protein design. For weeks, Horvitz’s team led information hazard deliberations, where they met with government agencies and biosecurity experts to figure out the safest way to share their findings.
“The paper is this amazing intersection of big tech and biotech, [with] the policy world and government really coming at it from their different angles,” commented Dr. Garrett Dunlap, a leader of the Engineering Biology Research Consortium who wasn’t involved in the study.
Their first submission to Science was rejected because the study description was too vague to make the paper reproducible. To Horvitz, this wasn’t surprising. “In some ways, design replication [by bad actors] is the exact same thing we’re trying to avoid,” he said.
Pulling a trick from his days researching AI at Stanford, Horvitz endowed a nonprofit group that would establish different levels of access to the paper and evaluate how much to share on a person-by-person basis. This was the first time Science used a tiered access system, setting a precedent that could allow biosecurity researchers to share their discoveries safely.
With new power comes new responsibility
Beal acknowledges that as biotechnology develops, scientists will need to stay on top of the growing risks. Similar to how the applications of cybersecurity expanded from the finance sector and stock markets to “every significant transaction online,” Beal suggested that every movement in the biotech world would need to have some awareness of its relationship to biosecurity.
Nevertheless, Beal and Horvitz hope security fears don’t prevent scientists from using AI to improve lives. “The biggest worry I have is about people overreacting and trying to shut everything down,” Beal said.
Horvitz is similarly excited by the improvements AI could bring to medicine. “One of my dreams is that over the next couple decades, we’ll see a whole bunch of [fatal] cancers transformed into chronic diseases that are maintained with drugs that stabilize the networks,” he said.
At the same time, Horvitz emphasized the need to stay vigilant about the potential risks of AI. “If you look at all the possible uses, you can start seeing glimmers of the good and the possible downsides,” Horvitz said. “For me, I’m full of excitement about the future.”