Researching and Improving AI Agents for Biology

BioSecBench-RespiratoryVirus-Countermeasures: Benchmarking AI agents on disease biology and countermeasures for respiratory viruses

Harmon Bhasin, Evan Seeyave, Gage Moreno, Yuyi Liu, Tristan Russell, Arya Kaul, Ioanna Pagani, Lenora Kepler, Reema Singh, Marcos Parras Molto, Tyler K. Chafin, Konstantinos Konstantinidis, Karl Widney, Lucas Yago Melo Ferreira, Mikail Kelleci, Paul Oluniyi, Sara Zufan, Cheikh Cambel Dieng, Faisal Al-Zaben, Naomi Jaramillo, Zachary Mudge, Dianzhuo Wang, Kenny Workman

Preprint · BioSecBench-RespiratoryVirus-Countermeasures·64 evals · 30.2% top score·Read paper

Recurrent epidemics and pandemic threats from respiratory viruses demand a deeper understanding of disease mechanisms and rigorous ways to evaluate vaccines and therapeutics. AI agents could support the molecular, cellular, and population analyses underlying this work, but their reliability across these tasks remains uncertain. We introduce BioSecBench-RespiratoryVirus-Countermeasures, a benchmark of 64 evaluations spanning viral structure and function, immunity and antigenic variation, host response and disease biology, antiviral activity and resistance, and clinical and population outcomes. Each evaluation pairs a scientific question and source materials with task-specific grading criteria for numerical, categorical, or structured outputs. Across 22 model–harness configurations spanning 18 models, the highest observed overall pass rate was 30.2% (Gemini 4 Carbon / PI), followed by Claude Sonnet 5.5 / PI (18.8%) and Claude Fable 5.2 / Claude Code (17.7%). Seven models led or tied for the lead in at least one of the 17 displayed evidence-source, decision-type, or failure-class groups, and the highest mean pass rate among the seven failure classes was 10.6%. These results identify limitations in current agents' reliability on the evaluated tasks and provide a baseline for assessing future improvements.