How do respiratory viruses make us sick, and what can we do to prevent or treat the diseases they cause? COVID-19 forced researchers to answer these questions while patients filled hospitals, testing vaccines and treatments as evidence about the virus accumulated. Even after vaccines and treatments become available, researchers have to assess whether they still work as viruses evolve, immune protection wanes, and resistance makes some drugs less effective.
Answering these questions draws on several fields of research. Structural biologists study viral proteins to identify targets for vaccines and drugs. Immunologists examine which immune responses provide protection and whether they hold up against new variants. Pharmacologists test antiviral activity and resistance, while clinical researchers assess whether interventions prevent infection or reduce severe disease. Together, these analyses provide the evidence for developing and evaluating vaccines and treatments, and we wanted to know whether today’s AI agents could carry them out reliably.
We built BioSecBench-RespiratoryVirus-Countermeasures to find out. The benchmark contains 64 questions spanning viral structure and function, immunity and antigenic variation, host response and disease biology, antiviral activity and resistance, and clinical and population outcomes. Each question asks an agent to analyze source materials and produce an answer checked against task-specific grading criteria. We tested 18 models across 22 combinations of models and agent software, with three attempts per question for a total of 4,224 trials. The highest observed overall pass rate was 30.2%.
At a glance
The best setup, Gemini 4 Carbon on PI, got 30.2% right. Every other setup scored below 20%.
Antiviral activity and resistance was the hardest area, with a mean pass rate of 7.0% across models.
Agents failed to act on contrary evidence and adequately verify their conclusions across all 64 questions. Each failure pattern appeared in unsuccessful attempts on every question.
The leaderboard
The highest pass rate was 30.2%. Gemini 4 Carbon on Pi topped the table, followed by Sonnet 5.5 on Pi (18.8%) and Fable 5.2 on Claude Code (17.7%). Every other setup scored below 20%. At the bottom, Nemotron 3.5 Lightning on Pi passed none of the questions, while Nemotron 3 Ultra on Pi scored 1.0%. Neither model refused any attempts.
At least one setup solved 50 of the 64 evaluations in at least one attempt. The other 14 were never solved.
For the four models we ran on two harnesses, the score differences ranged from 0.8 to 3.6 percentage points. Fable 5.2 and Opus 5.5 scored higher on Claude Code, while Sonnet 5.5 and GPT-6 Astra scored higher on Pi. Astra had the largest gap, scoring 13.5% on Pi and 9.9% on Codex.
Cost did not consistently track accuracy. GPT-6.1 Terra and Grok 4.7, both on Pi, passed the same share of attempts (11.5%), at an average cost of $0.20 and $1.60 per completed, non-refused run.
Models struggled most with antiviral activity and resistance
Antiviral activity and resistance tasks had the lowest average pass rate at 7.0%. Averaged over all 18 models, that compares with 7.2% for viral structure and function, 7.8% for immunity and antigenic variation, 9.0% for clinical and population outcomes, and 15.4% for host response and disease biology. Low scores were not solely due to refusals. Models refused none of the clinical and population outcome tasks, yet passed fewer than one in ten.
No model led every group. Gemini 4 Carbon had the highest overall pass rate, but Mistral Large 4.0 passed 22.2% of tasks built on functional assays and mutational screens, compared with Carbon’s 5.6%. Fable 5.2 led on bulk transcriptomics at 55.6%. Seven models led or tied for the lead in at least one of the 17 evidence-source, decision-type, or failure groups.
30.2%: the highest pass rate of any setup
50 of 64: evaluations that at least one setup solved
7.0%: the average pass rate on antiviral activity and resistance tasks
Agents ignored contrary evidence and failed to catch their mistakes
To see what went wrong, reviewer agents read all 3,002 completed attempts that failed, excluding refusals, and labeled the mistakes in each. We grouped questions by the mistakes found across their attempts, allowing a question to appear in more than one group.
Ignoring contrary evidence and ineffective verification each appeared in failed attempts across all 64 evaluations. Agents sometimes submitted an answer after identifying a reason to doubt it or computing a better-supported alternative. Others checked their work by repeating calculations that relied on the same faulty assumptions, or left contradictory results out of their final reports.
Average pass rates were below 11% across all seven overlapping failure groups. Gemini 4 Carbon led every group, but its scores remained below 32%. Sonnet 5.5 ranked second throughout, scoring between 16.9% and 21.6%. In every group, at least 15 of the 18 models scored below 20%, indicating that these difficulties extended across most of the models tested.
