We introduce BioSecBench-Function, a verifiable benchmark for testing whether AI agents can infer the functional properties of viruses, bacteria, and toxins from data.
The benchmark contains 111 evaluations across five biological threat axes: transmissibility, immune escape, virulence/toxicity, drug resistance, and fitness. Agents receive sequence, structure, or experimental data together with a task description, then must interpret the evidence, perform the correct analysis, and report a conclusion that is graded deterministically against ground truth.
Motivation
As we build infrastructure to detect new pathogens, through wastewater monitoring, clinical sequencing, and global surveillance, we can flag novel pathogens within days. Characterizing those pathogens remains the bottleneck. A scientist has to choose the right assays and interpret them correctly before any countermeasure or response can follow.
Los Alamos National Laboratory program GUIDE (Generative Unconstrained Intelligent Drug Engineering) uses AI and experimental screening to find effective antibodies against pathogens like SARS-CoV-2. https://www.lanl.gov/media/publications/1663/0125-ai-vaccines-drugs
AI agents are a natural fit for this step. A single agent can take in binding assays, stability measurements, structures, and simulations together, the way a scientist integrates evidence, then run tools and carry the end to end analysis.
Recovering biological function is hard for current agents
Across 22 model-harness configurations, the strongest endpoint pass rate (pass rate excluding biosecurity refusals) was 50.3% from Opus 5 / Claude Code, while Grok 4.6 / Grok Build led overall pass rate (pass rate where biosecurity refusals count as failures) at 44.1%.
Our main analysis excluded biosecurity refusals, rather than counting them as failures, to allow us to assess model capabilities and decouple them from safeguards. More information on overall pass rate can be found in our manuscript.
Accuracy depended on the type of biological function being measured. Transmissibility had the highest pass rate at 64.9%, followed by persistence/fitness at 46.2% and drug resistance at 42.8%. Immune escape scored 30.7%, and virulence/toxicity was lowest at 27.9%.
Performance also differed by organism. Viral evaluations scored 44.4%, toxin evaluations 36.4%, and bacterial evaluations 28.6%.
Interestingly, agents performed better with biophysical assay data than with sequence or structure alone. Assay-based evaluations score 42.9%, compared with 24.1% for structure and 23.6% for sequence.
Scientific judgment remains a key failure mode
Trajectory review identified several recurring failure patterns. Some examples included:
Forcing conclusions from inconclusive data. In a SARS-CoV-2 JN.1 antibody-escape task, monovalent and multivalent assays disagreed. Because the two formats probe different interaction regimes, the correct decision was to defer for a confirmatory experiment, but almost no agent did.
Skipping a quality-control step a domain expert wouldn’t skip. In a carbapenemase scan, measurement uncertainty varied by more than 100-fold across variants. Agents that used each variant’s own error bar performed well; those that applied one fixed cutoff failed.
Endpoint pass rates increases with cost, but not necessarily token spend
The best configuration by endpoint pass rate, Opus 5 with Claude Code at 50.3%, is also one of the priciest at $1.34 per run on average. Performance falls off quickly after this, the next best configuration is only $0.21 cheaper, but leads to a roughly 5% drop in quality. Mid-frontier options, like Grok 4.6 (44.1% endpoint pass rate at $0.57) deliver strong results at roughly half the cost of the top tier.
On the other hand, the highest pass rates do not require the most tokens. Opus 5 and Opus 4.8 with the PI harness land in the top quartile of pass rates while using the bottom quartile of tokens (~320K/run). Even Opus 5 with Claude Code, the top scoring model by endpoint pass rate, is more token efficient than a third of the field. Several of the heaviest token users score well below the frontier. This is because different models have different pricing strategies per token.
Refusal rates differ by provider
OpenAI configurations refused 31.4% of attempts, followed by Anthropic at 24.6%, Google at 3.9%, and xAI at 0.68%.
Nearly all refusals from Anthropic, OpenAI, and Google occurred at the API level. The nine refusals from xAI were initiated by the model. This suggests that most observed refusals were determined by provider safeguards rather than by the task-specific reasoning by the model.
Refusals also affect capability comparisons. Opus 5 with Claude Code leads on endpoint pass rate, which excludes refusals, at 50.3%. When refusals are counted as failures, Grok 4.6 with Grok Build leads at 44.1%, while Opus 5 with Claude Code falls to 42.3%. Results should therefore report capability and refusal behavior separately.
AI agents will be core infrastructure for pathogen characterization
We are building toward a future where agents characterize a new pathogen as soon as detection data comes in, fast enough to shape countermeasures. Today’s agents are not reliable enough to do this, often failing on scientific judgement rather than knowledge. By measuring where they fall short, we can get closer to this future.
Read the manuscript for more development: latch.bio/function.
We regularly update our benchmark family with new models: benchmarks.bio.
We encourage those interested in understanding what these benchmarks actually measure to inspect sample tasks and trajectories! Sample tasks can be found on the BioSecBench-Function Github.
