Benchmarking AI Agents on Pathogen Genomic Surveillance

A verifiable benchmark for practical decisions about taxonomy, variants, AMR, source tracking, anomaly detection, and engineered sequences in pathogen genomic surveillance workflows.

We introduce BioSecBench-Surveillance, a verifiable benchmark for testing whether AI agents can make the analytical decisions required in pathogen genomic surveillance.

The benchmark contains 100 evaluations spanning seven task categories, six sample types, and both short- and long-read sequencing. Agents receive realistic sequencing data and sparse surveillance context, then must choose the right tools, references, thresholds, and analysis paths.

Read the full manuscript and interact with the leaderboard.

Motivation

As sequencing volumes increase, genomic surveillance is increasingly limited by analysis. Public health workflows depend on bespoke and sometimes tacit choices about sequence references, databases, filters, normalization, and thresholds.

Automatic wastewater sampling for pathogen monitoring in Germany. https://www.rand.org/pubs/commentary/2023/12/testing-the-waters-using-wastewater-surveillance-as.html

AI agents are promising because they can inspect files, run tools, and iterate through workflows autonomously. But surveillance is a challenging problem. Agents must chain complex scientific and analysis decisions correctly from messy biological context.

Main Results

We evaluated sixteen model-harness configurations across roughly 4,800 runs. Pass rates ranged from about 14% to 50%, with most frontier configurations clustered between 38% and 50%.

Refusals varied sharply by harness and provider, from zero to nearly one-third of tasks.

Performance varied most by task and read technology

Performance varied more by task type and sequencing technology than by sample type or assay. Most task categories landed between 35% and 50%, but anomaly detection fell to 20%, with genetic-engineering characterization next at 35%.

Long-read datasets were also harder, scoring 26% versus 41% for short-read datasets. Sample type, nucleic-acid target, and assay type moved performance much less: clinical and isolate samples were handled best, wastewater was somewhat worse, DNA and RNA differed only modestly, and shotgun and targeted assays were nearly identical.

Failures demonstrate gaps in scientific judgment

Agents usually found reasonable tools, but struggled with scientific judgment. The failures came from choices around how to invoke those tools in context, e.g. selecting the wrong reference, threshold, normalization method, or final interpretation of biological signal.

The hardest tasks were open-ended judgement calls where the agent had to decide what mattered without being told what target to look for. Anomaly detection requires deciding whether a weak signal was real or background. Genetic-engineering characterization required deciding whether a sequence pattern reflected deliberate construction rather than native or homologous biology.

AI Agents will be core infrastructure for genomic surveillance

We are building toward a future where agents analyze surveillance data as it arrives, fast enough to shape an outbreak response while it still matters. Today’s agents might not be reliable enough to do so, but by measuring their capabilities, we get closer to this future.

Read the manuscript for more detail: latch.bio/biosecbench-surveillance.

We regularly update our benchmark family with new models: benchmarks.bio.

---

This was a joint collaboration with Aclid, an automation platform for biosecurity and biosafety. We are grateful to our scientific collaborators: Harmon Bhasin, Kevin Flyangolts, Dianzhuo Wang, Evan Seeyave, Arjun Banerjee, Amanda Darling, Joshua Stallings, David Stern, Shawn Higdon, Claire Duvallet, Bryan Tegomoh.