VariantBench: An Agentic Benchmark for Genetic Variant Discovery and Interpretation
A verifiable benchmark for variant discovery, statistical genetics, and personal genomics
Analysis of data from genome sequencing and related assays is an increasingly important bottleneck in clinical and research settings. AI agents hold promise in automating and scaling such analysis.
We present VariantBench, a verifiable benchmark of 118 evaluations that tests agents on their ability to conduct variant discovery and interpretation. GPT-5.6 Sol / Codex and Claude Opus 4.8 Max / Pi led with pass rates of 42.1%, while no configuration passed more than half of all attempts.
Read the manuscript. View the results.
Benchmark Design
VariantBench contains tasks derived from independently reproduced claims in peer-reviewed studies with publicly available data. Each task provides relevant files and scientific context without prescribing a method, then scores the agent’s structured answer using deterministic graders. Evaluations were reviewed for scientific relevance, robustness across reasonable workflows, redundancy, and resistance to guessing or data contamination.
The benchmark spans variant discovery and quality control, personal and clinical genomics, and statistical and population genetics across 14 subcategories.
Main Results
We evaluated 26 model-harness configurations across 9,204 trajectories. Performance varied by analysis domain and sequencing technology, with long-read and genotype-quality-control tasks proving especially difficult.
Agents (also) struggle with tricky sequencing regions
Agents often completed standard pipeline steps but failed when results depended on case-specific inspection or scientific judgment. They struggled with repetitive regions, structural variants, and the integration of multiple forms of biological evidence, even when their trajectories showed awareness of the relevant concepts.
Simulating Real-World Neoantigen Design
With consent from Sid Sijbrandij’s care management team, we tested agents on a real-world use case in which variant discovery and interpretation in tumor sequencing data guided the design of a personalized neoantigen vaccine.
Four evaluations reproduced successive stages of the candidate-selection pipeline, where agents were required to recover genes used for neoantigen vaccine design. Earlier stages were brittle and easily mismanaged memory or spun out wall-clock limits. Across the later stages, GPT-5.6 Sol / Pi and Claude Opus 4.8 / Pi reproduced elements of the correct workflow but failed to recover several neoantigens that advanced to experimental validation and clinical use. Their trajectories applied plausible but overly restrictive filters, demonstrating how scientific and clinical judgement still plays a major role in recovering clinically important candidates.
Toward Reliable Agents for Variant Analysis
VariantBench shows that current agents can perform useful work across genetic variant analysis but struggle when success depends on data-specific scientific judgment. The distribution of results across recently released models, however, shows that models are rapidly making inroads toward being more reliable tools.
The agents of tomorrow will be critical infrastructure for variant discovery and interpretation across clinical, statistical, and population genomics. Principled measurement of current agent failure modes will be essential for getting us there.
Read the manuscript. View on benchmarks[dot]bio.