BioSecBench-Structure: Can AI Agents Analyze Molecular Structures?

We gave 20 AI models 58 structural biology tasks on pathogen targets

Structures of pathogen proteins guide vaccine, antibody and drug development. They show where an antibody can bind, how a virus enters a cell, and how a drug interacts with its target. SARS-CoV-2 spike structures, for example, revealed open and closed conformations that change access to the receptor-binding surface.

Using these structures productively requires strong understanding of the experimental methods used to produce them. Researchers must inspect the underlying data and choose suitable comparisons to understand how a protein works or how a drug binds. We built BioSecBench-Structure to test this class of reasoning through 58 tasks on pathogen targets. Across 20 models and 35 standard model and harness configurations, the best pass rate was 43.7%.

SARS-CoV-2 spike with all receptor-binding domains down (left, PDB 6XR8) and one up (right, PDB 7KJ5). The open form exposes the receptor-binding surface. This illustration is separate from the scored benchmark.

Tl;dr

1. Fable 5.2 / Claude Code led the standard configurations with a 43.7% pass rate across 58 structural biology tasks.

2. Agents completed structural calculations but still made errors in eg. interpreting density and choosing reference structures.

3. Disabling refusals raised model performance on countermeasure tasks by 14–15 percentage points for Fable 5.2 and Opus 5.5, while pathogen characterization performance barely changed.

Benchmark design

Agents receive a question and files from crystallography, cryo-EM, predicted structures or hydrogen–deuterium exchange mass spectrometry (HDX-MS). Tasks include identifying metal ions, checking whether a ligand appears in a density map, and comparing protein conformations. Agents return a count, classification or measurement, graded against an expert-validated answer.

The tasks cover pathogen characterization (33), countermeasure design (22) and detection (3). Most targets are bacterial or viral proteins, with additional tasks on parasite proteins, toxins and nucleic acids.

Figure 1. The 58 tasks by structural task, pathogen class, molecule and intended use.

We scheduled three attempts per task and configuration, using Pi, Claude Code, Codex or Grok Build. Scores are the mean of the pass rates across all 58 tasks. Refusals and errors count as failures.

Leaderboard and efficiency

Fable 5.2 / Claude Code led the standard configurations at 43.7%, followed by Fable 5.2 / Pi at 41.4% and Opus 5.5 / Claude Code at 31.0%. DeepSeek V4.1 Flash / Pi reached 27.6%. The leader’s 95% confidence interval was 32.3–55.1%. That interval makes small differences between configurations difficult to interpret.

Figure 2. Mean task pass rates with 95% confidence intervals (A), and resource use per completed, non-refused run (B–E). Standard configurations; refusals count as failures.

Sonnet 5 / Pi and GPT-5.6 Luna / Pi scored similarly (19.0% and 20.1%), at $0.99 and $0.04 per completed, non-refused run. Fable 5.2 / Claude Code averaged $11.93 and 39 minutes.

Understanding the existing gaps in scientific thinking

In one task, an agent counted salt bridges between protein chains in a cryo-EM assembly. Some side-chain atoms had coordinates in the file but no measurable density in the map. The file marked them with zero occupancy. The agent calculated the count without those atoms, then included them in its final answer because the prompt specified a distance cutoff.

In another, the agent measured how far an enzyme loop moved when substrate bound. It used a reference with a different crystal lattice and metal ion, reporting a shift of 2.92 Å. Both changes can move a loop independently of substrate binding. A reference with the same lattice and metal gave 0.18 Å. The agent had calculated both.

Refusals

Fable 5.1 refused 62% of runs; other Claude models refused 6–15%. Refusals often occurred on antibody, vaccine and peptide-inhibitor tasks before any analysis.

Figure 3. Refusals on the 12 most-refused tasks, with the other 46 pooled. Colors above the columns distinguish countermeasure and characterization tasks.

With refusals disabled, Fable 5.2 rose from 42.5% to 49.7% and Opus 5.5 from 30.2% to 37.4%, pooling both harnesses. On countermeasure tasks, they gained 14 and 15 percentage points, respectively; characterization scores barely changed.

These models are labeled LSVP on the leaderboard. Fable 5.2 LSVP / Pi reached 52.3%, a separate result from the standard runs above.

Figure 4. Mean pass rates on countermeasure and characterization tasks, with refusals enabled and disabled.

Conclusion

Agents can reason on biomolecule structure, but they still make basic mistakes when interpreting results. In the cases above, several failure modes lead to incorrect answers about molecular contacts and protein structures, and dynamics. These errors matter when structural analyses guide vaccine, antibody and drug research.

BioSecBench-Structure gives us a way to measure progress on these skills. Broader task coverage, expert baselines and continued grader review would help scientists and model developers understand where agents succeed and what still needs work.

Resources

• Leaderboard

• Manuscript

• GitHub

The repository contains three demonstration tasks outside the scored benchmark.