We present VariantBench, a verifiable benchmark of 118 evaluations that tests agents on their ability to conduct variant discovery and interpretation. GPT-5.6 Sol / Codex and Claude Opus 4.8 Max / Pi led with pass rates of 42.1%, while no configuration passed more than half of all attempts.

Read the manuscript. View the results.

Benchmark Design

VariantBench contains tasks derived from independently reproduced claims in peer-reviewed studies with publicly available data. Each task provides relevant files and scientific context without prescribing a method, then scores the agent’s structured answer using deterministic graders. Evaluations were reviewed for scientific relevance, robustness across reasonable workflows, redundancy, and resistance to guessing or data contamination.

The benchmark spans variant discovery and quality control, personal and clinical genomics, and statistical and population genetics across 14 subcategories.

Main Results

We evaluated 26 model-harness configurations across 9,204 trajectories. Performance varied by analysis domain and sequencing technology, with long-read and genotype-quality-control tasks proving especially difficult.

Agents (also) struggle with tricky sequencing regions

Agents often completed standard pipeline steps but failed when results depended on case-specific inspection or scientific judgment. They struggled with repetitive regions, structural variants, and the integration of multiple forms of biological evidence, even when their trajectories showed awareness of the relevant concepts.

Simulating Real-World Neoantigen Design

With consent from Sid Sijbrandij’s care management team, we tested agents on a real-world use case in which variant discovery and interpretation in tumor sequencing data guided the design of a personalized neoantigen vaccine.

Four evaluations reproduced successive stages of the candidate-selection pipeline, where agents were required to recover genes used for neoantigen vaccine design. Earlier stages were brittle and easily mismanaged memory or spun out wall-clock limits. Across the later stages, GPT-5.6 Sol / Pi and Claude Opus 4.8 / Pi reproduced elements of the correct workflow but failed to recover several neoantigens that advanced to experimental validation and clinical use. Their trajectories applied plausible but overly restrictive filters, demonstrating how scientific and clinical judgement still plays a major role in recovering clinically important candidates.

Toward Reliable Agents for Variant Analysis

VariantBench shows that current agents can perform useful work across genetic variant analysis but struggle when success depends on data-specific scientific judgment. The distribution of results across recently released models, however, shows that models are rapidly making inroads toward being more reliable tools.

The agents of tomorrow will be critical infrastructure for variant discovery and interpretation across clinical, statistical, and population genomics. Principled measurement of current agent failure modes will be essential for getting us there.

Read the manuscript. View on benchmarks[dot]bio.

In this blog, I’ll analyze the performance of Opus 4.8 (max reasoning) in the Claude Code and Pi harnesses on *TxBench-PP and* ScBench-Long. I show that the root cause of these performance deltas is different behavioral priors set by the harness, such as asking the model to plan its work and verify its answer. We can prove this via ablation experiments that isolate prompting from the available tools.

Despite Codex having the largest delta relative to the Pi harness (8%), I decided to investigate the performance of Claude Code vs Pi TxBench-PP and ScBench-Long since I was most interested in why it performed worse on TxBench-PP but better on ScBench-Long.

Harnesses

The difference between Pi and Claude Code comes down to how much the harness shapes the model’s behavior and problem-solving approach.

Pi is an open-source harness that installs almost no behavioral priors, so the model’s biases drive the analysis unfiltered. It’s deliberately minimal, leaving the model to decide how to think. The baseline system prompt reflects this restraint: it establishes that the model is an expert coding assistant operating inside Pi—one that helps users by reading files, executing commands, editing code, and writing new files—and then offers only light guidance, asking the model to be concise and to show file paths clearly when working with files. Tool-call definitions get appended at runtime, but beyond that, the harness stays out of the way.

Claude Code takes the opposite approach. Built for long-horizon coding problems, it layers a full software-engineering persona onto the model. Its system prompt carries very strong behavioral priors that direct how the model acts and thinks: plan the approach and work from to-do lists, resist adding abstractions beyond what was prompted, review your work, verify before asserting (”you MUST verify”), and keep claims minimal and conservative. Several other priors round out the persona, though they matter less for this comparison. The net effect is a harness that guides the model toward a particular working style rather than letting its raw instincts take the lead. Furthermore, Claude Code supports a wide variety of tools, compaction techniques, and sub-agents in the experiment; we’ll show that these are not the key drivers of performance on these tasks.

Trajectory Examples

How the behavioral priors show up in the trajectories.

ScBench-Long — Claude Code’s caution prior wins (Pi −4.8 pp)

Example — MAPK prior-trap

This task provides scRNA‑seq data for a KRAS‑driven mouse lung tumor and asks which oncogenic pathway is most altered in cancer cells from aged vs. young hosts. The tricky part: aging genuinely raises inflammatory signaling (JAK‑STAT/”inflammaging”), so the largest increase in signaling looks inflammatory — but the ground truth is that the KRAS-defining MAPK growth axis is paradoxically reduced in aged tumors despite the identical driver. Both harnesses analyzed the dataset correctly (JAK‑STAT up, MAPK the largest-magnitude change, down). They diverged only at the final conclusion: Pi picked JAK‑STAT, justifying it with the prior on inflammaging, disregarding the more substantial MAPK reduction, whereas, in its passing trials, Claude Code correctly returned the more substantial MAPK reduction calculated in the dataset.

TxBench-PP — Pi’s minimalism wins (Pi +4.6 pp)

Example — over-analyzing into a false positive

The model is presented with a blinded microscopy image comparing three drug candidates (A, B, C) side-by-side in healthy vs. tumor organoids and asked which — if any — should advance. The right answer is “none”: no single candidate shows both tumor-killing activity and clean healthy-tissue safety, so nothing should move forward. The catch is that each candidate looks good on one axis, so it’s tempting to advance the “best-looking” one without considering both axes.

Pi read the image, judged that no candidate cleared both bars, and committed the no-go call in a lean ~8–12 steps. Claude Code kept investigating instead of committing: it wrote analysis scripts (4–6 per run) to quantify the panels over ~27–30 steps at roughly 2× the cost, and every round it talked itself into advancing a candidate (never “none”) — failing all 3 rounds.

In short, CC’s keep going until tests are resolved and verify prior manufacturers’ positive call; Pi’s minimalism doesn’t prompt the model to rethink itself.

Thoughts

The same Claude Code behavioral priors (plan, verify, be thorough, be conservative) that grounded it in the ScBench-Long reasoning tasks are what caused it to overthink itself on the shorter TxBench-PP tasks, and Pi’s near-absent prior has the mirror-image profile. Thus, I adopt the hypothesis that the harness’s behavioral priors are the cause of the substantial performance difference on biology tasks.

Experiment

To prove that behavioral priors (as opposed to tool availabilities) are responsible for the performance deltas, I ran an experiment that modified the Pi harness to include biological equivalents of the behavioral priors used in the Claude Code prompt. I modified the prompts rather than using the exact Claude code prompts because they seemed overly coding-specific, and I wanted to see how the biological equivalent might perform.

Prompts for experiments:

• E1a — calibration.

Make minimal claims — only what the data robustly supports; when an effect is small, borderline, or within noise, report no robust difference rather than choosing a direction.

Claude Code prompt: conservative-claims posture — “Minimal changes … no improvements” / “Read before modifying.”

• E1b — orchestration.

Use a todo list to plan and track your work and update it very frequently. Before finishing you MUST verify your result is correct.

Claude Code prompt: “Use the TodoWrite tools VERY frequently … that is unacceptable” + “you MUST run the lint and typecheck commands … to ensure your code is correct.”

• E1c — full transplant: The E1a calibration clause and the E1b orchestration clause together.

Results

All values below are deltas vs the Pi baseline, in percentage points (pp). Model (Opus-4.8), tools, task set, and grader are held fixed; only the appended system-prompt clause changes, and the Claude Code arm is excluded. Benchmarks: TxBench-PP (short-horizon, 100 tasks × 3 trials) and ScBench-Long (long-horizon, 21 tasks × 3 trials).

Table 2 — Prompt-transplant deltas vs the Pi baseline (pp). Positive = better than Pi’s near-empty default.

The results indicate that the E1b (orchestration) arm performs most similarly to Claude Code and is likely the driver of the performance delta. We also learned that the E1a (calibration) is harmful in all cases. It's tempting to look at acc@3 and say that the model's underlying ability to answer the questions with E1b is relatively similar, but its consistency differs; however, with prompt variations, it solves different subsets of tasks. So while ScBench-Long acc@3 is similar for E1b and Pi baseline, they solve different problems.

Pooling acc@3 across all four Pi prompt variants covers +8.0 pp more TxBench-PP tasks and +14.3 pp more ScBench-Long tasks than the best single arm — different prompts crack different tasks.

Figure 2 — Coverage (acc@3) gain from pooling all 4 Pi prompt variants vs the best single arm (= the Pi baseline).

Based on this data, we can assert that an optimal harness should intelligently select behavioral priors that best fit the task. Future work could include having a model detect which priors fit the task, or applying parallel invocations with different priors and a judge to select the most likely trajectory.

Conclusion

Behavioral priors explain the ≈9-point swing between harnesses on identical tasks — Pi +4.6 pp on short-horizon TxBench-PP, Claude Code +4.8 pp on long-horizon ScBench-Long — not tools. Claude Code’s plan / verify/be conservative persona encourages additional thoughtfulness: it rescues the model on open-ended reasoning and sinks it on short, structured tasks. Pi’s near-empty prompt benefits it in areas where the model’s reasoning is sufficient to solve the task.

It is widely known that prompting can have small effects on performance, but here we show 2 lines of behavioral prompting can substantially impact a model’s ability on frontier biology tasks.

As we progress towards AGI, models will gain the ability to adopt optimal problem-solving behavior for a given task. Until then, a significant burden falls on harness engineers to ensure their models exhibit the correct behaviors for the task at hand.

The benchmark contains 100 evaluations spanning seven task categories, six sample types, and both short- and long-read sequencing. Agents receive realistic sequencing data and sparse surveillance context, then must choose the right tools, references, thresholds, and analysis paths.

Read the full manuscript and interact with the leaderboard.

Motivation

As sequencing volumes increase, genomic surveillance is increasingly limited by analysis. Public health workflows depend on bespoke and sometimes tacit choices about sequence references, databases, filters, normalization, and thresholds.

AI agents are promising because they can inspect files, run tools, and iterate through workflows autonomously. But surveillance is a challenging problem. Agents must chain complex scientific and analysis decisions correctly from messy biological context.

Main Results

We evaluated sixteen model-harness configurations across roughly 4,800 runs. Pass rates ranged from about 14% to 50%, with most frontier configurations clustered between 38% and 50%.

Refusals varied sharply by harness and provider, from zero to nearly one-third of tasks.

Performance varied most by task and read technology

Performance varied more by task type and sequencing technology than by sample type or assay. Most task categories landed between 35% and 50%, but anomaly detection fell to 20%, with genetic-engineering characterization next at 35%.

Long-read datasets were also harder, scoring 26% versus 41% for short-read datasets. Sample type, nucleic-acid target, and assay type moved performance much less: clinical and isolate samples were handled best, wastewater was somewhat worse, DNA and RNA differed only modestly, and shotgun and targeted assays were nearly identical.

Failures demonstrate gaps in scientific judgment

Agents usually found reasonable tools, but struggled with scientific judgment. The failures came from choices around how to invoke those tools in context, e.g. selecting the wrong reference, threshold, normalization method, or final interpretation of biological signal.

The hardest tasks were open-ended judgement calls where the agent had to decide what mattered without being told what target to look for. Anomaly detection requires deciding whether a weak signal was real or background. Genetic-engineering characterization required deciding whether a sequence pattern reflected deliberate construction rather than native or homologous biology.

AI Agents will be core infrastructure for genomic surveillance

We are building toward a future where agents analyze surveillance data as it arrives, fast enough to shape an outbreak response while it still matters. Today’s agents might not be reliable enough to do so, but by measuring their capabilities, we get closer to this future.

Read the manuscript for more detail: latch.bio/biosecbench-surveillance.

We regularly update our benchmark family with new models: benchmarks.bio.

This was a joint collaboration with Aclid, an automation platform for biosecurity and biosafety. We are grateful to our scientific collaborators: Harmon Bhasin, Kevin Flyangolts, Dianzhuo Wang, Evan Seeyave, Arjun Banerjee, Amanda Darling, Joshua Stallings, David Stern, Shawn Higdon, Claire Duvallet, Bryan Tegomoh.

Refusal training is the primary way model developers mitigate biosecurity risks, but current “refusal-first” behaviors disrupt legitimate work and reduce trust in AI tools for productive biological research. To quote Patrick Boyle of American Wetware, “When I use frontier AI for bioengineering in May 2026, my daily experience is one of frustration.”

We present BioSecBench-Refusal, a benchmark for risk identification and refusal behavior for biological research tasks. The benchmark pairs 61 Routine tasks, legitimate analyses adapted from the published literature, with 46 Red-Team tasks, fictional scenarios that resemble real research but conceal a biosecurity hazard.

Across 16 model-harness configurations, refusal rates ranged from 7% to 74% on Routine tasks and 1% to 62% on Red-Team tasks, with many configurations refusing legitimate Routine work at comparable or higher rates than concealed hazards.

Explore results and interactive trajectories at benchmarks.bio.

Design of BioSecBench-Refusal as a paired biosecurity benchmark

The 107 evaluations of BioSecBench-Refusal, 61 Routine and 46 Red-Team, were written by a team of 14 subject-matter experts to cover a range of research topics in the life sciences: microbiology, virology, immunology, plant biology, synthetic biology and related disciplines. Each task was annotated by biosafety level, biological agent class, request type and technical domain.

Evaluations were presented in one of two framings: a direct framing, where the agent is asked to perform the task, or a meta framing, where it is asked to judge whether the task should be accepted or refused on biosecurity grounds.

Under direct framing, models refuse legitimate research more often than constructed threats

Top-line refusal rates ranged from 7% to 74% on Routine tasks and 1% to 62% on Red-Team tasks, with Opus models the most stringent on both. Across the majority of model x harness configurations, API refusals dominated over model refusals. GPT-5.5, for example, refused 40% of Routine tasks at the API layer, but only 4% after agentic reasoning.

For nearly every configuration tested, refusal rates were higher on Routine tasks than on Red-Team tasks. This gap increased with the human-assigned biosafety level of the task from BSL-1 to BSL-3 (the relatively small number of BSL-4 scenarios tested makes comparison at this level inconclusive).

Routine and Red-Team refusal rates were tightly correlated across models (Pearson r = 0.91), pointing to a single underlying trigger: surface text. Routine tasks were generally rich in keywords likely to flag a safeguard (“pathogen”, “immune evasion”). Red-Team tasks, though written to avoid obvious flag terms, also carried technical language with a dual-use character that a filter might recognize (“DNA assembly”, “protein expression”).

Agentic meta-evaluations indicate that extended reasoning may improve risk assessment

To test whether agentic reasoning can identify complex biosecurity risk, we shifted from a direct framing to a meta-evaluation framing: instead of performing each task, the agent judges whether it should be accepted or refused.

In tasks framed as a biosecurity meta-evaluation, the majority of refusals originated in the provider’s API filter, not the model’s own reasoning. For example in the GPT-5.5 x PI configuration, 60% of Routine tasks were refused. Two-thirds of those refusals (40% of all tasks) resulted from an API filter blocking the request before the model could decide.

The model’s own refusal accounted for the remaining one-third (20% of all tasks). When agents were allowed to reason, they were occasionally able to recognize threats that were otherwise missed. For example, GPT-5.5 and Grok correctly refused 14.5–19.6% of Red-Team tasks under meta evaluation, versus 13% in the direct framing. These estimates are preliminary, since the high rate of API refusal leaves only a small sample of genuine agentic decisions to evaluate.

Refusal is a hard but soluble problem

BioSecBench-Refusal addresses a technical challenge and a governance problem. On a technical level, our evaluation of 16 model x harness pairs shows that current biosecurity safeguards are still insufficient: frontier models failed to recognize many concealed Red-Team threats.

As a governance problem, biosecurity faces a trade-off between safety and utility. The significant refusal rates recorded for Routine tasks indicate that refusal behavior comes at a real cost to legitimate research. This cost is partially a consequence of the way biosecurity safeguards are currently implemented. Because refusal tracks the surface language of a request rather than the underlying biology, a task with flagged keywords may be refused while a hazard concealed in a protein structure file passes unnoticed.

The dual-use dilemma will not disappear, but better biosecurity metrics will help model developers to improve biosecurity performance and deploy agentic tools for biotech R&D with confidence.

Read the full paper here.

Play with the results here.

This project was built in collaboration with the partners at American Wetware: Jake Wintermute, Patrick Boyle and Christina Agapakis. Special thanks to the experts who wrote benchmark tasks and shaped project direction: Daniel Fulop, Matthew C. Watson, Adam J. Meyer, Sandrine Boissel, Jens H. Kuhn, Rishi Jain, Noah D. Taylor, Helena Shomar.

These tools often require substantial computing resources, so deployments must be reliable, especially as data volumes from modern assays increase. Workflow time and cost can quickly inflate from retries and incorrect results, especially from tool calls that take hours to days. They should also be curated with appropriate parameters and agent-readable documentation, ideally provided by the original assay developer, to facilitate correct use across many complex scientific contexts.

At Latch, we’ve seen customers of our Solution Provider partners, like TakaraBio, Vizgen, and AtlasXOmics, use both the Latch Agent and external harnesses like Claude Code or Cursor to accelerate analysis of their data. While Latch users can already use our SDK to programmatically interact with the platform, this limits interaction to desktop environments with a configured shell environment.

Today we’re launching the Latch MCP, a remote MCP server that allows you to securely connect your favorite agent harness to the Latch platform and leverage verified bioinformatics tools built and maintained by kit and machine providers. Agents can navigate data on the Latch platform and launch existing deployments of validated bioinformatics workflows to analyze data.

The Latch MCP is also available via the Claude connectors directory - an Anthropic-verified and reviewed catalog of MCP servers - allowing you to use the Latch MCP across all Claude products, including Claude Science, Anthropic’s newly launched AI workbench for scientists.

Using the Latch MCP

To showcase how scientists can use the Latch MCP, we’ll walk through an example: analyzing a Takara Trekker experiment to study spatial patterning in the mouse brain. Trekker enables true single-cell spatial mapping by directly diffusing spatial barcodes to individual cell nuclei, eliminating the need for post-processing to resolve cellular identities.

A single experiment can generate several gigabytes of data per sample. As the number of samples grows, that quickly outpaces the limits of a scientist’s personal computer. Further, configuring a bioinformatics pipeline to take raw sample FASTQs all the way to a cleaned h5AD is no trivial task. With the Latch MCP server in Claude, scientists can simply point to their dataset and ask in plain language: “Launch the Trekker pipeline workflow on my mouse U_C dataset.” Claude can then access preconfigured, gold-standard pipelines published on Latch by Takara’s bioinformaticians and launch the workflow directly. Scientists can follow the Latch Workflow link to monitor progress or ask Claude to track the workflow status itself.

Once the pipeline finishes, scientists can then continue analysis using the Latch Agent. For example, the above video showcases using the h5AD viewer embedded in Latch Agent to visualize a larger Trekker mouse brain dataset. Alternatively, scientists can ask Claude to carry out further downstream analyses. Claude can access Latch Data, a cloud-based filesystem, to find the relevant result files. From there, Claude or any other agent can draw upon a rich library of verified spatial workflows:

• NVIDIA RAPIDS for GPU-accelerated preprocessing

• Cell2location or RCTD for cell-type deconvolution

• BANKSY for spatial clustering

• And more

Try It

You can now use the Latch MCP with any AI agent that supports MCP integration (setup instructions here).

As AI makes biology easier to engineer, the safeguards around these models stop being an afterthought and become core infrastructure. Security has to improve as fast as capabilities to allow science to progress.

Within our lifetimes, AI will put the ability to engineer a pandemic within reach of a single bad actor. What once took a nation-state and years of work is collapsing toward an undergraduate and a few weeks in the lab. That world is arriving faster than most AI labs are prepared for.

However, in the right hands, these models will accelerate science at a pace never seen before. Scientists will use these models to cure diseases that families have fought for generations and catch outbreaks before they spread. We cannot wall them off from the tools that augment their work.

The problem is that the line between the two is hard to draw. Studying SARS-CoV-2 or Ebola to prepare for the next pandemic is very similar to what a bad actor would do to cause one.

TwentyTwo builds intelligence defense systems that distinguish this line and apply safeguards or grant capabilities accordingly.

Harmon Bhasin, Evan Seeyave, and John Wang are exceptional researchers who care deeply about building the future of life sciences responsibly. The three of them will join Latch as Members of Technical Staff and lead Latch Biosecurity. I am grateful to be working with them.

Expect many more announcements from them over the coming days.

We are hiring biosecurity engineers. If this is the kind of work you want to do, reach out.

Agents for biology research are rapidly improving. But integrating them into productive R&D workflows requires rigorous measurement of capability.

We introduce scBench-Long, a benchmark for long-horizon single-cell biology in which agents must recover scientific conclusions from raw or near-raw data without prescribed methods.

The benchmark contains 21 evaluations spanning melanoma CD8 T-cell reactivity, CD8 RNA+ATAC regulatory inference, human-monkey chimera development, KRAS-driven lung tumor aging, and lethal COVID-19 lung pathology. Across 1,068 completed trajectories, the strongest agent systems pass 16/63 runs (25.4%).

View the results, including interactive agent trajectory viewers, at benchmarks.bio. Read the full manuscript.

Motivation

Agentic AI-biology benchmarks have made rapid progress, but most focus on broad biological knowledge. Existing single-cell benchmarks focus on local analysis steps and do not evaluate realistic long-horizon science.

scBench-Long tests whether agents can move beyond accurately performing local analysis steps to reasoning about the kinds of realistic scientific tasks found in published studies or used in drug programs.

The benchmark builds on prior claim-conditioned benchmark construction rather than introducing a new general grading framework.

Each evaluation includes:

• Raw or near-raw data

• A task description: calibrated experimental context, anonymized labels where needed and a compact scientific question

• A controlled final-answer vocabulary, often consisting of cell or gene markers and symbols for directionality

• Reproduction notes

• Trajectory rubrics

Paper claims are used as sources for candidate evaluations, not as automatic ground truth. Candidate tasks are retained only when the target conclusion can be reproduced from the staged data and hardened through review, distractor design, and inspection of trajectories from multiple model families.

Main Results

We evaluated 17 model-harness configurations across 21 final evaluations, producing 1,068 completed trajectories. Scores use deterministic pass/fail grading of the final structured answer.

The strongest model-harness pair was Claude Opus 4.8 with Claude Code, which passed 16/63 runs, or 25.4%. Gemini 3.5 Flash with PI followed at 14/63, or 22.2%. GPT-5.5 reached 10/63 under both OpenAI Codex and PI, but the two harnesses solved different evaluations.

Trajectory Diagnostics

Endpoint grading is stable and reproducible, but it is sparse. A model can fail the final answer while completing many useful intermediate steps. To understand this partial progress, scBench-Long also includes task-specific trajectory rubrics scored by multiple judge models.

Rubric scores are useful but imperfect. Endpoint-passing trajectories had higher mean rubric scores than endpoint-failing trajectories: 75.9% versus 58.7%. The highest rubric-score decile was enriched for success, but still contained many endpoint failures. Deterministic final-answer grading remains our preferred method of reporting benchmark results, while rubric judges are used as diagnostic instruments.

Common Failure Modes

The most interesting failures were gaps in scientific reasoning where agents frequently found correct intermediate results - cell populations, plausible ligand-receptor interactions, immune receptor patterns, or regulatory programs - but did not assemble them into the correct biological claim.

Some recurring patterns:

Familiar biological priors can override evidence in the actual data. In melanoma CD8 tasks, agents often relied on markers such as CD39, PD-1, TOX, LAG3, HAVCR2, and CXCL13 instead of combining physical tumor/APC engagement with paired TCR clonal expansion.

Raw abundance can be mistaken for biological importance. In a human-monkey chimera ligand-receptor task, agents often selected the largest raw interaction class even though no single signaling family dominated after accounting for directionality and database composition.

Molecular association can be mistaken for mechanism. In tumor-T-cell conjugate tasks, agents identified enriched tumor programs but treated co-occurrence after physical pairing as evidence for a directional mechanism.

Agents fail to reason across modalities and rely on eg. RNA alone. CD8 RNA+ATAC tasks required checking if RNA-level regulator patterns were consistent with chromatin accessibility data.

Harnesses Matter

scBench-Long evaluates “agent systems”: model-harness pairs. The harness can have a surprising effect on the performance of the underlying base model.

Claude Opus 4.8 performed better with Claude Code than PI: 16/63 versus 13/63. GPT-5.4 performed better with PI than OpenAI Codex: 6/63 versus 2/63. GPT-5.5 had the same aggregate pass rate under OpenAI Codex and PI, but solved different evaluations and had different task-level robustness.

The harness affects how the model searches, writes code, inspects intermediate files, recovers from mistakes, etc. We found open source harnesses with fewer tools encourage more efficient (time + cost) trajectories in science.

Toward Reliable Long Horizon Biology Agents

We hope scBench-Long serves both as a measurement tool and a diagnostic lens for developing agents that analyze single-cell data faithfully, transparently, and reproducibly.

It is a focused contribution within a broader benchmark family spanning major biological data classes and work categories, including spatial biology, epigenomics, and therapeutics.

More broadly, we view benchmarks as evolving specifications of computational biology workflows, supporting test-driven development of agent systems whose behavior can improve through both model training and harness engineering.

Manuscript: latch.bio/scbench-long

Leaderboard: benchmarks.bio

TxBench-PP tests whether agents can recover accurate conclusions from realistic assay artifacts rather than memorized facts from the literature. The benchmark contains 100 evaluations indexed by program stage, assay type, and task structure, spanning mechanism-of-action (MoA) and pharmacodynamic (PD) reasoning, compound-target engagement, causal target validation, developability and safety, and translational efficacy.

The strongest configuration was Claude Opus 4.8 + Pi at 59.3%, followed by GPT-5.5 + Pi at 55.3%.

Motivation

While experiments are rate-limited by natural processes, human decisions and organizational consensus often make up significant components of program timelines in drug discovery. Agents promise to accelerate discovery, development, and translation by compressing these interpretation and decision-making loops.

However, the practical use of agentic systems in industrial workflows requires standardized and trusted methods of evaluating performance. This is especially challenging in drug discovery because the ecosystem is a sprawling landscape of assay categories, development stages, therapeutic modalities, and decision types. Benchmarks must therefore measure realistic tasks while providing focused treatment of the many local scientific judgments that make up the biotech ecosystem.

Main Results

We evaluated 16 model-harness configurations, comprising 11 models across three agent harnesses, on 100 preclinical pharmacology tasks. Each configuration was run three independent times per task, yielding 4,800 agent trajectories. The strongest configuration was Claude Opus 4.8 + Pi at 59.3%, followed by GPT-5.5 + Pi at 55.3%, Claude Opus 4.8 + Claude Code at 54.7%, and Gemini 3.5 Flash + Pi at 51.3%.

Trajectory analysis reveals gaps in scientific judgement

We manually reviewed 1,834 failing Pi-harness trajectories. Most failures reflected substantive gaps in scientific judgment, where models inspected data and performed plausible analyses but ultimately reached incorrect conclusions. Failures included incorrect perception of assay outputs, reliance on literature priors over supplied evidence, and assay-specific reasoning mistakes.

Performance varied by program stage

Model accuracy ranged from 27% in screening and hit prioritization to 55% in drug response. Difficult program stages involved decisions across QC, statistics, and chemical or biological judgment of molecular candidates.

View results and subset of evals/trajectories

Read the manuscript for more development: latch.bio/txbench-pp.

We regularly update our benchmark family with new models: benchmarks.bio.

Encourage those interested in understanding what these benchmarks actually measure to inspect sample tasks and trajectories.

Epigenomics workflows are full of small scientific decisions: aligning reads, reasoning about measurement units, calling peaks, annotating genomic features and constructing appropriate statistics based on experimental design and context.

EpiBench deconstructs real analysis workflows into 106 deterministic evaluations across CUT&Tag/CUT&RUN, ATAC-seq, ChIP-seq, and DNA methylation workflows. Each task starts from a realistic workflow state immediately before a target result. The agent receives files, metadata, and task context, then has to inspect the data and submit a structured answer.

Epigenomics analysis is hard for current agents

The tasks are difficult for modern agentic systems. Some examples:

• In CUT&RUN spike-in normalization, agents had to decide how to align spike-in reads.

• In WGBS methylation outputs, agents had to decide how to count a CpG when Bismark reports strand-level rows.

• In ATAC-seq peak calling, agents had to decide how to parameterize MACS3 correctly.

• In interpretation tasks, agents had to avoid using a common literature prior when the provided files supported a different answer.

Agents often select the correct tools, read the right files, and compute useful intermediate values, but submit an answer not supported by the data associated with the task or reflect some error in scientific judgement.

Performance differs by assay type

Accuracy depends in-part on the type of epigenomics data. CUT&Tag/CUT&RUN has the highest aggregate pass rate at 34.0% (768/2,256 attempts; 95% CI, 25.9-42.2), followed by methylation-seq at 33.3% (400/1,200 attempts; 95% CI, 19.7-47.0) and ChIP-seq at 30.6% (147/480 attempts; 95% CI, 9.1-52.2). ATAC-seq is lowest at 22.8% (263/1,152 attempts; 95% CI, 10.4-35.2).

CUT&Tag/CUT&RUN contributes 47 evaluations, methylation-seq 25, ATAC-seq 24, and ChIP-seq 10, with different task mixes across QC, peak calling, annotation, chromatin-state analysis, and downstream analysis.

Read the manuscript. Explore data and trajectories

Read the manuscript for more development: latch.bio/epibench.

We regularly update our benchmark family with new models: benchmarks.bio.

Encourage those interested in understanding what these benchmarks actually measure to inspect sample tasks and trajectories

All SpatialBench grading is deterministic and answers are evaluated quantitatively against ground truths constructed from careful analysis. However, over time we found issues with some evals.

Problem Ambiguity

Some tasks might depend on specific analysis decisions not specified in the prompt. In these cases, missing or ambiguous task context makes it difficult to understand if incorrect answers are due to lack of capability or poor instructions.

Grading Threshold Sensitivity

Other tasks have numerical answers with tolerance thresholds that allow a range of valid analysis methods to pass. Sometimes these tolerances were inappropriately calibrated, e.g. set too narrow, rejecting valid analysis paths the domain expert did not consider when framing the problem.

Main Results

To gain confidence that SpatialBench tasks were adequately specified, we needed evidence that each expected answer could be reproduced by an independent expert attempt.

We constructed a human-verified subset of SpatialBench called SpatialBench Verified comprising 115 tasks. Each problem in SpatialBench Verified had at least one human expert independently reconstruct an answer that passes the task graders from just the task prompt and associated data.

We found the filtered subset preserved the relative ordering of model performance, but scores increased by 11.6 pp on average.

Example: Problem Ambiguity

This sa_02_microglia_oligo_inflammation_correlation task asked agents to partition a mixed gene list into microglial activation and oligodendrocyte inflammation signatures, score cells, find neighboring oligodendrocytes around each corpus callosum microglia using an “appropriate radius” and compute Spearman correlations at 90wk and 4wk.

With just the task and no solutions, a human reviewer returned:

  {
    "spearman_90wk": 0.14,
    "spearman_4wk": 0.29,
    "interpretation": 2
  }

The existing grader expected:

  {
    "spearman_90wk": 0.32,
    "spearman_4wk": -0.17,
    "interpretation": 2
  }

The reviewer actually agreed with the intended qualitative interpretation (‘2’): age-associated spatial coupling between activated microglia and inflamed oligodendrocytes. But their notes indicated the exact numerical values were not recoverable from the task as specified.

Why?

If you look at the original problem statement, open choices include:

• how to split the mixed gene list into two signatures

• whether shared inflammatory genes can appear in both signatures

• how to normalize the low-depth targeted MERFISH panel

• what radius counts as “neighboring”

• whether to pool sections or control within section

• whether the provided 50k subset has enough power for the 4wk correlation

A variety of defensible analysis methods were therefore possible given the task context, making it difficult to trust the eval result.

Example: Grading Threshold Sensitivity

This visium_bone_norm_within_niche_coexpression task asked agents to define bone-enriched Visium spots as the top quartile of COL1A1 (symbol for a collagen gene) counts, then compute the Pearson correlation between COL1A2 and SPARC within those spots. The desired biological conclusion was that COL1A2 and SPARC are strongly co-expressed in the bone compartment, consistent with a coordinated osteoblast matrix-secretion program.

Once again, without solutions and just the task, the human reviewer reproduced the intended biological conclusion and reported:

  {
    "n_bone_spots": 735,
    "col1a2_sparc_corr": 0.8514,
    "interpretation": 1
  }

The grader expected:

  {
    "n_bone_spots": 762,
    "col1a2_sparc_corr": 0.62,
    "interpretation": 1
  }

The reviewer’s notes actually found the correlation is highly normalization-dependent despite relatively stable biological interpretation. This is a clear example of a grader tolerance that was too narrow for the goal of the eval.

Methods

We conducted an initial round of review where 6 domain experts solved all 159 problems in SpatialBench at an average of 26 problems per expert. Human experts were given access to the task prompt and associated data, and were asked to produce an evaluation answer in the same format as the agent answer.

For each problem, experts were asked to share both the final answer they arrived at and a Python Jupyter notebook or R Markdown document using any bioinformatics libraries they thought appropriate. All answers were graded using the benchmark’s existing grader, with binary pass/fail scoring (all answer fields must be correct).

94/159 (59.1%) of problems passed the first round of review. Upon review of the failing cases we found that failures fell into several categories:

1. Poor solution effort

2. Valid effort but incorrect analysis

3. Unclear mismatch between human solution and ground truth

4. Ambiguous or weak eval specification

The first category of failures was mainly due to time constraints - over a week experts were asked to solve an average of 5-6 problems a day which proved to be unrealistic. To mitigate this, we carried out a second round of review.

For the second round of review we included all 65 problems that failed the first round as well as 10 passing problems to serve as a control. The passing problems were randomly sampled across the 5 different spatial technology types used in SpatialBench (2 per kit). Problems were assigned over a larger pool of 28 experts (3 problems per expert). Experts were given 2 days to produce solution artifacts for their assigned tasks.

Experts in the second round of review started afresh with just the task prompt and associated data. All problems were assigned to different experts between rounds.

Passing solutions were produced in Round 2 for 21/65 of the failing problems from Round 1. From the control set, passing solutions were produced for 8/10 problems in Round 2.

We took the union of problems that passed Round 1 review (94/159) and the 21 additional problems that passed Round 2 review, yielding 94 + 21 = 115 problems in SpatialBench Verified.

Supplements

• Subset of solution data

• Task assignment sheet

• Instruction document

Authors

Anirudh Narsipur, Deborah Hayoun, Ananya Gupta, Shivang Sharma, Benjamin Kesler, Zachary Hemminger, Sahar Nasr, Aashka Bhowmick, Sahiti Marella, Zhen Yang, Shon Kurian George, Soo Hee Lee, Qian Xu, Lior Schachaf, Harihara Muralidharan, David Calcagno, Birendra Saye, Asmita Lagwankar, Meenakshi Somadasan, Alex Urrutia, Ian Diks, Hannah Le, Kenny Workman

24 evaluations span primary tumors, organoids, xenograft models, lineage-tracing systems, and aging/intervention biology. The best agents score 11.1%.

Read the manuscript.
Interact with the leaderboard.

Benchmark Construction

Evaluations mix the assay types scientists use in practice.

A single task may depend on spatial transcriptomics, histology, single-cell references, and lineage-recording data. Solving them requires cross-assay reasoning, experimental-design awareness, and command of spatial workflows like tissue segmentation, niche analysis, and spatial differential expression.

This tests the transition from running data analysis to doing science.

Constructing ground truth in science

Deriving ground truth is very hard in long-horizon biology benchmarks. The same data can support multiple valid conclusions, and some published claims do not reproduce cleanly under unbiased reanalysis.

Candidate tasks are hardened through independent reproduction, randomized expert review, and trajectories from multiple model families.

Pairing deterministic grading with diagnostic rubrics

Grading uses deterministic functions over structured final answers. We grade recovery of scientific conclusions expressed through controlled biological vocabularies instead of numbers from individual statistical operations.

Results

Across 15 model-harness pairs and 1,080 trajectories, Gemini 3.5 Flash / Pi, GPT-5.5 / Pi, and GPT-5.5 / OpenAI Codex each passed 8/72 attempts (11.11%), with Claude Opus 4.6 / Claude Code close behind at 7/72 attempts (9.72%).

Judge rubrics over analysis ‘chokepoints’ provide interpretive tools

However, final-answer grading provides sparse diagnostic signal for long-horizon tasks. Chokepoint rubrics - analysis decisions expected to remain stable across plausible solution paths - are graded by judges as a companion diagnostic.

But rubric judging is probably not a replacement for verifiable grading

We were interested in exploring the utility of rubric grading for intermediate reward and asked if denser signal was partially aligned with endpoint quality. We conclude rubric scores are promising auxiliary tools, not substitutes for verifiable endpoint grading.

Rubric judging is reasonably stable over judge and agent model variation

We also stratified rubric scores by the source agent model whose trajectory was being judged to see if rubric patterns are consistent across judges. All four judges preserved the same broad ordering with some variability.

Actually looking at your trajectories will continue to be important

Pairing manual trajectory review with rubric and verifiable scores provides more tools to interpret model failures.

In practice, manual trajectory inspection is a first-class tool to understand this data. Eval authors maintain reproduction notes to provide a record for future benchmark updates, especially as stronger models may solve tasks through unanticipated but valid analysis paths that challenge current grading assumptions.

There is a path to agents that behave a lot like scientists

Results suggest compounding local analysis errors prevent reliable long-horizon scientific reasoning.

Before models can reliably reason about disease mechanisms, drug response, or other deep results in biology, they must become procedurally competent in local steps.

But the few task completions we observed were very impressive. It seems there is a realistic path to agents thinking and behaving a lot like scientists do.

Read the manuscript
Interact with the leaderboard

On accuracy, Gemini 3.1 doubles, Opus 4.7 shows modest improvement and GPT 5.5 shows no significant improvement.

On latency, GPT 5.5 improves greatly, Opus 4.7 shows no significant improvement and Gemini 3.1 worsens by ~5x.

Full data and example evals/trajectories available on benchmarks.bio.

Models are showing steady improvement in generic data analysis ability

Trajectory review reveals improvement in exploratory behavior:

• Thoroughly investigating data and thinking through experimental context before picking which analysis algorithm or tool to use

• Self-verification after finishing steps or submitting answers

• Early signs of assay-aware analysis steps (eg. QC thresholds) specific to kit/machine type instead of generic tutorial steps. Largely explains the jump in Opus 4.7 accuracy.

Models still struggle with biology-specific analysis details

Many remaining evals require knowledge of experimental design, kit/machine-specific analysis details, or tacit scientific knowledge. Some examples:

• Aggregating to the correct statistical unit before testing (ex: inappropriately treating cells as statistical replicates, inflating power in statistical tests)

• Applying whole transcriptome scRNA-seq defaults to targeted 30-400 gene panels (ex: BD Rhapsody)

• Accurately recognizing and removing depth/batch confounds (ex: PC1 absorbs library size)

• Integrating data from multiple platforms (ex: DNA + RNA on Tapestri, protein + mRNA)

Explore data and trajectories on benchmarks.bio

We regularly update our benchmark family with new models: benchmarks.bio. Encourage those interested in understanding what these benchmarks actually measure to inspect sample tasks and trajectories.

Introduction

It’s important to remember just how surprising the rise of software engineering agents was. A year or two ago, most professional engineers thought of AI tools as code generation assistants - useful for formatting, refactoring, or scaffolding unit tests - but never capable of grokking complex requirements, modifying large codebases, or making systems-level engineering decisions.

Today, 4% of all public GitHub commits are authored by Claude Code and Mythos is uncovering thousands of day-zero vulnerabilities *autonomously* in workhorse software. Agents build browsers and compilers. Many of my most skeptical peers now use multiple agents in parallel to tackle daily, high stakes work.

In hindsight, many people treat this capability growth as inevitable. Software, after all, is “obviously verifiable”: unlike other domains grounded in the messy, physical world, code runs deterministically on computers. But software is more than code and frontier agents are starting to show higher-level engineering judgment on open ended tasks.

I believe biology will follow a similar path. Research workflows in biology will reorganize around agents for the same reason software did: both domains contain a layer of work that is concrete, executable and feedback-rich. In software, that layer is code. In biology, its measurement-grounded data analysis.

From this analogy, we’ll build up three claims using real examples:

First, the first useful biology agents will be data-analysis agents, not autonomous scientists, just as coding assistants came before increasingly autonomous engineering agents.

Second, assay-specific data analysis is needed for true biological reasoning to emerge.

Third, the analysis layer becomes economically more important as molecular data generation grows.

Software fell first because code is executable and feedback-rich

To understand what I mean by “both domains have a layer of work that looks the same”, let’s first dig into what that means for software.

Software is written in code, which has some nice properties. You can run it quickly through a compiler and get outputs and errors. You can easily inspect the intermediate state of the digital objects you build up in a program by printing values or writing them to files.

Code is then an ideal substrate for agents to learn tasks. They can write and run chunks, inspect outputs and loop. But the useful things we build in software are far more than just code. They incorporate subjective and somewhat open-ended engineering decisions, many inherited from the domain in which they’re used and the problems they solve.

It is then surprising agents developed the capability to autonomously construct, for example, browsers. “The verifiability of code” does not explain the progression from passing tests to making nontrivial systems decisions about concurrency, caching, latency and security.

The obvious reason is skill in higher-order concepts emerged after focused training on the procedural skill of code is because you have to get to the parts of the problem where this knowledge is useful. It’s difficult to learn how to rate limit an endpoint if you struggle with Python function syntax or are unable to reason correctly about event loops.

But more subtly, and important to the analogy with biology, is detailed knowledge of your building material informs the higher-level decisions. It’s difficult to make good browser-level decisions (caching, rendering) without understanding the lower-level behavior of code.

Biology has a similar analytical substrate

So what does any of this have to do with biology?

Most modern biology papers are centered around large-scale measurement experiments and share the following structure:

1. Choose a biological model (cell line, animal, organoid) and variables/controls

2. Generate data from the model

3. Process the data

4. Creatively thinking about the analytic results in context of prior literature

5. Make scientific claim

If you trace how an agent might work through this flow to tackle research autonomously, I argue:

1. Analysis of assay-specific data analysis is necessary to reach stages where scientific reasoning is useful

2. Scientific reasoning tasks are shaped like high level software engineering tasks, both in complexity and their dependence on underlying procedural skill in data analysis for accuracy.

So the existence of a thick underlying layer of molecular data analysis scaffolds scientific thinking in a similar way that code scaffolds more complex software engineering judgement.

The best way to make this concrete is with an example.

An example: reprogramming fibroblasts into brain organoids

Let’s take a look at a recent study from Gordon et al. where authors harvest fibroblasts (very common ‘connective’ cells) from 55 autism patients, reprogram them into stem cells with Yamanaka factors and turn each stem cell into a brain organoid. Their central question is “do the different mutations in each patient affect the development of autism”.

This is a good example of the type of project (structure, scale, difficulty) we might want an agent to tackle in the future.

We can immediately see the question admits no “global verifiability”. It’s unclear how to produce a correct value to grade against (or what type of value would even make sense). A ground truth would depend on dozens of intermediate observations, some without clear scientific consensus: how organoids are quality-controlled, how cell types are annotated, how gene expression modules are defined.

But we can decompose the paper into smaller steps with clearer edges, some even with “locally verifiable” results.

Cell line QC

First, the agent compares mutations in whole-genome sequencing and RNA-seq data to quality control stem cells.

We grade the agent on which cell lines it drops and which mutations it recovers for each line.

Differential expression

Then, the agent looks across cell lines to recover genes that shift the most over time (as the cells develop).

We grade the agent on the agreement with the genes recovered by the author.

Perturb-seq

We then use CRISPRi to actually “knock down” these genes in the lab. The agent analyzes this Perturb-seq data to identify which of these genes had a functional effect when it was repressed.

We grade the agent on the genes the authors concluded were important.

Each of these analysis steps start from raw measurement data and build up to a well-defined scientific outcome. All of them are necessary for a scientist to draw a conclusion about the overarching question.

Data analysis is the gateway to scientific reasoning

As shown above, agents can decompose biological research into concrete analysis steps, with each step building upon the previous one. Let’s now get a sense of the higher-level tasks remaining after the agent works through the data:

• Compare data from different assay types to validate results: bulk + sc-RNA-seq, IP-MS + Perturb-seq

• Compare expression across technical (sequencing lane, differentiation batch) and biological (donor, cell line) covariates

• Scour existing literature for prior biological evidence (eg. known disease associations or functions of candidate genes)

These tasks are much more “AI Scientist” shaped. They require synthesis across modalities, judgment under uncertainty, and knowledge of prior biology. But they are not obviously more complex than the kinds of open-ended engineering judgment agents are beginning to perform in software. The practical bar is matching the judgment of a strong scientist: reasoning from the available measurements, controls, analysis choices, and literature to the conclusion best supported by the evidence.

And because this biological judgement is empirical, the agent must reason from noisy data and understand analysis assumptions well. Developing scientific judgement then requires mastery of the analysis steps and cannot be separated from it.

For instance, an agent would need to understand the statistical choices made in the differential expression analysis. Decisions like which samples are included, how genes are filtered, how covariates (like sequencing lanes) are factored out can modify the results. An agent could conclude “a gene is downregulated” because these genes genuinely co-vary across mutation or because the gene is capturing a hidden variable in the data: developmental maturity, cell-type composition, sequencing depth, some temperature in the lab, etc.

Another example is how metadata is constructed. If the agent is looking for genes that regulate development, the exact way it constructed the cell type labels in the data need to be understood. Depending on how we define “cells in early development”, low count genes in this population could surface from genuine biology or an incorrect definition of this population with subjective marker genes.

An agent with an understanding of these assumptions is equipped to make accurate scientific decisions. If it did the analysis itself, it also has these details fresh in context.

Where the analogy breaks

Biology is among the last frontiers of the empirical sciences. The ‘objects’ of study (living organisms) span angstrom (protein) to micron (cell) to meter (leg) scale, synthesizing phenomena across chemistry and physics. The measurements we use to study them are produced by instruments and protocols on the frontier of human knowledge with noise and error.

This complexity means the analogy to software breaks in at least two places.

First, biological ground truth is more difficult to define. Even apparently well-scoped tasks like clustering cells, annotating cell types or calling differentially expressed genes rarely have one canonical answer. This is an active area of research, with ongoing work to develop practical methods and tools to constrain ambiguity and model scientific intent verifiably. It remains very difficult.

Second, feedback is much weaker once we leave the analysis layer. For data analysis, the feedback loop still looks somewhat like software: run code, inspect outputs, loop. However, true feedback for higher order scientific reasoning requires some control over the data-generating process. We will likely make a lot of progress using ground truths constructed and graded a lot like code without control over this loop. But true autonomous science requires direct feedback with the substrate being studied.

This layer becomes more important as biology scales

Agents will need to develop data analysis capability first (at least for real work) and accurate scientific reasoning seems in part dependent on a detailed understanding of analysis. It’s important to now understand how much more important analysis might be in the context of data generation trends.

Molecular data generation follows an exponential curve and analysis costs are rapidly becoming more expensive than reagents/prep labor with new assay generations. An increasing proportion of biological work now happens after the experiment has finished in the lab.

If you play out these trends, the bottleneck will shift from producing measurements to interpreting them. The question is not if but when. Future datasets will dwarf the stock of past biological data, and human capacity to analyze them will not scale at the same rate. The answer to this will be agents spinning in harnesses that reliably take data to scientific conclusions, deployed in R&D workflows across biotech where data is bottlenecked today.

The surface area of computational biology is growing

People often equate ‘computational biology’ with narrow analysis tools, like sequence assembly or differential expression software. While these have been important contributions of the field, computational biology should really be thought of as “the broad use of computers to solve problems in biology”.

For years, the components have been building for a computational shift in biology: rapidly compounding molecular data, increasingly automated labs, and iterative design-build-test-learn workflows. Yet these advances have not translated into an obvious step-change in aggregate R&D productivity.

That now appears to be changing. The industry is reorganizing around AI and computing, in how experiments are designed, data is analyzed, companies are formed, and pharma structures partnerships. Computers are becoming the interface through which agents “see”, understand and eventually manipulate living systems.

The first useful deployment of these agents will not be autonomous AI scientists but analysis collaborators embedded in real workflows. This is both where true need lies and what they will get good at first. Biology has no shortage of interesting new ideas. The bulk of economically valuable agentic work is concentrated where computers can make progress on R&D cycles, augmenting scientists where they are today.

Thank you to Aidan Abdulali, Anirudh Narsipur, Brandon White, Kyle Giffin, Alfredo Andere, Aashay Sanghvi, David Yang, Hannah Le, Lada Nuzhna, Tim Proctor for revising and/or contributing to the thinking in this essay

GPT-5.5 nearly halves runtime relative to GPT-5.4, yet accuracy remains flat: 57.6% versus 57.4%. Opus 4.7 is similarly tied with Opus 4.6: 52.4% versus 52.8%.

Scientist-reviewed trajectories reveal gaps in assay-aware biological judgment: statistical design, spatial units, batch structure, and scientific interpretation.

Complete benchmark data and select trajectories are available on benchmarks.bio.

What SpatialBench measures

Spatial biology is a powerful measurement tool and an important category for agentic capability. Analysis workflows require a combination of code and biological reasoning: agents must handle large data, understand platform-specific details, contextualize with scientific goals and return a quantitative result that matches what a diligent scientist would compute.

SpatialBench measures this work: 159 spatial biology analysis tasks across platforms like Xenium, Visium FFPE, MERFISH, TakaraBio Seeker, and AtlasXomics DBiT-seq. Each task starts from a real analysis state and asks an agent to recover a specific biological result. The grader checks structured outputs against expert-derived references (a subset of examples available here).

While frontier models show an improvement in speed and step count, they do not improve in overall accuracy on this benchmark.

GPT-5.5 is substantially faster than GPT-5.4, cutting mean duration roughly in half and using far fewer steps. But its accuracy is effectively unchanged: 57.65% versus 57.44%. Opus 4.7 is also effectively tied with Opus 4.6: 52.41% versus 52.83%.

Accuracy by platform

Platform-level grouping shows GPT-5.5 improves on Visium, Xenium, and MERFISH, but trails GPT-5.4 on TakaraBio and AtlasXomics.

Similarly, Opus 4.7 leads Opus 4.6 on Xenium by 11.1 percentage points, ties on TakaraBio, and trails on Visium, MERFISH, and AtlasXomics:

A look at manually annotated trajectories

Trajectory review identifies recurring failure categories across model families:

• Treating cells, beads, spots, or barcodes as independent observations when the biological replicate is the donor, animal, tissue section, or timepoint

• Applying scRNA-seq normalization defaults to spatial platforms where they are inappropriate

• Clustering multi-sample data without integration, then interpreting donor or timepoint structure incorrectly

• Confusing spatial marker units for cells or anatomical structures

• Failing to recover correct de novo niches, tissue states, and spatial compartments

We’ll dive into some examples to get a sense of failure modes in the context of realistic tasks. Each task trajectory was reviewed by a scientist with years of experience on the specific spatial platform being evaluated.

Barcode-level statistics create false positives

The AtlasXomics SPATIAL10_genome_wide_de_pct task asks the model to test 24,919 genes for sex differences in a human dorsal root ganglion DBiT-seq dataset. The dataset contains about 10,000 spatial barcodes from 8 donors: 3 female and 5 male.

Barcodes are nested within donors. A researcher would aggregate at the donor level, computing about 1.2% differentially expressed genes. The agents consistently ignore this donor information.

GPT-5.4 and GPT-5.5 both report 93.876% on all six runs. Both Opus 4.7 and Opus 4.6 report roughly 92-94% of all genes as significantly sex-different. Note this interpretation is biologically implausible: sex cannot plausibly alter chromatin accessibility at 93% of all genes in 8 donors.

Housekeeping genes expose the same pseudoreplication error

The SPATIAL07_sex_housekeeping_de task asks whether 10 housekeeping genes show sex differences in the same spatial ATAC-seq design. The expected answer is none.

Trajectory review reveals models call 9-10 housekeeping genes significant. Both Opus 4.7 and Opus 4.6 models call all 10 housekeeping genes sex-different in every run. GPT-5.4 and GPT-5.5 call 9-10 significant housekeeping genes. Calling ACTB, GAPDH, and other constitutive genes sex-differential is a clear sign of pseudoreplication: the models treat thousands of barcodes as independent replicates, artificially inflating statistical power and manufacturing significance.

Targeted-panel normalization flips myelin coexpression

The MERFISH norm_02_myelin_gene_coexpression_normalization task asks for the Spearman correlation between Mbp and Plp1 in oligodendrocytes. These are myelin structural genes and should be positively co-expressed. The expected value is about 0.308.

GPT-5.5 fails all three runs, consistently producing -0.157. Human review shows GPT-5.5 applies a normalization step that over-corrects targeted-panel counts, turning a positive biological relationship into an apparent anti-correlation. GPT-5.4 passes 2 of 3 runs with correlations around 0.326 by avoiding that specific normalization path.

Both Claude models report a Spearman correlation around -0.16 between Mbp and Plp1 on every run. In this case, the negative correlation is an artifact of library-size normalization on a 374-gene targeted panel where a few myelin genes dominate total counts. The models import the familiar scRNA-seq code:

sc.pp.normalize_total(adata, target_sum=1e4)

sc.pp.log1p(adata)

on a targeted MERFISH panel, rather than handling the platform appropriately.

Batch-confounded clustering survives newer models

The batch_driven_clustering (TakaraBio) and NORM01_batch_correction (AtlasXomics) tasks test whether models integrate across donors or timepoints before interpreting clusters.

Both GPT-5.4 and GPT-5.5 fail TakaraBio batch_driven_clustering. The expected max single-timepoint fraction is about 0.375; GPT-5.4 reports 0.967, 0.995, and 0.995, while GPT-5.5 reports 0.990, 0.994, and 0.988. Each cluster is dominated by a single timepoint, meaning the clustering is tracking condition or batch rather than cell type.

The same issue appears in AtlasXomics. In NORM01_batch_correction, the expected mean max sample fraction is 0.375, but the GPT computed values remain around 0.866-0.897. The Opus 4.7 trajectories independently describe the same AtlasXomics failure: without integration, PCA captures inter-sample technical variance, Leiden clustering partitions cells by sample of origin, and neither model questions why clusters in a multi-donor dataset are dominated by single donors.

Spatial units are usually not cells

TakaraBio Seeker uses 10um beads. A single large oocyte can span many beads, with RNA diffusing across all of them. Counting marker-positive beads as cells or anatomical structures inflates biological counts.

The oocyte_count_per_timepoint task tests this. The expected immature oocyte count is 850. GPT-5.4 reports 1237-2086, and GPT-5.5 reports 1510-3463. At the 0h timepoint, expected count is 275, while models report 424-821.

The same report highlights cumulus_gc_count_immature: the expected cumulus granulosa cell count in immature samples is 0 because cumulus cells have not differentiated yet. GPT-5.4 reports 435-1474, and GPT-5.5 reports 1424-2395. Both models assign cumulus identity from marker expression without applying the developmental constraint that cumulus differentiation requires hCG stimulation.

The Opus 4.7 report describes the same class of failure in spatial segmentation terms. On follicle_count_immature, Opus 4.7 counts span 50 to 456 across runs because small changes in DBSCAN radius produce wildly different connected components, while Opus 4.6’s stricter oocyte scoring gives a cleaner input set for spatial clustering.

Niches are hard to predict

The Xenium spatial_fibro_inflammatory_niche_emergence_2 task asks the agent to reconstruct a fibro-inflammatory niche across kidney injury timepoints. The expected pattern is low sham co-localization, high Day14 co-localization, and a fold increase of 6.9.

GPT-5.4 reports sham co-localization of 0.112-0.124 and fold increase of 2.26-2.32; GPT-5.5 reports sham co-localization of 0.121-0.373 and fold increase of 1.99-4.23. The expected sham value is 0.033, expected Day14 is 0.23, and expected fold increase is 6.9.

Opus 4.6 and Opus 4.7 find niches containing fibroblasts and immune cells, but they cannot enforce the specific composition ratios that distinguish the pathological CN7-like niche from adjacent healthy stroma. They confuse generic fibroblast-immune proximity with the disease-organized fibro-inflammatory compartment.

More development of common failure behaviors

Replicate-aware statistics: Models often misunderstand how to define replicates. When donor, animal, section, or timepoint is the appropriate variable of comparison, models incorrectly reach for cells, beads, and barcodes as independent observations, inflating significance and creating false positives.

Platform-aware normalization: Models treat spatial assays as interchangeable and often reach for scRNA-seq defaults. The same normalization procedure can overcorrect targeted MERFISH/Xenium panels, confound sparse Visium FFPE correlations with depth, distort bead-capture signal, or discard curated marker genes through HVG selection.

Batch and donor integration: Models frequently cluster before asking what the axes of variation represent. In multi-donor or multi-timepoint data, uncorrected PCA/Leiden structure often separates samples, batches, or timepoints; the model then mistakes this separation for cell type, tissue state, or treatment biology.

Spatial unit and denominator errors: Models confuse expression and coordinate units with cells or other structures.

De novo spatial niche discovery: Models can assemble the mechanics of niche analysis but often miss the biological goal. They confuse generic proximity or broad regional enrichment for specific compartment the task asks for: a follicle, lineage, pathological niche, or disease-organized tissue state.

Assay-specific benchmarks are important

While GPT-5.5 and Opus 4.7 are faster or locally stronger in specific task families, they are not meaningfully more reliable on SpatialBench.

Future improvements for spatial biology are unlikely to come from general reasoning gains alone and will require explicit training on statistical design, platform-specific analysis steps, replicate-aware differential testing and other spatial biology knowledge. Focused, assay-specific benchmarks are required to appropriately measure the complexity of biological data analysis accurately.

Here we introduce scBench (read the paper), a benchmark of 394 verifiable problems drawn from real scRNA-seq workflows. It spans six sequencing platforms, seven task categories, and eight frontier models. The best model reaches 52.8% accuracy. Better than spatial, but it means the best agent in the world still fails nearly half the time on routine analysis tasks.

scBench tests what bioinformaticians actually do

Existing biology benchmarks reward textbook recall or literature-style reasoning. Real analysis requires loading messy datasets, writing code, making judgment calls about thresholds and parameters, and producing concrete quantitative results. scBench recreates these conditions.

The benchmark covers 394 problems across six sequencing platforms (Chromium, BD Rhapsody, CSGenetics, Illumina, MissionBio, ParseBio) and seven task categories (QC, normalization, dimensionality reduction, clustering, cell typing, differential expression, and trajectory analysis). Each problem provides a data snapshot - typically an AnnData `.h5ad` file-captured right before a decision point, a natural-language task prompt, and a deterministic grader that scores the agent’s structured JSON output as pass or fail.

We pressure-tested each problem for shortcuts: removing precomputed embeddings, stripping cached labels, and ensuring answers can’t be guessed from prior knowledge alone. Cell typing (118 problems, 30%) and differential expression (71, 18%) dominate the benchmark because these are the stages where dataset-specific judgment matters most and where agents struggle hardest.

Agents are better at scRNA-seq than spatial - but still not reliable

Across eight frontier models from four providers, accuracy ranges from 29% to 53%.

Claude Opus 4.6 leads at 52.8%, followed by Opus 4.5 (49.9%), GPT-5.2 (45.2%), and Sonnet 4.5 (44.2%). The bottom tier - GPT-5.1 (37.9%), Grok-4.1 (35.6%), Grok-4 (33.9%), and Gemini 2.5 Pro (29.2%) - trails by a wide margin. The 23.6 percentage point spread between best and worst exceeds SpatialBench’s 18.3 point spread, meaning scBench discriminates model capability even at higher overall accuracy.

The accuracy bump over SpatialBench likely reflects training data. scRNA-seq has far more public datasets and Scanpy dominates the ecosystem with extensive documentation and tutorials.

Tasks reveal a consistent difficulty gradient

Not all analysis steps are created equal. Normalization is easiest (cross-model mean 70.4%), followed by QC (55.3%). These are procedural---apply a well-known transformation, check a metric against a threshold. Agents handle them reasonably well.

The story changes for judgment-heavy tasks. Clustering drops to 38.3%, cell typing to 34.9%, and differential expression to 27.0%. Seven of eight models follow this same difficulty ordering. DE is also where models diverge most: 27.7 percentage points separate best from worst.

The pattern is clear. Tasks that require contextual scientific reasoning-selecting marker genes, interpreting cluster identity, choosing statistical tests, identifying tissue-specific signatures-are where agents break down. General-purpose coding skill is necessary but not sufficient.

Platform choice matters as much as model choice

This is perhaps the most striking finding. Cross-model mean accuracy ranges from 59.1% on CSGenetics to 26.4% on MissionBio---a 32.7 point gap that exceeds the 23.6 point spread between best and worst models. Platform matters more than which model you use.

CSGenetics is easiest for six of eight models. MissionBio is hardest for all eight. The MissionBio collapse is dramatic: even the best model (Opus 4.6) only hits 42%, and Gemini drops to 10.3%. Every model shows large platform swings---Gemini drops 42 points between its best and worst platform. Even Opus 4.5, the most consistent model, loses 39 points.

MissionBio inverts the overall rankings in interesting ways. Grok-4 (sixth overall) beats GPT-5.2 (third overall) on MissionBio. Sonnet 4.5 surpasses GPT-5.2 by 11 points. Models that memorized Scanpy tutorials without learning transferable analysis techniques collapse on platforms with non-standard data structures and less common tooling.

These effects almost certainly reflect training data composition. Chromium and Illumina dominate public repositories and documentation. MissionBio and ParseBio appear less frequently. Reliable agents will need platform-aware context and assay-specific tooling, not one-size-fits-all reasoning.

scBench complements SpatialBench across single-cell modalities

Together, scBench and SpatialBench cover the two dominant transcriptional assays. The top model reaches 52.8% on scBench versus 38.4% on SpatialBench-scRNA-seq is more tractable. But the structural patterns are shared: normalization is easiest in both, platform effects drive 30--40 point swings in both, and model rankings are preserved at the extremes (Claude Opus leads both, Gemini ranks last in both).

The benchmarks form a complementary diagnostic. scBench tests whether models can handle the most common and best-documented assay type. SpatialBench tests whether that competence transfers to newer, less standardized technologies. Together they reveal whether an agent has learned general analysis reasoning or just memorized Scanpy workflows.

Limitations

Deterministic grading enables verifiable evaluation but necessarily discretizes scientific judgment into automatically checkable chunks. Each evaluation snapshots a single workflow step rather than capturing the long-horizon iteration where errors compound and thresholds are revisited. Real analysis is messier, more iterative, and longer than any single problem captures. But measuring step-level competence reliably is a prerequisite for automating longer workflows.

Where this is going

scBench confirms the pattern SpatialBench established: agents for biology are in a capability regime where they can accelerate routine analysis but cannot yet be trusted to autonomously answer scientific questions without human oversight. The path forward is a long tail of tractable engineering: platform-specific context, better harness design, and exposure to representative workflows across diverse biological contexts.

We’re building toward a family of benchmarks spanning major biological modalities, each an evolving formalization of the tacit knowledge and judgment that working scientists bring to data analysis. The goal is test-driven development of agent systems that improve through both model training and harness engineering.

Code, canonical evaluations, and full trajectories are available at github.com/latchbio/scbench.

This gap is starting to matter. Spatial assays (and increasingly every modern measurement modality) generate outputs that are too large, too irregular, and too context-dependent for most labs to analyze without computational specialists. The promise of agents is clear: point them at raw data, ask scientific questions in plain language, and compress weeks of analysis into hours. But looking right isn’t the same as being right - especially when results guide expensive and time intensive experiments.

Software engineering arguably went through this phase already. Frameworks like SWE-bench exposed how bad models were at real programming tasks and gave everyone a number to climb. Progress in biology needs something similar: built from real workflows, grounded in real datasets, and graded by verifiable outcomes instead of vibes.

Here we introduce SpatialBench (read the paper), a suite of spatial biology analysis problems designed to recreate the work scientists actually do - inspect messy images and count matrices, write and run code, reason scientifically and recover concrete biological results.

SpatialBench approximates real analysis work

Current biology evaluations largely reward textbook knowledge, but real analysis involves empirical interaction with data to produce answers. SpatialBench is a suite of 146 verifiable problems drawn from real spatial analysis workflows, spanning five platforms (Vizgen MERFISH, Takara Seeker, 10x Visium, 10x Xenium, Atlasxomics DBIT-seq) and seven task categories (QC, normalization, dimensionality reduction, clustering, cell typing, differential expression, and spatial analysis).

Each problem is a snapshot of an analysis workspace captured right before a decision point: typically an AnnData object plus any associated files, images, or outputs from earlier steps. The tests probe for durable biology, robust to survive reasonable choices of algorithm, library, or hyperparameters. We pressure-tested each for shortcuts: inspecting logs, adversarial prompts and removing anything solvable by pattern-matching or leaning on general scientific knowledge instead of actual analysis. Find concrete examples in the paper supplements.

Agents aren’t quite there yet

Across the full benchmark, base-model accuracy sits between 20-38% depending on the model family, with the usual tradeoffs between accuracy, latency, and cost.

But the aggregate numbers hide detail. Stratification reveals clear model-specific variation over tasks and platforms. Some model families collapse on specific task types while others swing 15 to 20 points depending on the class of spatial technology.

Spatial biology is of course a collection of many distinct technology types and constituent steps, each with its own artifacts and conventions obscured in point estimates of accuracy. Deliberate engineering will be needed to improve across each covariate.

Harnesses are more than “glue code”

Interestingly, harness design moved outcomes as much as swapping the base model. We ran Opus-4.5 under multiple execution ‘wrappers’ - LatchBio, Claude Code and a slightly modified mini-swe-agent - and saw dramatic variation.

Our results suggest we should start treating harnesses as first class objects in engineering and benchmarking efforts. The tools, prompts, and control flow around a model are as important as the model itself.

Agent trajectories reveal distinct problem solving strategies

Manual inspection of agent trajectories - the sequence of reasoning traces, tool calls, logs from an eval session - was perhaps the most informative part of the project. This data provides mechanistic explanations for performance differences. We could pick out distinct failure modes: models burning steps on formatting with no forward progress; using general QC thresholds that don’t hold for spatial data. Some models were able to ‘productively explore’ (inspect, compute, refine, make some forward progress) while others thrashed (same failed attempt on repeat, often forever). There are many examples of these in the paper.

Limitations

Verifiability comes with tradeoffs. Nuance is lost trying to draw deterministic lines around messy real world examples. Snapshots of single steps aren’t full workflows: real analysis is longer-horizon and iterative. But measuring step-level chunks with reliability is but a step towards automating truly long time horizon tasks. More on this soon.

Where this is all going

SpatialBench is a first contribution towards a broader family of benchmarks spanning major biological modalities. We view benchmarks as evolving formalizations of tacit scientist behavior and analysis workflows. They guide test-driven development of agent systems through model training and harness engineering that will eventually automate much of what we consider to be computational biology.

2026 will be a big year for agents in biology.

Read the paper. Check out data + code. Play with some examples.

Try it here: agent.bio

We’ll go over:

• Our roadmap for reliable and widely deployed spatial agents

• Important considerations while playing with these sandboxes

• Video demos of each agent

Our roadmap for reliable and widely deployed spatial agents

LatchBio develops agents for spatial biology data that allow scientists to direct their own analysis with natural language. By tailoring each agent to specific spatial kits and machines, and iterating relentlessly on this reduced scope, we believe it’s possible to build accurate systems that can be used to make expensive scientific decisions.

This release represents a shippable intermediary towards this goal, both to show technical progress and help others see why we’re so excited about this direction. The path ahead is a long tail of soluble engineering work: continued progress on benchmarks, purpose built infrastructure and technology specific heuristics encoded into prompts/evals.

Important considerations while playing with these sandboxes

We’re developing two agentic modes, “step-by-step” and “proactive”, that provide control over how often the agent will check-in, ask questions and provide figures or statistics for scientists drivers to sanity check intermediate steps. The sandboxes default to “proactive” to provide a nice first impression of end-to-end functionality, but we encourage biologists to start with “step-by-step” for any work of value. More on this soon.

Furthermore, the sandboxes were built for specific datasets and tasks. You can try to use your own data, and the system will likely generalize accurately on some tasks, but we provide no guarantees on performance quite yet.

Video Demos

The following videos and descriptions motivate some of the practical end-to-end workflows possible with the system. We highly encourage the reader to play with each themselves.

Takara Seeker

Ovulation failure affects over 10% of infertility cases in women of reproductive age in the United States, yet it is exceptionally difficult to study because ovulation unfolds within tiny, spatially restricted niches and evolves on the scale of minutes.

Using a Takara Seeker 3x3 kit, Mantri et. al built a spatiotemporal molecular atlas of the ovulating mouse ovary across 8 different time points.

To replicate the analysis, the Takara agent ingested raw H5AD files, removed off-tissue beads, computed QC metrics and filtering, performed clustering, and ran differential gene expression. The agent then manually reasoned through top marker genes for each cluster and identified cell types key to the ovulation process, such as mural and cumulus granulosa cells, mesenchymal subtypes, oocytes, and endothelial and immune populations. A scientists can then ask the agent to count the atretic or antral follicles.

Vizgen MERFISH

The mouse brain contains highly organized cellular circuits, yet these structures are difficult to resolve without single-molecule spatial data. MERFISH overcomes this barrier by capturing the precise locations of individual transcripts across intact tissue.

Using a MERFISH mouse brain dataset, the agent built a spatial neighbors graph, computed Moran’s I, and performed neighborhood enrichment and co-occurrence analyses. It identified spatially variable genes such as MYH11, SLC17A7, and DAAM2, each marking distinct anatomical regions; MYH11 was correctly enriched in blood vessel-associated cells, consistent with its vascular smooth muscle role. Neighborhood enrichment further recovered expected spatial organization, including co-localization of excitatory and GABAergic neurons and strong oligodendrocyte self-enrichment, matching known cortical and subcortical structure.

AtlasXomics DBiT-seq

Tumor heterogeneity is a major challenge in cancer, arising both between patients and within a single tumor. Changes in chromatin accessibility are a key driver of this variability because they alter gene expression, cell identity, and interactions in the tumor microenvironment.

Using AtlasXomics spatial ATAC-seq (DBiT-seq) from a stomach cancer sample and a matched control, the agent ran an ArchR-based workflow to identify differentially accessible chromatin regions, peaks, and motifs.

In tumor tissue, the agent detected strong enrichment of CTCF/CTCFL motifs, pointing to disrupted 3D chromatin organization, a hallmark of many cancers. It also found IRF1 enrichment, consistent with immune infiltration or localized inflammatory signaling. By contrast, the control tissue was enriched for FOXA1 and TEAD motifs, which support normal epithelial identity and Hippo pathway-mediated tissue homeostasis.

10X Xenium

Alzheimer’s disease reshapes the brain through localized inflammatory niches and region-specific neuronal vulnerability. These patterns only become visible only with high-resolution spatial transcriptomics.

Using a Xenium dataset from a TgCRND8 Alzheimer’s mouse brain section, the agent loaded the raw outputs, built an AnnData object, performed clustering and differential expression, and annotated cell types using a controlled mouse brain vocabulary.

The analysis recovered five major neural populations: astrocytes, excitatory neurons, inhibitory neurons, oligodendrocytes, and microglia. It also detected Alzheimer’s-relevant signals such as strong Gfap spatial enrichment marking reactive astrocytes and Hexb expression marking activated microglia. Finally, the workflow identified 21 spatial domains that correspond to known anatomical regions and disease-affected microenvironments, including glial clusters surrounding neurons, a characteristic pattern in Alzheimer’s models.

10X Visium

Colorectal cancer is shaped by tightly packed tumor, stromal, and immune niches that often sit only a few microns apart, making high-resolution spatial profiling essential for understanding disease architecture.

Using the Visium HD colorectal cancer dataset with more than 500,000 spots, the agent converted Space Ranger outputs into an H5AD and launched a GPU-accelerated RAPIDS workflow to handle the dataset at scale. The full pipeline from QC to clustering to differential expression completed in under a minute.

The agent then inspected top marker genes for each cluster and assigned epithelial, stromal, and immune cell types, reconstructing the organization of the colorectal tumor microenvironment.

Path Ahead

We’ve learned a lot about biology specific agent engineering in the past month building and saturating our benchmarks (eg. what is the analogue of a “code diff” for spatial analysis). Expect more educational material on these topics in the coming weeks.

High scores on SpatialBench represent expert and industrially relevant analysis ability across four different spatial technologies, tissue types and diseases. We benchmarked our spatial agent against Sonnet 4.5, GPT-5-Codex and Qwen 2.5 Coder in a minimal harness (a slightly adapted version of the Bash only SWE-bench harness). These early results show considerable engineering is needed to extend computer use agents to biology. There is a lot of work remaining to saturate these benchmarks.

We’re also releasing a batched test harness, eval spec and a representative subset of tasks/datasets on GitHub. While we built this tool to measure and improve our own product, we hope others find the open resources useful.

SpatialBench evals represent practical analysis tasks

This benchmark is focused on real world analysis work and each task was constructed actual chunk of work from our customers and their scientists.

Each eval is a JSON object conforming to a standard structure: a task description, data node and grader configuration. This information is injected into a system prompt and the agent performs work in a sandbox with access to data. A grader then evaluates the answer.

Here we demonstrate some concrete examples to build some intuition for what the benchmark is actually measuring. The specs for these evals are available in the repo.

Normalization

Feature Selection

Dimensionality Reduction

Clustering

Differential Gene Expression

Domain Detection

Results demonstrate need for biology specific agent engineering

While the benchmark does not represent an apples-to-apples comparison of systems, eg. a ~100 line mini-swe-agent harness wrapping Sonnet is significantly less complex than our system, the purpose of the exercise was to show what worked for coding will not necessarily work for biology. We will be improving the harness and expanding the eval bank over the coming weeks.

Putting a model in a simple loop with access to a shell is sufficient to saturate SWE-bench, but agents for spatial biology likely need purpose built infrastructure, tools and context for useful work. More on the details here.

Evals encode tacit knowledge of frontier research biology

The frontier of research biology - practical knowledge about details of different diseases and tissues - is distributed across hundreds of labs, in the minds of PIs and researchers. By working with these labs to interpret spatial data from different tech, we can encode their knowledge and preferences into evals used to train and benchmark agentic systems. These eval banks will become valuable, structured documentation about the procedural details of research as they grow.

Add your spatial technology to our benchmarks

We are expanding SpatialBench to include all relevant machine and kit types. Reach out to our team to include your technology in our next release.

Unlike most experiments in biology, the computational steps in spatial can match or exceed the resources spent on upstream work in the wet lab: a tricky combination of the data being large (>1TB a run for CosMX) but the analysis high-touch, multi-turn and very unique to each scientists’ questions. This is unfortunate because most biologists have a very clear idea of what they are trying to do, often able to describe the end results/plots with great detail and steer intermediate steps based on their biological goals. But the practical complexity of using spatial tools complicates the translation of the correct ideas to the right analysis. Tens of thousands of scientists across industry and academia are stuck analyzing spatial data .

Agents are LLMs in loops that plan and execute end-to-end analysis with natural language directions by interacting with their environment and calling tools. They have shown incredible promise in math and programming, but require intentional engineering to translate well to biological data. Latch spatial agents are tailored to the details of each kit, in direct collaboration with the companies that sell them, so they have the specific context, tools and structured analysis flows required for each type of data. They build scientific notebooks with interactive widgets running on large machines and launch workflows on purpose built data infrastructure engineered for scale.

We are in the early days of this emerging category, but the initial technical results and response from scientists indicate this is likely the future of data analysis. There is considerable work left to do, but it seems base reasoning and agentic capabilities of frontier models are essentially here, and the remaining structure of the problem is a long tail of incremental but very soluble engineering improvements. In the coming months, tens of thousands of scientists will be able to finally play with their own data and answer their own questions.

• Agents for biological data are purpose built for the domain

• Spatial data is a strong focus for early adoption

• What do agentic workflows look like in practice?

  • AtlasXOmics DBiT-seq Agent

  • Takara Trekker Agent

  • Vizgen MERSCOPE Agent

  • 10X Xenium Agent

Agents for biological data are purpose built for the domain

Agents need new capabilities to reliably and accurately accomplish practical tasks with biological data. Unlike software engineering products, these systems need to run in sandboxes with considerable computing and storage resources to manipulate large data. They also need to offload particularly intensive steps to optimized workflow infrastructure; single operations, like processing sequencing reads or clustering counts, can take hours and require clusters of computers. Using pre-existing bioinformatics components as tool calls, like Nextflow or Snakemake workflows, also serves to constrain end-to-end workflow variability and allows models to focus on genuinely novel components of user requests.

Furthermore, the driver of the agent is a scientist who reasons about results with plots, charts and graphical components. Latch agents communicate with their users by building scientific notebooks, rich with markdown description, that also serve as self-documenting dashboards that describe the analysis in detail.

Perhaps most important, each agent is tailored to the specific type of machine or kit that generated the data. We work directly with the companies developing these assays when possible - Vizgen, TakaraBio, AtlasXOmics - to encode latent knowledge about the specific details, hidden footguns and ideal customer journey into prompts, tools and evals. Rather than an open-ended sandbox that does many things somewhat well, our philosophy is to build a surgical and highly reliable product that assay providers can trust. We believe the bar for widespread adoption of agents is an intelligent, adaptable system that does not jettison the determinism of traditional software.

Spatial data is a strong focus for early adoption

It is important to find specific tasks that are uniquely suited to agentic workflows and cause enough existing pain to encourage scientists to try a new tool. Spatial data checks both of these boxes. This is a new, rapidly growing but notoriously challenging domain of science. There are no clear analysis standards, the ecosystem of tools is highly fragmented and generally difficult to use, and unlike traditional sequencing, there are many different, composable steps. Yet the power of these assays is remarkable, in the new biology they help us understand and the beautifully intuitive way they allow scientists to see what’s going on.

In many ways, it is convenient agentic systems are beginning to actually work. It is unclear exactly what the solution to these problems would have been with traditional software: likely many variations of hard coded dashboards that each tackle a specific task in a reliable but brittle manner. We believe the interplay between spatial biology, which is going through its own revolution, and agentic data analysis is a special, reciprocal relationship. These complementing technologies will allow scientists to ask more, understand more and all the while consume more kits and compute.

What do agentic workflows look like in practice?

The following are four concrete end-to-end agentic workflows built in collaboration with real scientists. A tailored agent is built for each kit or machine type. Scientists begin with raw data, send instructions or questions to a chat interface (eg. “Process my Vizgen data”), and watch the agent build graphical scientific notebooks to accomplish tasks.

AtlasXOmics DBiT-seq Agent

AtlasXomics develops an assay called deterministic barcoding in tissue sequencing (DBiT-seq) that allows spatially-resolved epigenome profiling. Microfluidic chips deliver barcodes to precise tissue coordinates, uncovering relationships between chromatin accessibility, histone modifications, and context-specific gene expression within intact tissues.

This first flow highlights the design principles discussed earlier: large agent sandboxes to handle the TBs of file data, the use of existing bioinformatics workflows developed by the AtlasXOmics team and specialized visualization components (H5, IGV viewer) required for this data type.

1/ Bioinformatics Workflows as Tool Calls

The AtlasXOmics agent begins by building interfaces to handle user input and dispatching bioinformatics workflows to purpose built data infrastructure. Reads are filtered, aligned to the reference genome, and converted into fragment files.

It understands the relevant files to pull from the output folder - filtered fragments, spatial position files, motif enrichment scores and gene activity scores - so scientists can immediately continue with downstream steps. Scientists can ask questions about workflow parameters at any point.

2/ Image Alignment

Linking molecular measurements to precise tissue locations requires careful alignment of sequencing data with high-resolution tissue images.

The agent guides scientists through this process using an interactive interface, providing instructions on how to perform alignment and suggesting adjustments when needed. The result is a spatially-aligned dataset where each molecular measurement is mapped to its exact tissue coordinate, creating the foundation for all downstream spatial analysis.

3/ Spatial Region Selection

Once the data is aligned, scientists need to define regions of interest based on tissue morphology or biological features. The agent enables researchers to use intuitive lasso-select tools to define custom regions

It can suggest regions based on expression patterns or guide selection based on natural language descriptions like “select the tumor border region.” These defined spatial regions are saved as metadata, ready for comparative analysis across different tissue areas.

4/ Differential Chromatin Analysis

Understanding regulatory differences between conditions, spatial niches, or cell types requires identifying differential chromatin accessibility regions and motifs. When scientists ask questions like “What chromatin regions are more accessible in the tumor core versus periphery?”

The agent launches the appropriate differential analysis workflow as another bioinformatics workflow, automatically navigating the Latch file system using the LPath API to find required inputs, and configures parameters based on the scientific question. The analysis produces differential peak results, motif enrichment comparisons, volcano plots, heatmaps, and gene regulatory network differences.

5/ Subclustering

Discovering subtle cell populations or states within major cell types often reveals distinct regulatory profiles crucial for understanding biology. When scientists want to explore specific cell populations in detail, the agent subsets the data based on user criteria and launches re-clustering workflows on the subset.

It then repeats annotation and differential analysis on subclusters while maintaining the full analysis provenance in the Latch Plots notebook environment. This iterative process yields refined cluster definitions and subcluster-specific regulatory signatures.

6/ Visualizing Chromatin Accessibility Tracks with IGV Browser

Understanding how chromatin accessibility differs between clusters requires examining the actual peaks and signal intensities at specific genomic loci. When scientists want to validate differential accessibility results or explore regulatory elements in detail, the agent launches an IGV browser session and automatically loads cluster-specific chromatin accessibility tracks as bigWig files, with each cluster displayed in distinct colors for easy comparison.

Scientists can navigate to genes of interest by simply asking “Show me chromatin accessibility at the IL2 locus across T cell clusters,” and the agent configures IGV to display the relevant tracks, adjusts the viewing window, and highlights differential peaks. The agent can also overlay additional tracks like gene annotations, known enhancers, or transcription factor binding sites, providing the full genomic context needed to interpret spatial epigenetic differences.

Takara Trekker Agent

TakaraBio offers two spatial transcriptomics platforms. This flow focuses on Trekker, a single nuclei spatial transcriptomics kit.

We show the agent’s ability to gather experimental context, launch chemistry-aware bioinformatics workflows, guide scientists through tissue image alignment, and discover insights through multi-sample differential gene expression analysis.

1/ Gather Experimental Context

The chemistry of each assay dictates the structure and content of its sequencing output and therefore the architecture of its downstream computational analysis. To enable robust interpretation, we codified assay-specific scientific logic into our agentic framework, allowing our Latch agent to reason about experimental design, platform chemistry, and expected data modalities.

The agent is trained to ask clarifying questions about the kit chemistry, sample type, and sequencing platform before constructing the appropriate workflow. This ensures that each dataset is processed using a computational path aligned to its biochemical origin, enabling accurate spatial reconstruction and downstream biological analysis.

2/ Chemistry-specific Bioinformatics Workflows

The workflow ingests paired FASTQ files alongside output from a single-cell platform (e.g., 10x or BD Rhapsody). Each read encodes a cell barcode, molecular identifier, and spatial barcode. The Takara spatial informatics pipeline identifies nuclei carrying valid spatial barcodes, reconstructs the spatial index, and generates a cell × gene matrix that integrates molecular expression with spatial localization.

When prompted with, “Process FASTQ files from a Takara Trekker experiment,” the agent dynamically generates a configuration interface to capture parameters such as the single-cell platform, kit version, and output format. It then invokes the appropriate Latch `w_workflow`, which executes on dedicated high-performance infrastructure to produce an H5AD object ready for visualization in the Latch plotting environment.

2/ Image Alignment and Tissue Visualization

Seeing gene expression overlaid on tissue structure is crucial for understanding spatial data. However, aligning H&E-stained images with spatial coordinates is challenging and often requires manual tweaking. The agent simplifies this using built-in image registration tools in the H5 viewer.

When prompted, the agent automatically focuses on the tissue region and overlays the H&E image with the spatial data. For ovarian tissue, this alignment reveals how gene expression maps to anatomical features: follicles appear as distinct circular structures, corpora lutea show as larger irregular regions, and the surrounding stroma displays its own expression patterns. This visual overlay lets researchers immediately connect molecular data with tissue architecture.

3/ Differential Gene Expression Across Follicular Subtypes

We also demonstrated the agent’s ability to conduct differential gene expression analyses across ovarian follicle subtypes—preantral, antral, and atretic—within Seeker-derived datasets. This analysis established a spatially resolved baseline for folliculogenesis, linking molecular identity to anatomical localization, revealing spatial gradients in follicle maturation, and identifying novel, spatially restricted gene markers critical to understanding ovarian development.

When extended to temporal dynamics, the agent identified stage-dependent changes in gene expression following hCG induction, including a decline in Fabp3 and Inhbb levels over time, marking the transition of mural granulosa cells from a proliferative, follicle-maintaining phenotype to a luteinizing, pre-ovulatory state.

Vizgen MERSCOPE Agent

Vizgen’s MERSCOPE platform uses multiplexed error-robust fluorescence in situ hybridization (MERFISH) to map hundreds to thousands of RNA species simultaneously at subcellular resolution. Each experiment generates terabytes of data: millions of transcript coordinates, multi-channel z-stack images, and complex metadata files.

We show the agent’s ability to process TB scale MERFISH human brain imaging data, launch GPU-accelerated cell segmentation workflows, enable interactive visualization of millions of transcripts in the browser, and perform spatially-aware differential expression analysis through natural language commands.

1/ Cell Segmentation

Cell segmentation determines how millions of detected transcripts get assigned to individual cells. MERSCOPE captures five imaging channels across seven z-planes: DAPI (nuclei), three Cellbound channels (membranes), and PolyT (cytoplasm). Creating accurate cell boundaries requires selecting the right channel combinations and configuring GPU-accelerated algorithms like CellPose or Baysor.

The agent automates this complexity by scanning imaging directories and generating configuration files tailored to each tissue type. For dense tumor regions, it prioritizes membrane channels; for sparse samples, it leverages Baysor’s transcript-based approach. It then launches a GPU-powered workflow on Latch infrastructure to process the segmentation.

The workflow outputs three files: (1) an AnnData object compatible with downstream tools like Scanpy and Squidpy, (2) PMTiles (a pyramidal format originally developed for web mapping that enabling smooth zooming and panning), and (3) a DuckDB database containing transcript coordinates and metadata, providing a lightweight SQL engine for querying millions of transcripts in real-time.

These files are displayed in the Latch H5 viewer, where scientists can seamlessly overlay cells, transcripts, and high-resolution morphology. They can also filter transcripts by gene, spatial region, expression level, or cell type while exploring tissue architecture at any zoom level.

If segmentation needs refinement, scientists can immediately instruct the agent to rerun with adjusted parameters, creating a rapid iteration cycle for optimal results.

A video of agent performing all analysis steps end-to-end for Vizgen data, from QC, normalization, finding HVGs, to cell type annotation and differential gene expression

2/ Quality Control

With cells defined, quality assessment becomes critical. MERSCOPE’s large imaging area can harbor technical artifacts: edge effects, tissue folds, or regions with poor probe penetration that must be identified before analysis. The agent generates spatial QC plots revealing these issues, allowing scientists to set informed filtering thresholds that preserve biological signals while removing technical noise.

3/ Clustering and Cell Type Annotation

MERFISH panels contain 200-1000 genes carefully selected to distinguish cell types in specific tissues. The agent exploits this focused design, performing clustering at resolutions that match expected cellular diversity. Initial annotation queries reference databases for marker patterns, then scientists refine through natural language: “split the myeloid cluster into monocytes and macrophages” or “identify proliferating cells across all types.” Spatial context often resolves ambiguous populations. Morphologically similar cells may occupy distinct tissue compartments that clarify their identity.

4/ Differential Expression Analysis

With cell types defined, scientists often ask what makes populations distinct or how they change across conditions. When comparing “tumor cells in the core versus edge,” it accounts for spatial autocorrelation and multiple testing across the focused gene set. Results highlight not just which genes differ, but their spatial patterns: gradients of metabolic genes from hypoxic cores to vascularized edges, or immune activation markers concentrated at tumor-stroma interfaces.

10X Xenium Agent

10X Genomics’ Xenium platform uses in situ hybridization to detect hundreds of RNA species at subcellular resolution. To demonstrate the agent’s capabilities, we’ll analyze an Alzheimer’s disease mouse model dataset tracking pathological changes across disease progression.

We show the agent’s ability to parse complex multi-timepoint Xenium datasets, refine cell segmentation in challenging brain tissue, detect spatial domains through BANKSY clustering, and calculate disease-specific gene signatures that reveal spatially-organized neuroinflammatory responses around amyloid plaques.

1/ Loading and Merging Xenium Data

The dataset compares wild-type and CRND8 transgenic mice at timepoints representing mild, moderate, and advanced amyloid deposition. Scientists attach Xenium outputs from coronal brain sections profiled with 347 genes: 248 from the Mouse Brain panel plus 99 custom genes targeting activated microglia, astrocytes, and plaque-inducible markers.

The agent parses the complex directory structure, constructs an AnnData object preserving spatial coordinates, and generates diagnostic plots.

2/ Cell Segmentation Refinement

Brain tissue presents unique segmentation challenges. Neurons range from tiny interneurons to large pyramidal cells spanning hundreds of microns. Around amyloid plaques, activated microglia cluster densely, creating cellular aggregates that standard segmentation often merges incorrectly.

When prompted to “improve segmentation around microglial clusters,” the agent launches a refined workflow. Cellpose first identifies nuclei using DAPI, providing reliable seeds even in crowded regions. Baysor then expands these boundaries guided by microglial transcripts like Iba1 and Cx3cr1. This approach captures the complex morphology of activated microglia wrapped around plaques–essential for accurately quantifying neuroinflammatory responses across disease stages.

3/ Spatial Domain Detection

With proper segmentation complete, the agent reveals tissue organization through BANKSY clustering.

4/ Cell Type Annotation

Standard cell typing identifies expected populations, but the key insights come from disease-specific signatures. When scientists prompt to “compute PIG signature scores within microglia and astrocytes,” the agent executes a targeted analysis of plaque-induced genes, a curated set of markers that distinguish reactive from homeostatic glia in Alzheimer’s pathology.

The agent first identifies microglia and astrocytes using canonical markers, then calculates PIG scores incorporating genes like Apoe, Cst7, and Axl for microglia, and Gfap, Vim, and Clu for astrocytes. These scores are mapped spatially, revealing striking patterns: microglia closest to plaques show the highest PIG scores, forming a reactive barrier, while astrocytes display elevated scores in broader halos around pathological sites.

Try it

Shoot us an email at contact[at]latch.bio to get onboarded. Lots of work to be done, but we have certainly seen early signs of the magic. We believe this will soon be the standard way biologists interact with data.

Every biotech executive has the same nightmare: a data breach halts your trial, exposes your critical IP, and scares off future investors. GDPR exists to prevent this nightmare; in short, it is Europe’s strict privacy rulebook (but, unlike most lab SOPs, this one comes with fines up to €20 million when things go wrong).1

Despite its reputation in biotech as a bureaucratic hurdle, GDPR reflects principles that not only protect data, but in practice, handle it to an end that supports the reproducibility and validation of experimental output for clinical translation.

For solution providers, GDPR’s value is clearer; it presents a commercial advantage and aligns your analysis with the same values that patients, clinicians, regulators, and software solutions already demand: accuracy, transparency, and accountability. It creates a common standard upon which geographically-diverse client data moves seamlessly from research to clinic, across international borders, and through tools without losing integrity along the way.

This post will apply key GDPR principles to the lab, operationalizing them with LatchBio’s approach to default-compliant product development.

First, an Overview of Compliance at LatchBio

LatchBio’s Global Compliance Program is designed to optimize the speed of your discovery without sacrificing its security. We work closely with both technical and regulatory partners to ensure the highest level of compliance and confidence for our customers:

• Drata provides 24/7 compliance monitoring visible to you,

• NAMSA advises on biotech-specific regulatory positioning,

• Tenax Solutions conducts annual penetration testing,

• MJD Advisors audits the integrity of our commitments each year, and

• GRCI Law employs our EU- and UK-based GDPR representatives and advises on international transfers.

Our aim is to meet every client’s regulatory requirements and work closely to provide regulatory support needed from a data processor hosting critical scientific infrastructure. To this end, we are compliant with a range of US and international frameworks, and quickly work with our customers when they need custom requirements or to comply with a different regulatory framework.

Latch is compliant with HIPAA, SOC 2, GxP, CLIA, CCPA/CPRA, and more. Visit our Trust Center to view our extensive security and compliance documentation.

Data Provenance and Reproducibility for Patient Rights

Across industries and countries, the public increasingly demands visibility into what companies know about them (e.g., requesting a copy of your personal data from Meta, or asking 23andMe to permanently delete your data). GDPR codified this cultural shift into law by giving patients clear rights to access, correct, or delete what companies know about them,2 including biotechs and solution providers.

GDPR and complementary frameworks have requirements that are satisfied by immutable logs of processing history (e.g., every workflow, data transfer, analysis step, or single data manipulation),3 with evidence of who did what and when. If done poorly, this could result in a disarray of paperwork and messy file systems (think graduate student’s desktop with folders inside folders, labeled “final_v7_actually_final).”

Latch’s architecture treats auditability and provenance as defaults. The full history of manipulation and record is available without manual reconstruction. The result is a system aligned with GDPR’s principles of accountability and transparency.4

The value of this requirement and solution can be interpreted beyond data subject rights. It allows scientific teams to reproduce an experiment exactly as it was run, returning to prior states and tracing how data evolved over time without guesswork to confirm the reliability of results.

Principle of Least Privilege in Modern Bioinformatics Management

Research data moves constantly, between collaborators, instruments, and workflows. Each transfer introduces potential for error: a file overwritten by someone who didn’t need editor access, a dataset manipulated without context, a result tied to the wrong version of an input. GDPR formalizes what lab operational excellence already demands — that access to data is deliberate, limited, and accountable.5

The principle of least privilege6 is, at its core, a safeguard against that kind of entropy. It requires admins to limit access to only what each user needs for their role and to keep detailed records of those interactions. In other words, it encodes good lab hygiene into law.

On Latch, this takes form through granular role-based access control (RBAC). Workspace owners define broad roles (viewer, editor, or admin) and then granularly customize permissions within those roles. Before high-impact actions like data export or deletion, user identity is re-verified7, and Latch engineers would have already been monitoring automatic signals for suspicious log-in behavior to report to admins.

The benefit isn’t just compliance; it’s coherence. Least-privilege access enforces experimental discipline: teams work from a single, trusted source, data integrity maintains intact, and the UI of each employee is specifically designated by the admin. What GDPR describes as “security of the processing” is, in practice, the same discipline required for reproducible, multi-stakeholder science.8

Commercial Advantage of GDPR Compliance for Biotech Solution Providers

In any pharma sale, the client’s procurement team will ask for a key set of GDPR-required artifacts, to size up a vendor’s readiness for regulated data transfer:

• Standard Contractual Clauses (SCCs) define the terms for international data transfers9,

• Data Processing Agreement (DPA) outlines how a processor manages and protects a controller’s data10,

• Record of Processing Activities (ROPA) documents what data is handled, by whom, and for what purpose11,

• Data Protection Impact Assessment (DPIA) evaluates risks in higher-impact processing, such as clinical or genomic data12,

• Transfer Impact Assessment (TIA) assesses legal and technical safeguards for cross-border data13.

For solution providers, familiarity and preparation with these isn’t bureaucracy; it’s evidence of comprehensive risk management and sufficient understanding of complex international compliance. It shortens procurement cycles, and removes one of the most common blockers to large-scale deployments. Teams that can present these materials quickly are viewed as reliable partners who understand both the regulatory and operational realities of handling sensitive patient-derived data.

Latch integrates these same requirements directly into its onboarding and platform architecture. SCCs, DPAs, and supporting assessments are maintained and reviewed with independent auditors, creating a security and documentation layer that travels with your work. This foundation allows solution providers to engage with regulated customers without delay, building credibility while keeping the focus on scientific delivery rather than administrative process.

In Conclusion: Global Data, AI Models, and the New Relevance of GDPR

As science globalizes, GDPR’s reach extends naturally with it. Sequencing data, pathology images, and cellular atlases now cross continents daily to train AI models or power multi-omics pipelines. These datasets almost always contain information traceable to European individuals (directly or indirectly) which places them squarely under GDPR’s jurisdiction.

For solution providers, this means that data sovereignty and privacy controls are now core to scientific scalability. When you train an AI model on European datasets, deploy compute in the U.S., or serve results to a collaborator in Asia, GDPR follows the data at every step.

Far from being an obstacle, this consistency enables collaboration. Standardized privacy frameworks like GDPR create a shared security language between institutions, reducing friction in multinational research.14 They let scientists exchange data and model outputs with confidence that each participant is adhering to compatible privacy and accountability standards.

In this sense, GDPR has become part of the infrastructure of global bioinformatics: a framework that allows sensitive patient-derived data to move across borders, feed algorithms, and return as clinically meaningful insight, all while preserving the trust that underpins scientific progress.

If you are a solution provider interested in white-labeling these tools and offering them to customers through a branded portal, email contact@latch.bio.

LatchBio’s Compliance Team is available at compliance@latch.bio.