How Good is Opus 5 at Biology?
An in depth analysis across our benchmark suite
Opus 5 results across our biosecurity, therapeutics, and -omics benchmarks are now live on benchmarks.bio, to generally positive results. I’ll share high level numbers first, and then a deeper dive into model and harness quirks we noticed across some of the 4,674 trajectories we generated.
A 1000 Foot View
We ran Opus 5 at maximum effort under both the Claude Code and Pi harnesses, here’s how it stacked up with other models:
Highlights: Opus 5 is the best model we’ve tested on variant discovery and interpretation (VariantBench, +7.6pp over Sol 5.6), small-molecule preclinical pharmacology (TxBench-PP, 7.0pp over Sol), genomic surveillance (Surveillance, +7.3pp over Sol), and long-horizon single-cell analysis (scBench-Long, +3.2pp over Sol). It beats every previous Anthropic model we have run, taking 6 of 7 benchmarks over Opus 4.8, most by 8 to 16 points.
Midlights: On short horizon single cell analysis tasks it lands two points behind Sol (60.1% vs 62.1%), still first among Anthropic models. On long horizon spatial biology tasks it more than doubles Opus 4.8 (25.0% vs 11.1%) but loses to Sol (38.9%).
Lowlights: It regresses on epigenomics analysis tasks, with Opus 4.8 scoring 4.2% higher than it.
Full per benchmark results are on benchmarks.bio, so I won’t report them here.
Opus 5 with the Pi harness is on the pareto frontier for short horizon tasks with its high accuracy at low costs, tokens, duration, turns, and duration per turn. With the Claude Code harness, there are minor improvements in accuracy across metrics that allow it to stay at the frontier, albeit for much more expensive resource use. For long horizon tasks, Opus 5 is uniformly beaten by Sol across all metrics.
Below, we’ll discuss:
Model Quirks: Context Management, File System Management, Tool Calls, Statistics Use, and Long Tail Timeouts
Harness Quirks: Deliberation, Hypothesis Forming, and Self-Auditing
Model Quirks
Context and a Multi-Step Reasoning System
Opus 5 has drastically decreased the thinking characters per action. Averaged across runs, the reasoning it writes per action (chain-of-thought characters divided by that run's tool calls) is 137 for Opus 5 against Opus 4.8's 207 and Sol's 490; the medians rank the same way (104, 194, 434). Across a whole run Opus 5 averages 2,971 characters of visible reasoning to Opus 4.8's 7,288, less than half.
We observed Opus 5 keeping more of its working memory on the filesystem: it writes a script, checkpoints the result to a .h5ad or a log, and reads it back when needed. Because it splits work across separate processes, intermediate objects have to be persisted and reloaded (a.write('primary_qcmetrics.h5ad'), then sc.read_h5ad(...)), so the bulky tables reach the transcript only as small summaries. On one CosMx run it went further, maintaining a running ANALYSIS_NOTES.md that it appended to and re-read. Opus 4.8’s single-pass style instead tends to print those intermediates to stdout, so more of the analysis flows through its transcript.
In general, we also noticed Opus 5 manipulates the filesystem in smaller chunks than Opus 4.8. 77% of scBench-Long runs and 72% of SpatialBench-Long runs decompose into a numbered, checkpointed pipeline. At the extreme it wrote a 36-stage-pipeline, s1_load_qc.py through s36_tcr_state.py, the last dozen stages sensitivity and control checks beyond what the task asked for. It used an independent de-novo re-clustering and an orthogonal TCR-contig test to throw out a plasma-cell “enrichment” that was really a doublet artifact (“rejected as a size/doublet artefact, HLA-DR-negative, ~2× the RNA of singlet counterparts, no TCR excess”) and to confirm the real signal (“C1q-high macrophages showed a real excess of TCR+ events, OR 3.6, p=5.6e-16”).
In the aforementioned CosMx run, it wrote its own driver scripts to fan out parallel jobs, cat > drive2.sh launching two s03_immune_subcluster.py processes with wait $P1; wait $P2. Compare this to Opus 4.8, which wrote a single 4,392-character deconvolution pipeline and ran it in one pass.
Across the board, we find that Opus 4.8 writes more, but shorter, commands. This is especially true on long-horizon benchmarks. Opus 5 tends to launch 4 to 5 background jobs per run with nohup and polling loops that babysit them. In contrast, Opus 4.8 stays monolithic, running fewer and much longer whole-pipeline scripts, but still in the background. Sol, on the other hand, seldom schedules any background jobs.
Opus 5 issues commands at a higher tempo than Opus 4.8 (2.8 per minute vs 2.0) yet throws fewer errors per run (1.4 vs 1.8). When turns do throw errors, it typically seems to recover in a single turn. It also looks before it computes: roughly 40% of its shell commands are pure inspection (ls, cat, grep, head), which is about the same split as Opus 4.8. There are more commands and shorter commands, but we did not notice a reliability tax for the fragmentation when it comes to managing the filesystem.
Tool Calls
By name, Opus 5 is overwhelmingly a shell user: about 27 Bash calls per run, then a tail of Write (2.9), Read (1.6), WebFetch and WebSearch (about one each; it consults the literature), and the orchestration tools TaskCreate, TaskUpdate, and Monitor, with which it schedules and watches its own background work.
Opus 5 issues significantly more background jobs. On the long benchmarks Opus 5 launches 4.5 per run with nohup and polling loops, against Opus 4.8’s 1.4 and Sol’s zero.
Its tooling errors are mundane rather than reasoning failures. Across 400 runs the tool results carry 366 Python tracebacks, 151 ValueErrors, 91 ModuleNotFoundErrors, 79 file-not-found, 63 KeyErrors, 59 AttributeErrors, 30 command-not-found, and only 11 Killed/OOM. This killed/OOM category is pretty harmful, as we’ll explain later, since it can keep Opus 5 trapped in a vicious polling cycle.
Statistical Analysis
Opus 5 is fond of a statistical detour, and the trajectories are full of them: permutation tests, batch effect testing, and cross-validation.
This penchant for statistics is not new for the Opus suite. In general, Opus 4.8 and Opus 5 tend to issue more tests than Sol, and also a more diverse set of tests.
Which tests, specifically? Its statistical workhorse is the Wilcoxon rank-sum test for differential expression, in 41% of runs, with FDR multiple-testing correction (15%), t-tests (10%), negative-binomial and logistic GLMs (about 14% together), Spearman and Pearson correlations (roughly 6% each), permutation tests, and cross-validation filling out the tail.
More statistics does help on average, though, and the striking part is how cheap it can be. Splitting Opus 5’s 574 scBench runs into thirds by overall statistical intensity, the pass rate climbs 0.44 to 0.56 to 0.63 while median runtime goes 3.3 to 10.2 to 12.0 minutes: reaching the middle third roughly triples the time for a +0.11 gain, and the top third adds another +0.07 for two more minutes.
There are instances where increased statistical reasoning can be harmful, however.
For instance, take 2 different rollouts of a depth-confounded gene-counting task. The first diagnosed the confound and reported a cross-method consensus (“86 genes were significant in a majority (≥4/7)… 86 is the cross-method consensus”), passing in 24 minutes. A sibling rollout diagnosed the same confound and then spent 103 minutes running sensitivity sweeps and validation (“very robust (152-210), let me check sensitivity to the T-cell definition and cross-validate with an NB GLM”), and then ended up getting the problem wrong. The extra 79 minutes of robust checking didn’t help very much.
Long Tail Timeouts
Opus 5 sometimes runs for much longer than expected, especially on long horizon tasks. On scBench-Long its median run is 38 minutes to Sol’s 17, its 90th percentile is 116 minutes to Sol’s 38, and 9% of its runs pass 3 hours against 1% for Opus 4.8.
Given the higher timeouts that Opus 5 hit, we were concerned that we were obscuring the model behavior by restraining it to the max cutoff times that we gave our human experts. However, timeouts do not seem to change absolute ranking; if we delete every no-answer run and reweight, Opus 5’s SpatialBench-Long pass rate rises only from 25.0% to 27.8%, still about 11 points behind Sol. We also believe that these timeouts are important to enforce consistency with how long our human experts take.
What accounts for these long delays? Generally, Long staged pipelines are not the problem. 70% of the long runs that succeed use background staging, and the aforementioned 36-stage run from the earlier section finished fine. All three hard six-hour timeouts end the same way, parked on a polling loop waiting for a background job that never reports. For instance, “I’ll stop polling and end my turn here. The background monitor b7refde6z will surface the motif activity tables when the scan finishes…”, except nothing ever showed up to wake it.
The same reflex is often recoverable. On one run Opus 5 finished its answer but would not stop, reading the instructions as forbidding it to end a turn while a job was still running; it blocked on a stray helper through 17 minutes of timeouts, then killed it and was careful to note it (“it contributed no evidence and I’m not using any of its partial output”)
Harness Quirks
We paired 508 random runs between the Claude Code and Pi harness, and found some pretty stark differences in the behaviors of these two models. As we previously found, these differences might emerge because of a few sentences of behavioral prompting; however, Opus 5 appears to follow behavioral priors to an extreme we haven’t seen before.
Under Claude Code, Opus 5 deliberates about three times as much, 25.4k characters of thinking per run against Pi’s 8.8k. Its reasoning also interrogates the problem rather than tracking the next step. For instance, on a task which asks how many genes are genuinely differentially expressed once sequencing depth is confounded with the condition, the Claude Code run opens by questioning the setup:
wondering if the cell barcode itself encodes the donor identity… the task requires median depth within T cells, so I need to first…
where the paired Pi run just states the next action
“Now I’m moving on to cluster and annotate.”
These differences do not lead to major changes in accuracy. The two configurations agree on 87% of trials, so the model still settles the large majority. The other 13% pretty much evenly split between Claude Code being correct or Pi being correct, indicating the extra work Claude does rescues about as many runs as it breaks.
There are instances where this additional effort can help. On a task which asks how many monocyte subtypes exist, Pi clustered naively and over-split into 4, the mistake an unchecked clustering invites. In contrast, Claude Code audited its way out, pooling a treatment-free baseline and running bootstrap stability across 13 clustering resolutions before committing: “Bootstrap Jaccard stability (25 resamples × 13 resolutions)… one cluster (n=146, 1.8%) is monocyte-T doublets, an artifact. That leaves 2.”
On another task, which asks for a marker gene’s detection-based specificity, this instinct turns on it. Pi computed the number directly and shipped the correct 19.6%, while Claude Code computed the same ~19%, then talked itself out of it with an ambient-contamination hypothesis and instead reported 85%, flagging its own doubt on the way out: “binarizing on ‘Fabp7 detected’ gives 19% specificity, but this is contamination, not biology… I reported the band center, 85%.”
While these trends hold in general, there do seem to be biological domains where Claude Code outperforms Pi, and vice versa, so it may be worth choosing your harness based on the task. You can find a specific breakdown at benchmarks.bio.
Concluding Thoughts
Some highlights from our analysis:
Opus 5 is the strongest model we have run on most of this suite, beating Opus 4.8 on six of seven benchmarks and topping VariantBench, TxBench, Surveillance, and scBench-Long. But it does not lead the Pareto frontier across the board. Its configuration is great on short work, and Sol beats it on the long tasks.
Opus 5 reasons far more economically than Opus 4.8, writing less than half the visible chain-of-thought (about 3k vs 7.3k characters per run) and offloading its working memory to the filesystem instead of the transcript.
Its defining habit is small commands, checkpointed pipelines, and background jobs it babysits. This can sometimes cause a heavy runtime tail if the background jobs crash or timeout.
Opus 5 reaches for statistical tooling a little more than Opus 4.8, but this tooling does not always buy accuracy, and can sometimes burn large amounts of tokens and money.
The harness is a bigger knob than one may presume. The same Opus 5 under Pi costs about a quarter as much for the same accuracy; Claude Code often has more in depth analysis and deliberation, but this doesn’t translate to higher scores across the board. Which harness wins is often task-dependent, so it can be worth choosing based on the biological domain.