Why An Open Source Harness Outperforms Claude Code On Frontier Biology Tasks
In current generation models, behavioral priors introduce by harnesses such as Claude Code, and Codex cause substantial performance swings on our benchmarks.
At LatchBio, we benchmark AI models on frontier-level biology tasks - benchmarks.bio. We’ve seen a trend where harness changes cause large swings in performance on the same model. For example, on TxBench-PP (a short-horizon benchmark), the Pi harness beats Claude Code by 4.6pp (percentage points), and on ScBench-Long (long-horizon tasks), Claude Code wins by 4.8pp. What is causing these substantial performance variations between harnesses?
In this blog, I’ll analyze the performance of Opus 4.8 (max reasoning) in the Claude Code and Pi harnesses on *TxBench-PP and* ScBench-Long. I show that the root cause of these performance deltas is different behavioral priors set by the harness, such as asking the model to plan its work and verify its answer. We can prove this via ablation experiments that isolate prompting from the available tools.
Despite Codex having the largest delta relative to the Pi harness (8%), I decided to investigate the performance of Claude Code vs Pi TxBench-PP and ScBench-Long since I was most interested in why it performed worse on TxBench-PP but better on ScBench-Long.
Harnesses
The difference between Pi and Claude Code comes down to how much the harness shapes the model’s behavior and problem-solving approach.
Pi is an open-source harness that installs almost no behavioral priors, so the model’s biases drive the analysis unfiltered. It’s deliberately minimal, leaving the model to decide how to think. The baseline system prompt reflects this restraint: it establishes that the model is an expert coding assistant operating inside Pi—one that helps users by reading files, executing commands, editing code, and writing new files—and then offers only light guidance, asking the model to be concise and to show file paths clearly when working with files. Tool-call definitions get appended at runtime, but beyond that, the harness stays out of the way.
Claude Code takes the opposite approach. Built for long-horizon coding problems, it layers a full software-engineering persona onto the model. Its system prompt carries very strong behavioral priors that direct how the model acts and thinks: plan the approach and work from to-do lists, resist adding abstractions beyond what was prompted, review your work, verify before asserting (”you MUST verify”), and keep claims minimal and conservative. Several other priors round out the persona, though they matter less for this comparison. The net effect is a harness that guides the model toward a particular working style rather than letting its raw instincts take the lead. Furthermore, Claude Code supports a wide variety of tools, compaction techniques, and sub-agents in the experiment; we’ll show that these are not the key drivers of performance on these tasks.
Trajectory Examples
How the behavioral priors show up in the trajectories.
ScBench-Long — Claude Code’s caution prior wins (Pi −4.8 pp)
Example — MAPK prior-trap
This task provides scRNA‑seq data for a KRAS‑driven mouse lung tumor and asks which oncogenic pathway is most altered in cancer cells from aged vs. young hosts. The tricky part: aging genuinely raises inflammatory signaling (JAK‑STAT/”inflammaging”), so the largest increase in signaling looks inflammatory — but the ground truth is that the KRAS-defining MAPK growth axis is paradoxically reduced in aged tumors despite the identical driver. Both harnesses analyzed the dataset correctly (JAK‑STAT up, MAPK the largest-magnitude change, down). They diverged only at the final conclusion: Pi picked JAK‑STAT, justifying it with the prior on inflammaging, disregarding the more substantial MAPK reduction, whereas, in its passing trials, Claude Code correctly returned the more substantial MAPK reduction calculated in the dataset.
TxBench-PP — Pi’s minimalism wins (Pi +4.6 pp)
Example — over-analyzing into a false positive
The model is presented with a blinded microscopy image comparing three drug candidates (A, B, C) side-by-side in healthy vs. tumor organoids and asked which — if any — should advance. The right answer is “none”: no single candidate shows both tumor-killing activity and clean healthy-tissue safety, so nothing should move forward. The catch is that each candidate looks good on one axis, so it’s tempting to advance the “best-looking” one without considering both axes.
Pi read the image, judged that no candidate cleared both bars, and committed the no-go call in a lean ~8–12 steps. Claude Code kept investigating instead of committing: it wrote analysis scripts (4–6 per run) to quantify the panels over ~27–30 steps at roughly 2× the cost, and every round it talked itself into advancing a candidate (never “none”) — failing all 3 rounds.
In short, CC’s keep going until tests are resolved and verify prior manufacturers’ positive call; Pi’s minimalism doesn’t prompt the model to rethink itself.
Thoughts
The same Claude Code behavioral priors (plan, verify, be thorough, be conservative) that grounded it in the ScBench-Long reasoning tasks are what caused it to overthink itself on the shorter TxBench-PP tasks, and Pi’s near-absent prior has the mirror-image profile. Thus, I adopt the hypothesis that the harness’s behavioral priors are the cause of the substantial performance difference on biology tasks.
Experiment
To prove that behavioral priors (as opposed to tool availabilities) are responsible for the performance deltas, I ran an experiment that modified the Pi harness to include biological equivalents of the behavioral priors used in the Claude Code prompt. I modified the prompts rather than using the exact Claude code prompts because they seemed overly coding-specific, and I wanted to see how the biological equivalent might perform.
Prompts for experiments:
E1a — calibration.
Make minimal claims — only what the data robustly supports; when an effect is small, borderline, or within noise, report no robust difference rather than choosing a direction.
Claude Code prompt: conservative-claims posture — “Minimal changes … no improvements” / “Read before modifying.”
E1b — orchestration.
Use a todo list to plan and track your work and update it very frequently. Before finishing you MUST verify your result is correct.
Claude Code prompt: “Use the TodoWrite tools VERY frequently … that is unacceptable” + “you MUST run the lint and typecheck commands … to ensure your code is correct.”
E1c — full transplant: The E1a calibration clause and the E1b orchestration clause together.
Results
All values below are deltas vs the Pi baseline, in percentage points (pp). Model (Opus-4.8), tools, task set, and grader are held fixed; only the appended system-prompt clause changes, and the Claude Code arm is excluded. Benchmarks: TxBench-PP (short-horizon, 100 tasks × 3 trials) and ScBench-Long (long-horizon, 21 tasks × 3 trials).
Table 2 — Prompt-transplant deltas vs the Pi baseline (pp). Positive = better than Pi’s near-empty default.
The results indicate that the E1b (orchestration) arm performs most similarly to Claude Code and is likely the driver of the performance delta. We also learned that the E1a (calibration) is harmful in all cases. It's tempting to look at acc@3 and say that the model's underlying ability to answer the questions with E1b is relatively similar, but its consistency differs; however, with prompt variations, it solves different subsets of tasks. So while ScBench-Long acc@3 is similar for E1b and Pi baseline, they solve different problems.
Pooling acc@3 across all four Pi prompt variants covers +8.0 pp more TxBench-PP tasks and +14.3 pp more ScBench-Long tasks than the best single arm — different prompts crack different tasks.
Figure 2 — Coverage (acc@3) gain from pooling all 4 Pi prompt variants vs the best single arm (= the Pi baseline).
Based on this data, we can assert that an optimal harness should intelligently select behavioral priors that best fit the task. Future work could include having a model detect which priors fit the task, or applying parallel invocations with different priors and a judge to select the most likely trajectory.
Conclusion
Behavioral priors explain the ≈9-point swing between harnesses on identical tasks — Pi +4.6 pp on short-horizon TxBench-PP, Claude Code +4.8 pp on long-horizon ScBench-Long — not tools. Claude Code’s plan / verify/be conservative persona encourages additional thoughtfulness: it rescues the model on open-ended reasoning and sinks it on short, structured tasks. Pi’s near-empty prompt benefits it in areas where the model’s reasoning is sufficient to solve the task.
It is widely known that prompting can have small effects on performance, but here we show 2 lines of behavioral prompting can substantially impact a model’s ability on frontier biology tasks.
As we progress towards AGI, models will gain the ability to adopt optimal problem-solving behavior for a given task. Until then, a significant burden falls on harness engineers to ensure their models exhibit the correct behaviors for the task at hand.