We introduce OncologyBench, a verifiable benchmark of 25 multistep oncology evaluations that tests how multi-agent systems analyze evidence and make clinical and research decisions.
In oncology, one decision usually rests on several analyses. Whether a patient should have chemotherapy after surgery can depend on the tumor’s mutations, gene expression, pathology, and immune composition, as well as the patient’s medical record. Different specialists assess these sources of evidence, and the clinician making the decision works from their reports.
OncologyBench mirrors this workflow: several agents each analyze one piece of evidence, and a final agent weighs their findings and makes the decision.
We evaluated 31 model-harness combinations on 25 decisions built on 92 analytical tasks. Each configuration attempted every evaluation three times, yielding 2,325 attempts. The strongest configuration, Claude Opus 5 on Pi, passed 40.0% of attempts (95% CI: 24.0% to 57.3%). An attempt passes only when every Task and the final Decision are correct. The evaluations draw on publicly available primary literature and datasets, plus one patient case donated specifically for benchmarking.
These 25 evaluations sample clinical and research workflows. Benchmark performance is distinct from clinical validation.
At a glance
The strongest configuration, Claude Opus 5 on Pi, passed 40.0% of attempts, and the top six were all Claude configurations.
Analysis did not guarantee a correct decision: 57% of attempts passed every Task; 19% passed every Task and the final Decision.
In one pediatric case, who asked changed response style and, for Grok 4.7, the analytical conclusions.
Models are better at analysis than reasoning
Correct analysis did not guarantee a correct final decision. Across all 2,325 attempts, 57% passed every Task; 19% passed every Task and the final Decision.
Each OncologyBench evaluation separates analysis from synthesis. Several Task agents work in parallel, each analyzing one type of data. A final Decision agent reads their prompts, answers, generated artifacts, and execution records, but never the raw data. This resembles a clinician weighing specialist reports. Unlike the long-horizon evaluations in SpatialBench-Long and scBench-Long, these Decisions sit downstream of several parallel Tasks.
This structure helps distinguish analytical failures from failures at the final decision stage. Across configurations, the share of attempts that passed every Task exceeded the share that passed the Decision. Grok 4.7 on Pi had the largest gap: 69% of attempts passed every Task, while 27% passed the Decision. Opus 5.5 had the smallest gap: 4 percentage points on Pi and 8 percentage points on Claude Code.
57% of attempts got every Task right
19% passed every Task and the final Decision
2 in 3 attempts with every Task right still failed the Decision
Cost and performance varied across configurations
The relationship between cost and performance depended on the model and harness. Claude configurations occupied the top six places. Across all Claude configurations, the mean reported cost was $8.27 per evaluation attempt. Kimi K3 on Pi passed 24.0% of attempts at $3.33 per attempt, placing it on the cost frontier. These costs reflect the full agent workflow, and the comparison is specific to the models, harnesses, and workloads tested.
Who asks changes the answer
In a separate experiment on one donated pediatric case, we asked the same question in six voices: a mother, a father, the patient, a patient advocate, the treating oncologist, and the reporting pathologist. We compared Opus 5.5, Sonnet 5.5, Grok 4.7, and GPT-6 Astra.
The child’s tumor and saliva had both been sequenced. One of the four Tasks asked where a change in TP53, a gene that normally holds tumor growth in check, had come from. The sequencing vendor’s report listed TP53 as unchanged, but comparing the tumor’s DNA with the saliva’s shows that the change arose in the tumor. The right advice depended on recognizing that.
In this case, who asked affected response style and, for Grok 4.7, the analysis. Opus 5.5 and Sonnet 5.5 adapted their language to each reader, from plain words for the child to precise evidence for the pathologist. GPT-6 Astra used a similar register for families and experts. Grok 4.7 often missed the TP53 change: it recognized the gene fusion but judged TP53 by point mutations alone. It reached the correct TP53 answer more often for expert askers than for family askers (16% versus 6%), and relied on the vendor’s report more often when a family member asked (43% of attempts versus 19%).
The persona findings come from a single pediatric case. Testing additional cases would establish how broadly these patterns hold.
