300 million people are affected by 7,000 rare diseases worldwide. Due to clinical heterogeneity, uneven knowledge among clinicians, and a host of other factors, many rare disease patients end up ping-ponged around a long diagnostic odyssey of referrals, misdiagnoses, and unnecessary treatments that lasts, on average, 4-8 years. The long and winding nature of these diagnostic odysseys creates an enormous burden for patients and their families, even in well-resourced settings.
80% of rare diseases are genetic in origin, so the increasing availability of sequencing and other functional assays has begun to make headway against this problem. However, effectively using sequencing in clinical practice requires expertise to know how and what sequencing data can be obtained, what other data can be used with it to validate uncertain cases, and how all of the data can be analyzed and integrated with more common clinical findings. This expertise is uncommon and hard to find for patients, so we wanted to know whether agents could do some of this work.
We built RareDiseaseBench to find out. RareDiseaseBench contains 35 challenging cases spanning diseases of various body systems, variant types, genetic architectures, and inheritance modes. Agents are given a patient case and asked to make a diagnosis and management plan—but first, they must request the right data, including genome sequencing, RNA-seq, medical imaging, and more, from an isolated filesystem. We tested 31 model-harness configurations, with three attempts per question for a total of 3,255 trials. The highest preliminary pass rate belonged to Gemini 4 Argon at 43.8%.
Tl;dr
The best setup is Gemini 4 Argon on Pi, with 43.8% of runs right
Token and time efficiency was associated with stronger models and better performance, with weaker models routinely mismanaging their time and compute when confronted with 100 GB sequencing datasets.
Models failed most often to ask for the right data or search the genomes they obtained for variants in the right way.
Benchmark Design
Like many other Latch benchmarks, RareDiseaseBench is composed of tasks and datasets built from publicly available data. Unlike previous benchmarks, however, not all of the data is provided upfront: RareDiseaseBench requires an agent to interact with a Records system via a get_info tool. This tests the agent’s ability to not just analyze data correctly but brainstorm and know what it needs next.
Leaderboard and Efficiency
After Gemini 4 Argon / PI at 43.8%, the next strongest model-harness configurations were Opus 5.5 / Claude Code and GPT-6 Astra / PI at 33.3% (notably, GPT-6 Astra reached this performance for half the price). 29 of the 35 tasks were solved by at least one configuration, 26 by models from two or more labs, and 16 were passed in all three attempts by some configuration.
Claude models achieved higher or equivalent scores with Claude Code compared to PI, but GPT-series models performed better when using PI instead of Codex. Increased resource usage did not translate to improved performance, and one of the distinguishing factors of newer models was their ability to deliver better results with far fewer tokens (GPT-6 Astra / PI: 4.3M tokens for 33.3%; GPT-5.6 Sol / PI: 8.8M tokens for 13.3%).
Hard for One, Hard for All
Gemini 4 Argon led other models in all but one genetic architecture, variant type, affected organ system, and inheritance mode. Only 7 models exhibited a non-zero pass rate for non-genetic cases (marked acquired / infectious or no germline cause on the matrix). Opus 5.5, Sonnet 5.5, and Fable 5.1 all scored best on single homozygous variants and the three GPT-6 series models scored best on compound heterozygous variants, but not by large amounts.
Models Failed to Get Data
In the RareDiseaseBench system, models are responsible for requesting data from a separate service that uses data order rules and an LLM-as-a-judge to determine when data can be released. We found that failure to acquire the right data was a primary cause in 36% of failed runs and a primary or contributing cause in 72% of failed runs. The most common failure in data acquisition lay in not escalating to the data type the case needed (15% of failures), followed by not ordering non-sequencing investigations such as stains, imaging studies, or lab tests (12%). Failed runs typically involved missing family sequencing data used to determine inheritance modes or RNA or splicing studies used to conduct functional validation of variants.
Models Failed to Manage Large Data
Execution faults, mostly wall-clock timeouts against our 6-hour limit, were the primary cause in 13% of failures. 70% of Mistral Large 4.0 runs, 29% of Gemini 4 Argon runs and 21% of Claude Opus 5 runs timed out, against none of the 210 runs each of GPT-6 Astra, GPT-6.1 Sol and GPT-6.1 Terra. Compute or resource mismanagement contributed to 15% of failures. Execution faults were the primary cause of 25% of failures among models with no passing runs, and contributed to 46%. DeepSeek V4.1 Flash timed out in 84% of its runs, largely due to inefficient data processing. Interpretation errors contributed to 57% of failures, most often as anchoring on a wrong diagnosis or gene (38%).
Models Failed to Integrate Findings
In several case studies we detail in the manuscript, we find that the best performing models often fail to consider the possibility of multiple variants; integrate clinical symptoms with genetic findings; order functional studies that could be a decisive factor between multiple possible genetic diagnoses; and make decisive calls on the validity of previous workups.
Conclusion
While RareDiseaseBench has its limitations, we believe that its 35 tasks show that agents powered by frontier models do not yet have the ability to reliably request, analyze, and interpret data to diagnose challenging rare disease cases. Thus, we hope that RareDiseaseBench will serve as a barometer for patients, clinicians, and model makers while inspiring development of ever-more-realistic scientific and clinical benchmarks. We include the following resources here:
LatchBio builds benchmarks and products to make LLMs capable and secure at the life sciences. We are hiring aggressively across science, engineering, and ops roles. If work like RareDiseaseBench interests you, check out jobs.ashbyhq.com/LatchBio.