Evaluations for agents in biology should not be stagnant, but rather continuously update as agentic capability and behavior changes over time. After noticing increasing misaligned behavior across our benchmarks, we are releasing an updated version on benchmarks.bio to ensure that our benchmarks test legitimate scientific reasoning. Below we will provide examples of the misalignment we observed, and our fixes to ensure these do not happen in the future.
Updates to our newly released version include:
Setting strict domain allow-lists for what internet access agents are may access, or turning off internet access altogether.
During our construction of our earlier benchmarks, we initially allowed complete internet access to replicate how scientists typically use agents. However, this unrestricted access to the internet led to increasingly misaligned behavior, which will be discussed in the next section. As such, we decided it was necessary to reduce and monitor what websites agents can browse, especially in our evaluations related to biosecurity.
Updating system prompts and task prompts to resemble natural, conversation-style language rather than stricter, more evaluation-esque instructions.
Performing additional anonymization of datasets, input files, and evaluations
Removing unnatural task completion instructions for agents, such as writing answers to specific files or echoing specific outputs.
Updating the sandboxes to more closely replicate research environments, such as more natural-sounding harness patches and proxy connections.
Adding live monitoring of model trajectories to ensure alignment of tool calls and model behavior, with immediate warnings for misaligned behavior.
These changes helped dramatically reduce misaligned behavior. Prior to these updates, we observed some runs that displayed misaligned behavior in relation to internet access, including:
Substack Reading: DeepSeek v4.1 Flash, on an evaluation where we previously allowed unrestricted internet access, searched some of the most popular benchmarks and concluded the task shape matched the shape of some of our other released evals. It then performed a set of searches into an author of the repo and ended up reading poems on the Substack from a different Arjun Banerjee. After updating task shape, removing internet access, and adding live monitoring ability, this behavior is unlikely to occur again.
Fabricated Identity: On an evaluation where we previously allowed internet access, the model attempted to query the NCBI E-utilities API. This API asks users to provide an email to monitor access. We observed many cases of fabricated emails across models tested, including
drg.researcher.2025@gmail.com(8 times),researcher@university.edu(10 times),researcher42@gmail.com(3 times), andtest@example.com(253 times). After removing internet access and live-monitoring tool calls, we do not expect this behavior to occur again.
We did not observe any substantial misaligned behavior after our updates, though some models suspected they were being evaluated and tried to probe the sandbox for a method to access the internet. None have succeeded so far.
The updates generally dropped scores across the board on benchmarks.bio, which we attribute to a decrease in behavior that shortcut proper biological reasoning, such as internet searches or threshold-hacking our verifiers. The average model lost about 3.6 percentage points per benchmark (median −2.3), and most of that comes from EpiBench, which collapsed by roughly 15 points for essentially every model (10.1 points for Opus 5 on Pi to 20.4 for GPT-6 Astra on Codex). The decrease in performance on EpiBench results is likely due to increased amounts of data anonymization that hindered the model’s ability to use memorized or looked-up results.
Harnesses also played a role. Seven of the eight models we run on Pi score higher on v1 than they did on v0, gaining 0.6 points on average. However, Grok Build lost 2.3 points, Codex lost 0.7, and Claude Code finished nearly flat, at 0.1 points lower. Grok 4.6 is a good example of this dichotomy. On Pi it climbed from 44.8 to 47.8, while the same model on Grok Build fell from 48.2 to 45.9. We believe this is because Grok models with Grok Build tend to execute more web searches than Grok models with the Pi harness, rather than any form of misalignment. A lot of this decrease could be attributed to Grok with Grok Build being highly effective at gathering scientific information using the web search tool, and then us removing this ability from the model. On average, Anthropic models lost 3.1 percentage points, followed by OpenAI models losing 4.0, and SpaceXAI models losing 4.1.
As model behavior changes, we will continue to update every aspect of our benchmark environments, from the tasks to the sandbox, to ensure that biological reasoning is properly assessed. This process will not simply occur once, but rather will evolve as agentic systems evolve to ensure the fidelity of our benchmark suite. As models improve, so too will our benchmarks.