In current generation models, behavioral priors introduce by harnesses such as Claude Code, and Codex cause substantial performance swings on our benchmarks.
This is a strong example of why harnesses matter. The same model can behave very differently when the surrounding defaults, tools and loops change. "Which model won?" is less useful than "which system produced a run we can trust?"
Wow, this is great! It makes sense that the system prompt is a big driver but it’s mostly out of sight out of mind in the harnesses, I don’t really know what’s in it. So then we should really be defining some building blocks to compose custom personas and select the right ones for a task. Then it could be extended to dynamically select the right persona too but so far dynamic routing has been difficult.
Agreed, an optimal harness should select the correct priors for the task. However, I also agree that this is very difficult in practice. It's also difficult that models improve and drift across generations, so harness designers would have to analyze each model version and make appropriate updates.
As post-training improves, models will know how to approach specific types of problems and won't need behavioral instructions in the system prompt.
This is a strong example of why harnesses matter. The same model can behave very differently when the surrounding defaults, tools and loops change. "Which model won?" is less useful than "which system produced a run we can trust?"
Wow, this is great! It makes sense that the system prompt is a big driver but it’s mostly out of sight out of mind in the harnesses, I don’t really know what’s in it. So then we should really be defining some building blocks to compose custom personas and select the right ones for a task. Then it could be extended to dynamically select the right persona too but so far dynamic routing has been difficult.
Agreed, an optimal harness should select the correct priors for the task. However, I also agree that this is very difficult in practice. It's also difficult that models improve and drift across generations, so harness designers would have to analyze each model version and make appropriate updates.
As post-training improves, models will know how to approach specific types of problems and won't need behavioral instructions in the system prompt.
Do you guys do custom RL post-training? I guess post-training is the next iteration on prompt engineering now, need to try that!