Study: Agent Harnesses Shift Cost, Not Scores

Swapping the harness around a coding agent barely moves its benchmark score but can triple the bill, according to a study from two Shandong University researchers that VentureBeat reported on Oct. 23. The paper, posted to arXiv on Oct. 3, ran five models through Claude Code, OpenCode and mini-SWE-agent on SWE-bench Verified.

Key takeaways

• On a 447-task pool, the heaviest harness (Claude Code) and the lightest (mini-SWE-agent) landed within five points of each other on both models tested.
• With the same model, tasks and price list, cost per task differed by up to 3x across harnesses.
• The main cost driver was the fixed preamble sent on every step: 16,581 tokens for Claude Code, 7,025 for OpenCode and 829 for mini-SWE-agent.

How the coding agent harness study was run

A harness is the layer around a model that supplies the system prompt, tools and context management. Authors Yangze Liu and Zhongyi Han held the model fixed and changed only that layer, testing Qwen3.6-35B-A3B, Qwen3.8-27B, DeepSeek-V4-Flash, GLM-5.3-Flash and HY4-Preview. Both Qwen models covered a 447-task pool; all five ran a 45-task hard subset.

Early web-enabled runs showed 71% of DeepSeek and Qwen trials fetching the upstream fix from GitHub, so network access was blocked for scored runs.

Claude Code vs. mini-SWE-agent: scores

On Qwen3.6-35B-A3B, Claude Code solved 310 of 445 scored tasks and mini-SWE-agent solved 318, a paired gap of 1.8 points. On Qwen3.8-27B, the counts were 347 and 353 of 444, a 1.4-point gap.

On the hard subset, changing the harness flipped the result on about 13% of tasks. That matched the rate seen when the same harness was simply rerun. The only effect that clearly beat the noise was a deficit: OpenCode trailed by up to 9 points on the large pool, and for one model about half that gap came from runs cut off by the 32,000-token output limit.

Vendor harnesses fared no better. DeepSeek's own CLI passed 21 of 45 hard tasks on DeepSeek-V4-Flash, the same as Claude Code and one fewer than mini-SWE-agent. "A vendor's own harness on its own model buys no score advantage," the authors wrote.

Why AI coding agent costs differ by harness

The bill tracks the harness's own text, multiplied by the number of steps. On GLM-5.3-Flash, a trial cost 1.81 yuan under Claude Code, 0.97 under mini-SWE-agent and 0.56 under OpenCode.

Step count matters too: on DeepSeek-V4-Flash, mini-SWE-agent took 141 steps to Claude Code's 104. Cached-input pricing, which covered 96% to 99% of input tokens, scaled the bill without changing the ranking.

Context: a second study points the same way

A separate project, HarnessTax, tested 21 model-harness pairs across seven models and three harnesses: Claude Code, OpenAI's Codex CLI and the minimal Pi, on SWE-bench Lite and Terminal-Bench 2.0. VentureBeat attributed it to UC Berkeley's Sky Computing Lab and Arena and reported cost gaps of up to fivefold with pass rates moving only a few points.

The findings also add to a running debate on this site over how much credit harness design deserves, from LLM bug-fixers that broke working code to large refactoring runs.

What's next for agent benchmarks

The authors warn that small benchmarks cannot settle many claims. With 45 tasks, a 13-point gap was detected only about half the time. With 447 tasks, gaps of about 5 points could be resolved, which is still coarser than many advertised harness gains.

For teams choosing tools, the upshot is to compare first-call size, per-step growth and step count before paying for a heavier harness.

FAQ

Does Claude Code make models score higher on SWE-bench?

Not in this study. On two Qwen models over 447 tasks, Claude Code and mini-SWE-agent were within five points, and mini-SWE-agent scored slightly higher on both.

What makes one coding agent harness more expensive?

Mostly the fixed system prompt and tool schemas resent on every step. Claude Code's preamble was about 20 times larger than mini-SWE-agent's, though step counts also affect the total.

References

• What Does a Harness Buy? Tokens, Mostly — arXiv, Yangze Liu and Zhongyi Han (Oct. 3, 2026)
• HarnessTax: How Much Does the Harness Matter for Coding Agents? — HarnessTax project page
• A coding agent's harness can triple its cost without making it more accurate, study finds — VentureBeat (Oct. 23, 2026)