Top AI Coder Fully Solves 26.4% of Real Edits

The best AI coding agent in a new Alibaba-backed benchmark fully completed only 26.4% of real users' web app change requests, even though it met about 70% of the individual requirements on average. The results come from Co-Design Bench, a paper posted to arXiv on Oct. 26 by researchers from Alibaba Group, Fudan University and four other universities, and highlighted by The Decoder on Nov. 8.

Key takeaways

• GPT-6 Astra led all ten model endpoints, passing 69.9% of criteria per task but fully satisfying just 26.4% of tasks.
• Every model scored lower on repository edits that mixed three or four types of change than on edits with one or two.
• The 273 tasks were drawn from 37,463 deduplicated queries in 26,314 real user coding sessions.

What Co-Design Bench tests

Most coding benchmarks ask an agent to fix a known bug or rebuild a page from a screenshot. Co-Design Bench instead hands agents a runnable web app and a real user's revision request, sometimes with an image, audio clip, video or document attached.

The benchmark has two tracks. Multimodal Editing covers 100 localized edits, such as pulling a value from a receipt and updating every total that depends on it. Repository Editing covers 173 multi-file changes across components, routes, state and data flow. Of those 173 tasks, 166 require two or more kinds of edit.

The authors favored requests that had already tripped up agents in practice, ranking candidates by signals such as users rephrasing a request or rolling back a change.

How the AI coding agents were graded

Experts wrote 2,331 hidden acceptance criteria. An AI judge first grades each criterion using source code and browser evidence. A separate LLM verifier then challenges every Pass or Partial verdict, looking for stubs, hardcoded outputs and features users cannot actually reach. Two frontend experts also checked rubric scores.

All models ran inside the same Claude Code harness, with each run capped at three hours.

Coding agent benchmark results by model

GPT-6 Astra ranked first overall, followed by Claude Opus 5 and Qwen3.8-Max. On the multimodal track it fully passed 41.0% of tasks, against 31.0% for Qwen3.8-Max, 30.0% for Opus 5 and 6.0% for Grok 4.6. On the repository track its full-pass rate fell to 17.9%.

Rankings shifted between tracks. The much smaller Qwen3.8-27B placed second on repository edits and fourth overall, though the authors said that narrow gap "does not establish a general model advantage."

More effort did not guarantee better results. On the multimodal track, GLM-5.2 took a median of 69 turns per task, more than double GPT-6 Astra's 32, and still scored lower.

Context: benchmarks near saturation

The authors built the test because older web-editing benchmarks are close to saturated once agents can run and fix their own code. The gap they describe is practical: an agent that gets most of a request right but misses one detail still leaves cleanup for a human.

The findings echo earlier work on this site showing that LLM bug-fixers broke working code more often than benchmark scores suggested.

What's next

The authors note limits. Each task uses one fixed request, so the benchmark does not simulate follow-up feedback from users, and the AI-based judging "cannot eliminate judgment error." They argue that leaderboards should report both requirement coverage and complete task success.

FAQ

What is Co-Design Bench?
A benchmark of 273 web app revision tasks taken from real user coding sessions, graded against 2,331 hidden expert-written criteria.

Which AI coding agent did best?
GPT-6 Astra, which met 69.9% of criteria per task on average but fully completed only 26.4% of tasks.

Why do scores drop on real edits?
Real requests often combine layout, style, interaction and content changes. All ten models did worse when a repository task mixed three or four of these.

References

• Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models — arXiv, Xilin Wei et al. (Oct. 26, 2026)
• Co-Design Bench — Multimodal Collaborative Coding Evaluation for Agentic Models — project page
• AI coding agents still need human cleanup on nearly three out of four tasks in a real-world test — The Decoder (Nov. 8, 2026)