ViBench 2.0: Top Agent Finishes 65% of Apps
Replit released ViBench 2.0 on Oct. 28, testing whether AI agents can build whole products feature by feature. GPT-6.1 Sol fully passed 65% of builds, GPT-6 Astra 60% and Claude Opus 5.5 49%; no open-weight model passed a full build.
Replit on Wednesday, Oct. 28, released ViBench 2.0, a benchmark that asks AI coding agents to build whole software products from an empty project and then keep adding features to their own code. The best model, OpenAI's GPT-6.1 Sol, shipped a build that passed every test plan in 65% of attempts. No open-weight model passed a single full build.
Key takeaways
- ViBench 2.0 covers 17 products modeled on apps such as Uber, GitHub and Google Docs, with 241 feature requests and 544 browser test plans.
- GPT-6.1 Sol led with 65% of builds fully passing, ahead of GPT-6 Astra at 60% and Anthropic's Claude Opus 5.5 at 49%.
- The top three models each passed more than 90% of individual test plans, so the gap is about finishing a product without a single broken feature.
How the ViBench 2.0 benchmark works
Most coding benchmarks hand an agent a defined task inside an existing codebase. In ViBench 2.0, according to Replit's announcement, "the agent starts with an empty project instead." It builds a first version from a short spec, then adds up to 18 features on top of code it wrote itself.
That design punishes early shortcuts. An agent must pick database schemas and routes before it knows what later features will demand. Test plans check behavior step by step, such as whether split-fare shares in a ride app add up exactly to the ride cost, or whether a refund can be issued twice.
A grading agent, Claude Opus 5.5, runs every test plan in a real browser. Replit said it planted 34 bugs one at a time in working builds, and the grader "never passed a plan while the bug was visible." GPT-6.1 Sol re-graded 477 of Opus's passes and agreed with 99.2% of them.
ViBench 2.0 results: GPT-6.1 Sol, GPT-6 Astra, Claude Opus 5.5
The official runs were carried out by Fireworks AI, four builds per product per model, at medium effort. The headline metric, pass@1, counts a build only if every test plan passes.
| Model | Builds fully passing | Cost per build |
|---|---|---|
| GPT-6.1 Sol | 65% | about $12 |
| GPT-6 Astra | 60% | about $90 |
| Claude Opus 5.5 | 49% | not stated |
Speed differed sharply. Opus 5.5 took about 70 minutes per build, while Sol was roughly 2.5 times slower. Replit said Opus produced about 6% fewer tokens than Sol, so most of the gap came from API provider speed. Fireworks' Specialized Intelligence Index lists Sol at 64.7% pass@1, $12.34 and 10,680 seconds per task.
Among cheaper options, DeepSeek V4.1 Flash passed 77% of test plans for about $2.50 per build, but no open-weight model completed a full build cleanly.
Where AI coding agents fail
Replit found that nearly three-quarters of failed builds from top models broke only where features intersect. About 1 in 40 top-model builds never started on an empty database because later schema code referenced a table before creating it.
Design was not graded. Claude models tended toward plainer screens and GPT models added illustrations and large headlines, but the best-looking builds were no more likely to pass.
Context: from ViBench to ViBench 2.0
The original ViBench, built by Replit and Georgian's AI Lab, tested apps such as whiteboards and resume builders. At publication, Claude Opus 4.6 led with 45.7% of tasks passed, and seven of nine models did worse extending their own code than a clean reference, Georgian said. Version 2.0 stretches that "vibe-on-vibe" setting into long product evolution, a gap that matters as models like Claude Opus 5.5 take on longer agentic jobs. Earlier research showed how LLM bug-fixers can break working code.
What's next
Eight products are public on GitHub in the Harbor format and can be rerun with one command from release v2.0.6; the other nine are held out. Replit said a detailed methodology write-up is "coming soon." All results so far are run or reported by Replit and its partners, not independently reproduced.
FAQ
What is ViBench 2.0?
A Replit benchmark in which AI agents build 17 full products from scratch and add up to 18 features each, graded by 544 browser test plans.
Which AI model scored highest on ViBench 2.0?
OpenAI's GPT-6.1 Sol, with 65% of builds passing every test plan, at about $12 per build.
Can I run ViBench 2.0 myself?
Yes, for the eight public products, using the OpenHands-based harness in release v2.0.6. The other nine products are not released.
References
- ViBench 2.0: A Benchmark for Long-Horizon Software Building — Replit (Oct. 28, 2026)
- Specialized Intelligence Index — Fireworks AI (updated Oct. 28, 2026)
- Vibe Check: Why Georgian and Replit Created a Benchmark on Vibe Coding — Georgian (June 25, 2026)
Editor responsible for AI Newsway. Writes about AI, developer tools, cloud and productivity technology.
Comments 0
Comments are currently disabled.