A consumer product with more than 200,000 users and a fintech platform that has processed over 100,000 bank statements have done something unusual with their private repositories: licensed them to an evaluation company. Specific Labs, a Y Combinator F25 company, says it has acquired and licensed operational data and codebases from real companies over the past year, and assembled them into Real-SWE. In doing so it turned proprietary code into a commodity the AI industry has started paying for.
Key takeaways
- Specific Labs licensed production repositories from operating businesses rather than scraping open-source projects, creating evaluation material no model could have read during pretraining.
- Every major coding benchmark is built on public repositories models have already trained on, which is the supply gap the licensing market exists to fill.
- Verifiers are injected only at grading time and are drawn from each codebase's existing test suites, so the licensing deal has to cover the tests as well as the code.
The economics are straightforward once stated. Public evaluation material has a shelf life, because anything published eventually lands in a training corpus and stops distinguishing recall from reasoning. Private code has no such decay — but it has an owner, and acquiring it means negotiating rather than crawling.
What a licensed codebase has to include
Access to source alone is not enough to build an evaluation from. Specific Labs runs each agent in an isolated sandbox, stores tasks in Harbor format, and injects verifiers at grading time that are either inspired by or taken verbatim from tests already in the codebase. A usable license therefore covers the repository, its test suite, and the issue history that makes a task realistic.
The tasks that result look nothing like a curated puzzle. One routes each invoice through a business's own tax mode, including destination pricing through a tax authority provider. Another moves ownership of offerings and usage from services to customers without breaking existing reads. A third discovers every enabled AWS region and sweeps block-storage inventory without one failing region collapsing the rest.
Every company has its own rules and ways of writing code. Agents have to understand those conventions and make changes that work with what is already there.
Scope follows from that. Real-SWE solutions touch a median of 11 files, close to double the six-file median of public suites like FrontierCode and DeepSWE. Changes that alter how a business runs rarely stay inside one service.
What the licensed data revealed
The first results give licensors something to point at. Claude Fable 5.1 running inside Claude Code led at 38.8%, with GPT-6 Astra on Codex CLI at 33.8% and Gemini 3.8 Flash on Gemini CLI at 31.2%. Resolution rate is pass@1 averaged over eight independent runs per task, reported with 95% confidence intervals.
Crucially for anyone considering a similar deal, the scores are tied to a harness, not just a model. Real-SWE pairs each system with its native tool because that reflects how enterprise engineers work, which means the licensed repository is measuring a whole workflow — model, scaffold and IDE conventions together. The same weights driven through a different scaffold could land somewhere else entirely.
Why this market is likely to grow
Buyers of an AI coding assistant have little way to tell whether a leaderboard number survives contact with their own systems, and vendors have little incentive to build the test that would tell them. An independent party paying for private code sits in between, and the more crowded public benchmarks become, the more that position is worth. Anthropic, OpenAI and Google all have models on this leaderboard, and none of them controls the data behind it.
The caveats are real. Ten tasks is a small sample, and the exercise does not by itself prove that public benchmarks are inflated by contamination. Licensing also concentrates evaluation power in whoever can afford the deals, and a benchmark whose contents nobody outside can inspect asks for a different kind of trust than an open dataset does — a tension visible in other recent agent benchmarks that published methodology without results.
FAQ
Why would a company license its private source code?
Private code is valuable to evaluators precisely because it has never been published, so it cannot have leaked into model training data. That scarcity is what an evaluation company is paying for, and the licensor gets paid without open-sourcing anything.
Does Real-SWE publish the code it licensed?
No. The code and its reference solutions are not publicly available, which is the entire premise of the benchmark. Specific Labs publishes task descriptions and scores rather than the underlying repositories.
Which model scored highest on Real-SWE?
Claude Fable 5.1 running through Claude Code led at 38.8%, ahead of GPT-6 Astra on Codex CLI at 33.8% and Gemini 3.8 Flash on Gemini CLI at 31.2%. Every system tested failed the majority of tasks.
