1d:I[70616,["/_next/static/chunks/0kcf5dt6yxdox.js","/_next/static/chunks/1su5n1uv1ti4s.js","/_next/static/chunks/0gvke1wv6udfg.js","/_next/static/chunks/1dfrle3jgmlzh.js","/_next/static/chunks/162lgvduyr-fa.js","/_next/static/chunks/2gcnnopam8e9w.js","/_next/static/chunks/2g60iab5a1pkg.js"],"Breadcrumb"] 1e:I[90762,["/_next/static/chunks/0kcf5dt6yxdox.js","/_next/static/chunks/1su5n1uv1ti4s.js","/_next/static/chunks/0gvke1wv6udfg.js","/_next/static/chunks/1dfrle3jgmlzh.js","/_next/static/chunks/162lgvduyr-fa.js","/_next/static/chunks/2gcnnopam8e9w.js","/_next/static/chunks/2g60iab5a1pkg.js"],"ArticleGrid"] 1f:T1665,

A consumer product with more than 200,000 users and a fintech platform that has processed over 100,000 bank statements have done something unusual with their private repositories: licensed them to an evaluation company. Specific Labs, a Y Combinator F25 company, says it has acquired and licensed operational data and codebases from real companies over the past year, and assembled them into Real-SWE. In doing so it turned proprietary code into a commodity the AI industry has started paying for.

Key takeaways

Specific Labs licensed production repositories from operating businesses rather than scraping open-source projects, creating evaluation material no model could have read during pretraining.
Every major coding benchmark is built on public repositories models have already trained on, which is the supply gap the licensing market exists to fill.
Verifiers are injected only at grading time and are drawn from each codebase's existing test suites, so the licensing deal has to cover the tests as well as the code.

The economics are straightforward once stated. Public evaluation material has a shelf life, because anything published eventually lands in a training corpus and stops distinguishing recall from reasoning. Private code has no such decay — but it has an owner, and acquiring it means negotiating rather than crawling.

What a licensed codebase has to include

Access to source alone is not enough to build an evaluation from. Specific Labs runs each agent in an isolated sandbox, stores tasks in Harbor format, and injects verifiers at grading time that are either inspired by or taken verbatim from tests already in the codebase. A usable license therefore covers the repository, its test suite, and the issue history that makes a task realistic.

The tasks that result look nothing like a curated puzzle. One routes each invoice through a business's own tax mode, including destination pricing through a tax authority provider. Another moves ownership of offerings and usage from services to customers without breaking existing reads. A third discovers every enabled AWS region and sweeps block-storage inventory without one failing region collapsing the rest.

Every company has its own rules and ways of writing code. Agents have to understand those conventions and make changes that work with what is already there.

Scope follows from that. Real-SWE solutions touch a median of 11 files, close to double the six-file median of public suites like FrontierCode and DeepSWE. Changes that alter how a business runs rarely stay inside one service.

What the licensed data revealed

The first results give licensors something to point at. Claude Fable 5.1 running inside Claude Code led at 38.8%, with GPT-6 Astra on Codex CLI at 33.8% and Gemini 3.8 Flash on Gemini CLI at 31.2%. Resolution rate is pass@1 averaged over eight independent runs per task, reported with 95% confidence intervals.

Crucially for anyone considering a similar deal, the scores are tied to a harness, not just a model. Real-SWE pairs each system with its native tool because that reflects how enterprise engineers work, which means the licensed repository is measuring a whole workflow — model, scaffold and IDE conventions together. The same weights driven through a different scaffold could land somewhere else entirely.

Why this market is likely to grow

Buyers of an AI coding assistant have little way to tell whether a leaderboard number survives contact with their own systems, and vendors have little incentive to build the test that would tell them. An independent party paying for private code sits in between, and the more crowded public benchmarks become, the more that position is worth. Anthropic, OpenAI and Google all have models on this leaderboard, and none of them controls the data behind it.

The caveats are real. Ten tasks is a small sample, and the exercise does not by itself prove that public benchmarks are inflated by contamination. Licensing also concentrates evaluation power in whoever can afford the deals, and a benchmark whose contents nobody outside can inspect asks for a different kind of trust than an open dataset does — a tension visible in other recent agent benchmarks that published methodology without results.

FAQ

Why would a company license its private source code?

Private code is valuable to evaluators precisely because it has never been published, so it cannot have leaked into model training data. That scarcity is what an evaluation company is paying for, and the licensor gets paid without open-sourcing anything.

Does Real-SWE publish the code it licensed?

No. The code and its reference solutions are not publicly available, which is the entire premise of the benchmark. Specific Labs publishes task descriptions and scores rather than the underlying repositories.

Which model scored highest on Real-SWE?

Claude Fable 5.1 running through Claude Code led at 38.8%, ahead of GPT-6 Astra on Codex CLI at 33.8% and Gemini 3.8 Flash on Gemini CLI at 31.2%. Every system tested failed the majority of tasks.

a:["$","div",null,{"className":"mx-auto max-w-7xl px-4 py-8 sm:px-6","children":[["$","script",null,{"type":"application/ld+json","dangerouslySetInnerHTML":{"__html":"{\"@context\":\"https://schema.org\",\"@type\":\"BreadcrumbList\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https://ainewsway.com\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"#benchmark\",\"item\":\"https://ainewsway.com/tag/benchmark\"}]}"}}],["$","script",null,{"type":"application/ld+json","dangerouslySetInnerHTML":{"__html":"{\"@context\":\"https://schema.org\",\"@type\":\"ItemList\",\"@id\":\"https://ainewsway.com/tag/benchmark#itemlist\",\"name\":\"#benchmark Articles\",\"numberOfItems\":1,\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"url\":\"https://ainewsway.com/articles/companies-are-now-licensing-their-private-source-code-as-ai-evaluation-data-mu27wxbw\",\"name\":\"Companies Are Now Licensing Their Private Source Code as AI Evaluation Data\"}]}"}}],["$","$L1d",null,{"items":[{"label":"# benchmark"}]}],["$","h1",null,{"className":"mb-8 text-3xl font-bold text-zinc-900 dark:text-zinc-100","children":["#","benchmark"]}],[["$","$L1e",null,{"articles":[{"id":"51bdc760-56dd-4d02-ae9f-5a537a4c7e92","slug":"companies-are-now-licensing-their-private-source-code-as-ai-evaluation-data-mu27wxbw","title":"Companies Are Now Licensing Their Private Source Code as AI Evaluation Data","subtitle":"Specific Labs paid a consumer product and a fintech platform for repository access, creating an eval asset that cannot be scraped","content":"$1f","excerpt":"A consumer product with more than 200,000 users and a fintech platform that has processed over 100,000 bank statements have done something unusual with their pr...","coverImage":"https://cdn.ainewsway.com/uploads/2026/09/cf92a6fd-ce44-4fbd-bf74-e49441abc078.webp","coverImageCaption":"Production source code of the kind Specific Labs now licenses from operating businesses to evaluate AI coding agents","authorId":"6773f67d-d7fa-4b19-8f38-384f10ac761b","status":"PUBLISHED","publishedAt":"2026-09-15T05:15:32.369Z","viewCount":0,"likeCount":0,"readingTime":5,"isFeatured":false,"isBreakingNews":false,"metaTitle":"Companies Are Licensing Private Code as AI Eval Data","metaDescription":"Specific Labs paid a fintech platform and a consumer product for repository access to build Real-SWE, an AI coding benchmark no model could have memorized.","metaKeywords":["Real-SWE","Specific Labs","AI benchmark licensing","data contamination","AI coding agents","enterprise code","Claude Code","developer tools"],"ogImage":null,"aiSummary":"Specific Labs licensed private production repositories from a consumer product with 200,000 users and a fintech platform to build Real-SWE, an evaluation asset no model could have read during pretraining. The licensing has to cover test suites too, since verifiers are injected at grading time. Claude Fable 5.1 led the first results at 38.8%, well below the scores agents post on public benchmarks.","aiTags":[],"sentiment":null,"articleType":"NEWS","videoEmbed":null,"videoType":null,"seoScore":85,"isAutoGenerated":true,"isBackfill":false,"pipelineStage":null,"copyrightScore":null,"similarityScore":null,"matchedSourceId":null,"createdAt":"2026-09-15T05:15:06.046Z","updatedAt":"2026-09-15T10:07:58.827Z","contentUpdatedAt":"2026-09-15T10:07:58.825Z","deletedAt":null,"author":{"id":"6773f67d-d7fa-4b19-8f38-384f10ac761b","name":"Seung Jung","username":"Seung Jung","avatar":null},"categories":[{"id":"32f2303b-b593-473f-bf9c-6a99acbe6ce1","name":"Developer Tools","slug":"developer-tools"}],"tags":[{"id":"eeeafe40-7aaa-4305-8e8d-aa0adde8da6d","name":"Real-SWE","slug":"real-swe"},{"id":"98246980-4d05-4ceb-ae83-091033f73aa8","name":"AI Coding Agents","slug":"ai-coding-agents"},{"id":"b4959a7c-29fe-47b5-a58d-c6f2b9fc343e","name":"Benchmark","slug":"benchmark"},{"id":"3ee386e7-a0f5-4bf3-9cc9-8f8dbbf62172","name":"Data Licensing","slug":"data-licensing"},{"id":"efe1c7b1-1e8a-4582-85ff-1b17323a22a7","name":"Developer Tools","slug":"developer-tools"}],"translations":[{"id":"f79e4c9e-c818-41cf-8d8c-c45b55a9f413","language":"ko","title":"기업들이 자사 비공개 소스코드를 AI 평가 데이터로 팔기 시작했다","subtitle":"Specific Labs가 소비자 제품 기업과 핀테크 플랫폼에 대가를 치르고 저장소 접근권을 확보했다. 크롤링으로는 만들 수 없는 평가 자산이다","excerpt":"Specific Labs가 소비자 제품 기업과 핀테크 플랫폼의 비공개 저장소를 라이선스해 Real-SWE를 만들었다. 독점 코드가 긁어모을 수 없는 AI 평가 데이터로 바뀌었다.","metaTitle":"비공개 소스코드를 AI 평가 데이터로 파는 기업들","metaDescription":"Specific Labs가 핀테크 플랫폼과 소비자 제품 기업에 대가를 치르고 저장소 접근권을 확보해 Real-SWE를 만들었다. 어떤 모델도 암기했을 리 없는 AI 코딩 벤치마크다."},{"id":"03d91678-8b13-4c9a-b6cf-ef9cf77f3d38","language":"ja","title":"企業が自社の非公開ソースコードをAI評価データとして販売し始めた","subtitle":"Specific Labsは消費者向けプロダクト企業とフィンテック基盤に対価を支払い、リポジトリへのアクセス権を取得した。スクレイピングでは作れない評価資産だ","excerpt":"Specific Labsが消費者向けプロダクト企業とフィンテック基盤の非公開リポジトリをライセンスし、Real-SWEを構築した。独自コードが収集不可能なAI評価データに変わった。","metaTitle":"非公開ソースコードをAI評価データとして売る企業","metaDescription":"Specific Labsがフィンテック基盤と消費者向けプロダクト企業に対価を払ってリポジトリへのアクセス権を得て、Real-SWEを構築した。どのモデルも記憶しようがないAIコーディングベンチマークだ。"},{"id":"30e279cc-3863-4614-82cf-27017780077b","language":"zh","title":"企业开始把自家非公开源代码当作AI评测数据出售","subtitle":"Specific Labs向一家消费产品公司和一家金融科技平台付费换取代码库访问权，做出了一份无法靠爬取获得的评测资产","excerpt":"Specific Labs从一家消费产品公司和一家金融科技平台手中获得非公开代码库授权，搭建出Real-SWE，把专有代码变成了爬不到的AI评测数据。","metaTitle":"企业把非公开源代码卖作AI评测数据","metaDescription":"Specific Labs向金融科技平台和消费产品公司付费获取代码库访问权，搭建出Real-SWE——一个任何模型都不可能背下来的AI编程基准。"},{"id":"6a2042cc-a56d-46b7-a175-9d93cca86a7b","language":"es","title":"Las empresas ya licencian su código fuente privado como datos de evaluación de IA","subtitle":"Specific Labs pagó a una empresa de producto de consumo y a una plataforma fintech por el acceso a sus repositorios y creó un activo de evaluación que nadie puede rastrear","excerpt":"Specific Labs licenció repositorios privados de una empresa de producto de consumo y una plataforma fintech para construir Real-SWE, convirtiendo código propietario en datos de evaluación imposibles de rastrear.","metaTitle":"Licencian código privado como datos de evaluación de IA","metaDescription":"Specific Labs pagó a una plataforma fintech y a una empresa de producto de consumo por acceder a sus repositorios y construyó Real-SWE, un benchmark que ningún modelo pudo memorizar."}],"celebrities":[{"id":"0632936d-a299-490f-b5e5-dfa87ba3eaa0","nameKo":"Anthropic","nameEn":"Anthropic","slug":"anthropic","profileImage":"https://ui-avatars.com/api/?name=Anthropic&size=400&background=84CC16&color=fff&bold=true"}]}]}],"$L20"]]}]

#benchmark

기업들이 자사 비공개 소스코드를 AI 평가 데이터로 팔기 시작했다

뉴스레터 구독

#benchmark

기업들이 자사 비공개 소스코드를 AI 평가 데이터로 팔기 시작했다