1d:I[70616,["/_next/static/chunks/0kcf5dt6yxdox.js","/_next/static/chunks/1su5n1uv1ti4s.js","/_next/static/chunks/0gvke1wv6udfg.js","/_next/static/chunks/1dfrle3jgmlzh.js","/_next/static/chunks/162lgvduyr-fa.js","/_next/static/chunks/2gcnnopam8e9w.js","/_next/static/chunks/2g60iab5a1pkg.js"],"Breadcrumb"] 1e:I[90762,["/_next/static/chunks/0kcf5dt6yxdox.js","/_next/static/chunks/1su5n1uv1ti4s.js","/_next/static/chunks/0gvke1wv6udfg.js","/_next/static/chunks/1dfrle3jgmlzh.js","/_next/static/chunks/162lgvduyr-fa.js","/_next/static/chunks/2gcnnopam8e9w.js","/_next/static/chunks/2g60iab5a1pkg.js"],"ArticleGrid"] 1f:T144c,

A team of 51 physicists and computer scientists re-examined six widely used physics benchmarks and found that the low scores frontier models post on them mostly measure the benchmarks, not the models. After experts corrected faulty reference solutions and repaired or removed flawed questions, GPT-5.6-Sol's mean@4 score on HLE-Physics rose from 47.3% to 78.7%.

Key takeaways

On HLE-Physics, GPT-5.6-Sol's mean@4 climbed from 47.3% to 78.7% after expert re-grading; on CMT-Benchmark it went from 61.0% to 87.2%.
Corrected pass@4 reached 94.4% on the 54 CritPt challenges that survived expert review, a level the authors describe as near-saturation.
The paper directly disputes the impression created by low physics scores in the Artificial Analysis Intelligence Index (2026), arguing most audited errors were grading artifacts rather than reasoning failures.

What the auditors actually did

The study, published as arXiv:2609.13009, started from a mismatch its authors kept noticing. Published benchmark results said frontier models still struggle with advanced physics. Physicists who use those same models in their research reported a different experience.

Rather than argue from anecdote, the team ran frontier models across six benchmarks and put the results in front of domain experts. Faculty and graduate researchers with relevant subfield expertise read each problem statement, each reference solution and each model response, restricting the scope to text-only problems with verifiable final answers.

Their job was to sort failures into categories: genuine model errors, grader errors, incorrect reference solutions, and questions too ambiguous or underspecified to have a defensible answer. The headline finding is that most audited cases initially scored as incorrect fell into the latter three buckets.

Why the corrected scores moved so far

Once the experts fixed erroneous reference solutions and repaired or excluded broken questions, scores rose sharply across the board. The two largest documented jumps belong to GPT-5.6-Sol: HLE-Physics from 47.3% to 78.7% and CMT-Benchmark from 61.0% to 87.2%, both measured as mean@4.

On CritPt, corrected pass@4 hit 94.4% across the 54 problems retained after review. The authors report that audited subsets of UGPhysics, PRISM-Physics and PHYBench also rose substantially. Crucially, corrected scores are computed only on the retained subsets — the numbers describe performance on problems that survived expert scrutiny, not on the original full sets.

The measurement problem this exposes

The result lands on a live dispute about what public large language model leaderboards are measuring. If a benchmark's reference answers are wrong, a model that reasons correctly gets marked down, and the aggregate index built on top of that benchmark propagates the error to anyone reading it.

That failure mode is not unique to physics. Evaluation scaffolding has repeatedly proven to be the variable that moves scores: Nvidia's AVO harness took Claude Opus 5 from 30% to a perfect ARC-AGI-3 run without changing the model at all. Here the scaffolding under scrutiny is the grading itself.

What comes after saturation

The authors do not present the corrected numbers as vindication of model capability in general. Their conclusion is narrower and sharper: current benchmarks substantially understate what frontier models can do on well-posed, closed-ended physics problems, and those problems are close to exhausted as a source of signal.

That points the field toward harder, expert-validated evaluations — open-ended research tasks where a verifiable final answer does not exist, and where an expert has to judge the reasoning rather than check a number. Building those is considerably more expensive than scraping problem sets, which is part of why the broken ones stayed in circulation.

FAQ

Which physics benchmarks were audited?

Six: HLE-Physics, CMT-Benchmark, CritPt, UGPhysics, PRISM-Physics and PHYBench. The audit covered text-only problems with verifiable final answers, and corrected scores were computed on the subsets retained after expert review.

Does this mean frontier models have solved physics?

No. The paper's claim is that models are near-saturation on closed-ended physics problems with verifiable answers, which is a narrow slice of physics work. The authors explicitly call for more demanding, expert-validated evaluations rather than declaring the problem solved.

Why do the original benchmark scores matter if they were wrong?

Because aggregate indices repeat them. Low physics scores featured in the Artificial Analysis Intelligence Index (2026) shaped the public read on model reasoning; if the underlying grading was faulty, that read was too.

a:["$","div",null,{"className":"mx-auto max-w-7xl px-4 py-8 sm:px-6","children":[["$","script",null,{"type":"application/ld+json","dangerouslySetInnerHTML":{"__html":"{\"@context\":\"https://schema.org\",\"@type\":\"BreadcrumbList\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https://ainewsway.com\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"#llm-evaluation\",\"item\":\"https://ainewsway.com/tag/llm-evaluation\"}]}"}}],["$","script",null,{"type":"application/ld+json","dangerouslySetInnerHTML":{"__html":"{\"@context\":\"https://schema.org\",\"@type\":\"ItemList\",\"@id\":\"https://ainewsway.com/tag/llm-evaluation#itemlist\",\"name\":\"#llm-evaluation Articles\",\"numberOfItems\":1,\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"url\":\"https://ainewsway.com/articles/physicists-re-graded-the-physics-benchmarks-gpt-56-sol-jumped-from-473-to-787-mu4u16mv\",\"name\":\"Physicists Re-Graded the Physics Benchmarks. GPT-5.6-Sol Jumped From 47.3% to 78.7%.\"}]}"}}],["$","$L1d",null,{"items":[{"label":"# llm-evaluation"}]}],["$","h1",null,{"className":"mb-8 text-3xl font-bold text-zinc-900 dark:text-zinc-100","children":["#","llm-evaluation"]}],[["$","$L1e",null,{"articles":[{"id":"c70b1852-b75c-46fa-a442-9a6b8bc15874","slug":"physicists-re-graded-the-physics-benchmarks-gpt-56-sol-jumped-from-473-to-787-mu4u16mv","title":"Physicists Re-Graded the Physics Benchmarks. GPT-5.6-Sol Jumped From 47.3% to 78.7%.","subtitle":"A 51-author audit found most wrong answers on leading physics evaluations were grader errors, bad reference solutions and underspecified questions — not failures of model reasoning","content":"$1f","excerpt":"A 51-author audit of six physics benchmarks found most wrong answers were grader errors, not model errors. Corrected scores jumped as much as 31 points.","coverImage":"https://cdn.ainewsway.com/uploads/2026/09/ab3901a1-82c8-4f5f-abc6-7c6f1b9f04d3.webp","coverImageCaption":"Enrico Fermi at the blackboard — the kind of expert physics judgment the new audit used to re-grade six machine benchmarks.","authorId":"6773f67d-d7fa-4b19-8f38-384f10ac761b","status":"PUBLISHED","publishedAt":"2026-09-17T01:10:19.820Z","viewCount":0,"likeCount":0,"readingTime":4,"isFeatured":false,"isBreakingNews":false,"metaTitle":"Physics Benchmarks Re-Graded: GPT-5.6-Sol Hits 78.7%","metaDescription":"Experts audited six physics benchmarks and found grading errors, not model errors. GPT-5.6-Sol rose from 47.3% to 78.7% on HLE-Physics after correction.","metaKeywords":["physics benchmark","HLE-Physics","CMT-Benchmark","GPT-5.6-Sol","LLM evaluation","benchmark saturation","CritPt"],"ogImage":null,"aiSummary":"A 51-author study re-graded six widely used physics benchmarks with domain experts and found most answers scored as wrong were grader errors, faulty reference solutions or underspecified questions. GPT-5.6-Sol's mean@4 rose from 47.3% to 78.7% on HLE-Physics and 61.0% to 87.2% on CMT-Benchmark. The authors conclude current benchmarks understate model physics ability and are near saturation, calling for expert-validated evaluations.","aiTags":[],"sentiment":null,"articleType":"ANALYSIS","videoEmbed":null,"videoType":null,"seoScore":85,"isAutoGenerated":true,"isBackfill":false,"pipelineStage":null,"copyrightScore":null,"similarityScore":null,"matchedSourceId":null,"createdAt":"2026-09-17T01:09:48.633Z","updatedAt":"2026-09-17T01:10:19.821Z","contentUpdatedAt":null,"deletedAt":null,"author":{"id":"6773f67d-d7fa-4b19-8f38-384f10ac761b","name":"Seung Jung","username":"Seung Jung","avatar":null},"categories":[{"id":"e2b5fffc-de32-42aa-86fe-c75f219d6b4b","name":"AI & Machine Learning","slug":"ai-ml"}],"tags":[{"id":"b4959a7c-29fe-47b5-a58d-c6f2b9fc343e","name":"Benchmark","slug":"benchmark"},{"id":"65cfd7aa-e136-47fb-a2a4-f49011dc0c0e","name":"GPT-5.6","slug":"gpt-5-6"},{"id":"d42635e5-4977-4268-a9b0-5f40e3aef277","name":"LLM Evaluation","slug":"llm-evaluation"},{"id":"f67b7e35-d31e-45e3-bdb1-18006cce2197","name":"Physics","slug":"physics"},{"id":"1e9dae7d-e1b8-4d75-a6d8-3926a69bc094","name":"arXiv","slug":"arxiv"}],"translations":[],"celebrities":[{"id":"b086135e-3126-46cd-b591-67f073dff1bb","nameKo":"OpenAI","nameEn":"OpenAI","slug":"openai","profileImage":"https://ui-avatars.com/api/?name=OpenAI&size=400&background=F97316&color=fff&bold=true"}]}]}],"$L20"]]}]

#llm-evaluation

Physicists Re-Graded the Physics Benchmarks. GPT-5.6-Sol Jumped From 47.3% to 78.7%.

Subscribe to Newsletter

#llm-evaluation

Physicists Re-Graded the Physics Benchmarks. GPT-5.6-Sol Jumped From 47.3% to 78.7%.