1d:I[70616,["/_next/static/chunks/0kcf5dt6yxdox.js","/_next/static/chunks/1su5n1uv1ti4s.js","/_next/static/chunks/0gvke1wv6udfg.js","/_next/static/chunks/1dfrle3jgmlzh.js","/_next/static/chunks/162lgvduyr-fa.js","/_next/static/chunks/2gcnnopam8e9w.js","/_next/static/chunks/2g60iab5a1pkg.js"],"Breadcrumb"] 1e:I[90762,["/_next/static/chunks/0kcf5dt6yxdox.js","/_next/static/chunks/1su5n1uv1ti4s.js","/_next/static/chunks/0gvke1wv6udfg.js","/_next/static/chunks/1dfrle3jgmlzh.js","/_next/static/chunks/162lgvduyr-fa.js","/_next/static/chunks/2gcnnopam8e9w.js","/_next/static/chunks/2g60iab5a1pkg.js"],"ArticleGrid"] 1f:T1aaf,

Run your agent on the same task five times before you ship it, because the accuracy figure you are quoting almost certainly describes a different system than the one your users will meet. That is the practical upshot of new work from IBM Research, which measured a ReAct agent at 77.4% average success on the AppWorld benchmark and then found that only 53.0% of those tasks came back correct on all five attempts.

Key takeaways

The 24.4-point spread between those two figures — what IBM calls the consistency gap — appeared with decoding pinned at temperature zero and nothing about the tasks changing between runs.
A diagnostic called the Consistency Analyzer locates the unstable steps from a single recorded trace, needing one extra model call per decision and no labels, graders or environment replay.
Guidelines generated from that diagnosis lifted the all-five pass rate to 69.0% and pushed average accuracy up to 81.0% at the same time, and the toolkit is on GitHub.

An averaged score is the wrong contract

Leaderboards quote Mean@k: run the suite k times, average the pass rate, publish the number. It is a fair description of typical behaviour and a poor description of what a production workflow promises, because reconciling a transaction or checking a contract clause is not a task you get to attempt repeatedly and keep the best result.

The metric that matches that promise is Pass^k, the share of tasks an agent gets right on every single one of k runs — and IBM is emphatic that it should not be confused with the optimistic Pass@k familiar from code-generation papers, which needs only one attempt out of k to land. On AppWorld's 168-task test_normal split, switching from the first metric to the second knocks the headline down by nearly a quarter, and the shortfall reaches roughly 30 points on the hardest tier.

Not a sampling bug, a distribution problem

What makes the finding awkward is that the usual levers do not touch it. Greedy decoding and fixed seeds govern how a probability distribution is collapsed into a token, so they cannot help when the distribution itself puts near-equal mass on two plausible next actions; on hosted infrastructure, ordinary sources of numerical jitter such as floating-point non-associativity and request batching are enough to reorder that near-tie between runs. Chain dozens of such decisions together and a small per-step flip probability becomes a large per-run one.

IBM's framing is that reliability sits on a different axis from capability, which means the reflex to upgrade the model is misdirected: a stronger AI agent posts a better average without necessarily becoming more repeatable.

Finding the flip points cheaply

The Consistency Analyzer works backwards from one trajectory the agent has already produced. For each decision in that trace it issues a single call asking for five completions against the recorded context, scores how much the output moves, and writes the result into a per-step scorecard. Nothing is re-executed against the environment, no ground truth is required, and the method touches no model internals — which is what makes it viable on production traffic, where replaying a task end to end is frequently impossible.

Flagged steps are then turned into guidelines in ALTK-Evolve's existing format and retrieved at inference time. The published examples read like engineering habits rather than answer keys: anchor a regex to line starts when counting checkbox markers, confirm a search returned the right record before acting on it. Critically, the analyzer hunts for instability rather than for failures, so it flags steps the agent happened to get right on the run being examined.

What moved, and by how much

Generating guidelines from one baseline trajectory per task and then testing on five fresh runs took Pass^5 from 53.0% to 69.0% and Mean@5 from 77.4% to 81.0%, halving the gap to 12.0 points and converting nearly a third of previously flaky tasks into ones that passed every time. Medium-difficulty work gained most in absolute terms at 22.9 points, ahead of 14.3 for hard tasks and 12.2 for easy ones, and average accuracy never went backwards at any tier — a constraint IBM set deliberately, since buying consistency with accuracy would simply relocate the unreliability.

Transfer looks real too. The same guidelines applied to a different task in the same scenario still delivered a 13.0-point Pass^5 improvement, and on the weaker gpt-oss-120b the similar-task gain of 8.7 points actually beat the 6.0-point same-task gain — a pattern more consistent with capturing reusable failure modes than with memorising one run.

Why it matters beyond one benchmark

Enterprise buyers have spent the year being sold agents on single-shot scores, and this is a concrete, cheap instrument for interrogating those claims: ask a vendor for Pass^3 and see what comes back. IBM has open-sourced the analyzer and guideline generation in the ALTK-Evolve repository, with the methodology detailed in an arXiv technical report and summarised in the team's Hugging Face write-up.

It also fits a broader pattern in agent evaluation this year, where the headline capability number keeps surviving scrutiny that the behaviour behind it does not — much as a 25-turn pressure test found models whose reasoning held while their conduct did not.

FAQ

What is the difference between Pass^k and Pass@k?

Pass@k counts a task as solved if any one of k attempts works, which is reasonable when output can be verified and retried. Pass^k counts it only when all k attempts work, which is closer to what a user experiences asking the same question twice. Pass^k is always the lower of the two.

Does lowering temperature fix agent inconsistency?

No. The 24.4-point gap was measured at temperature 0.0. Decoding parameters decide how a distribution is turned into a token but leave the distribution untouched, and hosted endpoints shift those probabilities slightly from run to run regardless.

Is the Consistency Analyzer available to use?

Yes, as open source in the ALTK-Evolve repository. It runs against traces you already have, needs one additional model call per decision step and no ground-truth labels, so the cost of a first measurement is low.

a:["$","div",null,{"className":"mx-auto max-w-7xl px-4 py-8 sm:px-6","children":[["$","script",null,{"type":"application/ld+json","dangerouslySetInnerHTML":{"__html":"{\"@context\":\"https://schema.org\",\"@type\":\"BreadcrumbList\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https://ainewsway.com\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"#agent-reliability\",\"item\":\"https://ainewsway.com/tag/agent-reliability\"}]}"}}],["$","script",null,{"type":"application/ld+json","dangerouslySetInnerHTML":{"__html":"{\"@context\":\"https://schema.org\",\"@type\":\"ItemList\",\"@id\":\"https://ainewsway.com/tag/agent-reliability#itemlist\",\"name\":\"#agent-reliability Articles\",\"numberOfItems\":1,\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"url\":\"https://ainewsway.com/articles/an-agent-that-scores-77-only-works-every-time-on-53-of-tasks-mu3nf7bd\",\"name\":\"An Agent That Scores 77% Only Works Every Time on 53% of Tasks\"}]}"}}],["$","$L1d",null,{"items":[{"label":"# agent-reliability"}]}],["$","h1",null,{"className":"mb-8 text-3xl font-bold text-zinc-900 dark:text-zinc-100","children":["#","agent-reliability"]}],[["$","$L1e",null,{"articles":[{"id":"8fd8f568-1ea7-4abf-880d-34776b05ab2d","slug":"an-agent-that-scores-77-only-works-every-time-on-53-of-tasks-mu3nf7bd","title":"An Agent That Scores 77% Only Works Every Time on 53% of Tasks","subtitle":"IBM Research says the number every leaderboard reports hides a 24-point reliability gap — and it has a cheap way to measure and close it","content":"$1f","excerpt":"IBM Research found a ReAct agent scoring 77.4% on AppWorld succeeded on all five repeat runs for only 53% of tasks. Its fix halved the gap.","coverImage":"https://cdn.ainewsway.com/uploads/2026/09/bc7c5f14-8ee6-4ec5-9952-b2f43b41b15f.webp","coverImageCaption":"IBM Research argues that an agent's average benchmark score says little about whether it will repeat the same result on the same request.","authorId":"6773f67d-d7fa-4b19-8f38-384f10ac761b","status":"PUBLISHED","publishedAt":"2026-09-16T05:17:24.681Z","viewCount":0,"likeCount":0,"readingTime":5,"isFeatured":false,"isBreakingNews":false,"metaTitle":"The 24-Point Reliability Gap Hiding in Agent Benchmarks","metaDescription":"IBM Research shows a 77.4% agent passes all five repeat runs on only 53% of tasks, and open-source consistency guidelines halve the gap.","metaKeywords":["AI agent reliability","ALTK-Evolve","AppWorld benchmark","IBM Research","Pass^k metric","agent consistency","ReAct agent","GPT-4.1"],"ogImage":null,"aiSummary":"IBM Research reports that a ReAct agent running GPT-4.1 on the AppWorld benchmark averages 77.4% success but succeeds on all five repeated runs for only 53.0% of tasks, a 24.4-point consistency gap that average scores conceal. Its Consistency Analyzer resamples a single recorded trajectory to locate flip-prone decisions, then writes guidelines that are injected at inference. The gap narrowed to 12.0 points with no loss in average accuracy, and the tooling is open source.","aiTags":[],"sentiment":null,"articleType":"ANALYSIS","videoEmbed":null,"videoType":null,"seoScore":85,"isAutoGenerated":true,"isBackfill":false,"pipelineStage":null,"copyrightScore":null,"similarityScore":null,"matchedSourceId":null,"createdAt":"2026-09-16T05:16:59.210Z","updatedAt":"2026-09-16T06:11:44.198Z","contentUpdatedAt":"2026-09-16T06:11:44.197Z","deletedAt":null,"author":{"id":"6773f67d-d7fa-4b19-8f38-384f10ac761b","name":"Seung Jung","username":"Seung Jung","avatar":null},"categories":[{"id":"e2b5fffc-de32-42aa-86fe-c75f219d6b4b","name":"AI & Machine Learning","slug":"ai-ml"}],"tags":[{"id":"971d527f-0508-47c9-aac1-39914a5f883c","name":"AI Agents","slug":"ai-agents"},{"id":"e08adcd1-380e-4eb0-8fb9-8e7803613d07","name":"IBM","slug":"ibm"},{"id":"b4959a7c-29fe-47b5-a58d-c6f2b9fc343e","name":"Benchmark","slug":"benchmark"},{"id":"dbb28dbd-4512-4daa-9191-85e040c30141","name":"Open Source","slug":"open-source"},{"id":"90225f12-a5de-4c5e-9931-264eb09a3432","name":"Agent Reliability","slug":"agent-reliability"}],"translations":[{"id":"129a4fc2-2abc-4f78-9d6f-a490b15d7b8e","language":"es","title":"Un agente que saca 77% solo acierta siempre en el 53% de las tareas","subtitle":"IBM Research sostiene que la cifra que publican todos los rankings esconde una brecha de fiabilidad de 24 puntos, y propone una forma barata de medirla y reducirla","excerpt":"IBM Research halló que un agente ReAct con 77,4% en AppWorld acertaba en las cinco repeticiones solo en el 53% de las tareas. Su solución redujo la brecha a la mitad.","metaTitle":"La brecha de fiabilidad de 24 puntos en los agentes de IA","metaDescription":"IBM Research muestra que un agente con 77,4% de precisión pasa las cinco repeticiones en solo el 53% de las tareas. Unas guías abiertas reducen la brecha."},{"id":"403e980e-6400-4099-92ed-4103364a31e7","language":"ja","title":"正答率77%のエージェント、毎回成功する課題は53%だけ","subtitle":"IBMリサーチは、あらゆるリーダーボードが掲げるその数字が24ポイントの信頼性ギャップを覆い隠していると指摘する。安価に測り、縮める手立ても示した","excerpt":"IBMリサーチによると、AppWorldで77.4%を記録したReActエージェントが5回すべて成功した課題は53%にとどまった。示された手法はこのギャップを半減させた。","metaTitle":"エージェント評価に潜む24ポイントの信頼性ギャップ","metaDescription":"IBMリサーチによれば、正答率77.4%のエージェントが5回すべて通過した課題は53%のみ。オープンソースの一貫性ガイドラインが差を半分に縮めた。"},{"id":"99121def-e466-4eb8-8436-e2bac41e83fb","language":"ko","title":"정확도 77% 에이전트, 매번 성공하는 과제는 53%뿐이다","subtitle":"IBM 리서치는 모든 리더보드가 내세우는 그 숫자가 24%포인트짜리 신뢰성 격차를 가린다고 지적했다. 이를 값싸게 측정하고 좁히는 방법도 함께 내놨다","excerpt":"IBM 리서치는 AppWorld에서 77.4%를 기록한 ReAct 에이전트가 다섯 번 모두 성공한 과제는 53%뿐임을 밝혀냈다. 제시한 해법은 이 격차를 절반으로 줄였다.","metaTitle":"에이전트 벤치마크에 숨은 24%포인트 신뢰성 격차","metaDescription":"IBM 리서치에 따르면 정확도 77.4% 에이전트가 다섯 번 모두 통과한 과제는 53%에 불과하다. 오픈소스 일관성 가이드라인이 격차를 절반으로 줄였다."},{"id":"e4d14f4f-29e8-4485-af6a-de1537e2de34","language":"zh","title":"得分 77% 的代理，只有 53% 的任務每次都做對","subtitle":"IBM 研究院指出，各家排行榜引用的那個數字掩蓋了 24 個百分點的可靠度落差，並提出一套便宜的量測與縮小方法","excerpt":"IBM 研究院發現，在 AppWorld 拿下 77.4% 的 ReAct 代理，五次重跑全數成功的任務只有 53%。其提出的做法把這道落差縮減一半。","metaTitle":"代理基準測試裡藏著 24 個百分點的可靠度落差","metaDescription":"IBM 研究院指出，準確率 77.4% 的代理只有 53% 的任務能五次全過，開源的一致性指引把落差縮減一半。"}],"celebrities":[{"id":"9a32f05b-ed4a-4e70-9b0a-cbeb4c82f3ee","nameKo":"IBM","nameEn":"IBM","slug":"ibm","profileImage":null},{"id":"b086135e-3126-46cd-b591-67f073dff1bb","nameKo":"OpenAI","nameEn":"OpenAI","slug":"openai","profileImage":"https://ui-avatars.com/api/?name=OpenAI&size=400&background=F97316&color=fff&bold=true"}]}]}],"$L20"]]}]

#agent-reliability

Un agente que saca 77% solo acierta siempre en el 53% de las tareas

Suscríbete al Boletín

#agent-reliability

Un agente que saca 77% solo acierta siempre en el 53% de las tareas