An Agent That Scores 77% Only Works Every Time on 53% of Tasks

Run your agent on the same task five times before you ship it, because the accuracy figure you are quoting almost certainly describes a different system than the one your users will meet. That is the practical upshot of new work from IBM Research, which measured a ReAct agent at 77.4% average success on the AppWorld benchmark and then found that only 53.0% of those tasks came back correct on all five attempts.

Key takeaways

• The 24.4-point spread between those two figures — what IBM calls the consistency gap — appeared with decoding pinned at temperature zero and nothing about the tasks changing between runs.
• A diagnostic called the Consistency Analyzer locates the unstable steps from a single recorded trace, needing one extra model call per decision and no labels, graders or environment replay.
• Guidelines generated from that diagnosis lifted the all-five pass rate to 69.0% and pushed average accuracy up to 81.0% at the same time, and the toolkit is on GitHub.

An averaged score is the wrong contract

Leaderboards quote Mean@k: run the suite k times, average the pass rate, publish the number. It is a fair description of typical behaviour and a poor description of what a production workflow promises, because reconciling a transaction or checking a contract clause is not a task you get to attempt repeatedly and keep the best result.

The metric that matches that promise is Pass^k, the share of tasks an agent gets right on every single one of k runs — and IBM is emphatic that it should not be confused with the optimistic Pass@k familiar from code-generation papers, which needs only one attempt out of k to land. On AppWorld's 168-task test_normal split, switching from the first metric to the second knocks the headline down by nearly a quarter, and the shortfall reaches roughly 30 points on the hardest tier.

Not a sampling bug, a distribution problem

What makes the finding awkward is that the usual levers do not touch it. Greedy decoding and fixed seeds govern how a probability distribution is collapsed into a token, so they cannot help when the distribution itself puts near-equal mass on two plausible next actions; on hosted infrastructure, ordinary sources of numerical jitter such as floating-point non-associativity and request batching are enough to reorder that near-tie between runs. Chain dozens of such decisions together and a small per-step flip probability becomes a large per-run one.

IBM's framing is that reliability sits on a different axis from capability, which means the reflex to upgrade the model is misdirected: a stronger AI agent posts a better average without necessarily becoming more repeatable.

Finding the flip points cheaply

The Consistency Analyzer works backwards from one trajectory the agent has already produced. For each decision in that trace it issues a single call asking for five completions against the recorded context, scores how much the output moves, and writes the result into a per-step scorecard. Nothing is re-executed against the environment, no ground truth is required, and the method touches no model internals — which is what makes it viable on production traffic, where replaying a task end to end is frequently impossible.

Flagged steps are then turned into guidelines in ALTK-Evolve's existing format and retrieved at inference time. The published examples read like engineering habits rather than answer keys: anchor a regex to line starts when counting checkbox markers, confirm a search returned the right record before acting on it. Critically, the analyzer hunts for instability rather than for failures, so it flags steps the agent happened to get right on the run being examined.

What moved, and by how much

Generating guidelines from one baseline trajectory per task and then testing on five fresh runs took Pass^5 from 53.0% to 69.0% and Mean@5 from 77.4% to 81.0%, halving the gap to 12.0 points and converting nearly a third of previously flaky tasks into ones that passed every time. Medium-difficulty work gained most in absolute terms at 22.9 points, ahead of 14.3 for hard tasks and 12.2 for easy ones, and average accuracy never went backwards at any tier — a constraint IBM set deliberately, since buying consistency with accuracy would simply relocate the unreliability.

Transfer looks real too. The same guidelines applied to a different task in the same scenario still delivered a 13.0-point Pass^5 improvement, and on the weaker gpt-oss-120b the similar-task gain of 8.7 points actually beat the 6.0-point same-task gain — a pattern more consistent with capturing reusable failure modes than with memorising one run.

Why it matters beyond one benchmark

Enterprise buyers have spent the year being sold agents on single-shot scores, and this is a concrete, cheap instrument for interrogating those claims: ask a vendor for Pass^3 and see what comes back. IBM has open-sourced the analyzer and guideline generation in the ALTK-Evolve repository, with the methodology detailed in an arXiv technical report and summarised in the team's Hugging Face write-up.

It also fits a broader pattern in agent evaluation this year, where the headline capability number keeps surviving scrutiny that the behaviour behind it does not — much as a 25-turn pressure test found models whose reasoning held while their conduct did not.

FAQ

What is the difference between Pass^k and Pass@k?

Pass@k counts a task as solved if any one of k attempts works, which is reasonable when output can be verified and retried. Pass^k counts it only when all k attempts work, which is closer to what a user experiences asking the same question twice. Pass^k is always the lower of the two.

Does lowering temperature fix agent inconsistency?

No. The 24.4-point gap was measured at temperature 0.0. Decoding parameters decide how a distribution is turned into a token but leave the distribution untouched, and hosted endpoints shift those probabilities slightly from run to run regardless.

Is the Consistency Analyzer available to use?

Yes, as open source in the ALTK-Evolve repository. It runs against traces you already have, needs one additional model call per decision step and no ground-truth labels, so the cost of a first measurement is low.