AWS has open-sourced a benchmark that refuses to use fixtures. Instead of scoring agents against canned scenarios, aws-bench provisions real infrastructure in disposable AWS accounts and asks the agent to work on it.
The tasks are the ones cloud engineers actually get paged for. Diagnosing misconfigurations. Provisioning infrastructure. Operating live environments.
How a run works
Each scenario deploys a set of resources into an isolated account, defined as CDK stacks. Once the environment is up, the agent under test executes its task inside a sandboxed container using scoped credentials.
Scoring happens afterward through an automated verifier. That verifier is either an LLM judge or a programmatic check run against the live AWS state the agent left behind.
The shipped datasets cover both basic and advanced difficulty. Task categories span observability, compute and data, databases and storage, multi-region EC2, serverless, streaming and IoT, reference architectures, and multi-service troubleshooting.
The project builds on Harbor, an existing open-source agent evaluation framework, extending it with AWS resource provisioning, scenarios and verifiers. Built-in adapters cover Claude Code, Codex, Kiro CLI and Mini-SWE-Agent, and anything Harbor already supports — Gemini CLI and OpenCode among them — should work.
The setup cost is real
This is not a benchmark you clone and run over lunch.
Running it requires credentials with access to an organization's management account, plus permission to create and manage member accounts and organizational units. Few engineers hold that level of access casually, and fewer security teams hand it out for an experiment.
The harness is currently pinned to us-east-1. It also creates persistent resources that can accrue charges whether or not a benchmark is running. Anyone piloting it should budget for that.
The code and datasets are on GitHub under Apache-2.0. AWS positions the release primarily at researchers and model providers, though nothing stops an engineering team from adding scenarios that mirror its own architecture.
What is conspicuously missing
There are no numbers.
AWS published no baseline results and no leaderboard. Both sit on the roadmap. Releasing an evaluation harness without a single scored run is an unusual choice, and it means nobody can yet say how current agents perform on these tasks.
That absence is arguably the honest option. It also arrives at a moment when benchmark credibility is under real strain.
The trust problem this lands in
Researchers at UC Berkeley's Center for Responsible, Decentralized Intelligence recently demonstrated an agent that posted near-perfect scores on Terminal-Bench and SWE-Bench without solving a single task. The exploit targeted the scoring machinery rather than the problems.
That result is not an isolated embarrassment. It points at a structural issue: evaluation harnesses are software, and capable agents are increasingly good at finding the seams in software.
aws-bench inherits some of that exposure. Most of its tasks are graded by an LLM judge, which is exactly the kind of surface that proved gameable elsewhere. AWS documents the risk directly, noting that leftover state in an account can produce both false passes and random failures.
Real infrastructure raises the cost of faking a result, since a verifier querying live AWS state is harder to fool than a diff against expected output. It does not eliminate the problem. An agent that learns to satisfy a judge is still learning to satisfy a judge.
The more interesting question is what happens when the leaderboard appears. Benchmarks stay useful roughly until they become a marketing target, and this one will face that pressure from the moment the first vendor publishes a score.
