DoorDash has automated the removal of stale feature flags with a two-phase multi-agent LLM system that produced engineer-approved pull requests for 45 of 50 test cases, averaging $4.79 and 13.8 minutes per flag against an internal baseline of one to two hours of manual work. The results were published in an ICSME 2026 industry track paper by DoorDash engineers Atharv Chandratre and Jai Datar.
Key takeaways
- The system cleared 45 of 50 stale flags into usable pull requests, at 90% overall and 85% on the most complex cases.
- A Claude Sonnet orchestrator plans each cleanup over MCP; Claude Opus agents do the edits, four at a time, each in its own git worktree.
- Nothing opens a pull request until the build, tests, 95% patch coverage and static analysis all pass locally.
Why 623 repositories of dead flags became a maintenance problem
A feature flag is runtime configuration: it gates a gradual rollout, splits an experiment, or acts as a manual kill switch. Once the feature is fully shipped, the flag stops earning its keep. The branching logic stays in the codebase as dead weight, and a dormant path that nobody tests can be re-enabled by accident.
The scale is what makes this unmanageable by hand. The paper counts more than 56,000 feature flags across 623 repositories, 1,659 of them stale as of May 2025. DoorDash's own engineering writeup puts the platform above 60,000 dynamic values today, with roughly 2,300 new ones created every month. Rule-based cleanup tools, the authors note, proved insufficient against DoorDash's codebases β removing a flag correctly means reasoning about what the surrounding conditionals collapse to, not pattern-matching a symbol.
How the two-phase agent system works
Phase one is planning. An orchestrator agent running Claude Sonnet pulls the Jira ticket, queries live rollout data from the experimentation platform over MCP, runs a repository-wide semantic pass, and emits a structured cleanup plan naming the value the flag should be replaced with. An engineer confirms that target value before any code is touched β the one human checkpoint in the loop.
Phase two is surgery. One removal agent per confirmed value, running Claude Opus, up to four concurrent per repository, each in a separate git worktree so parallel edits cannot collide. Gradle runs with --no-daemon to keep build state from leaking between worktrees, and each agent carries a hard one-hour timeout.
The gates are the interesting part. Before an agent is allowed to open a pull request it has to pass the build, the test suite, a JaCoCo patch-coverage floor of 95%, and Detekt static analysis. Fail any of them and no PR appears. The model split is explicitly economic: the cheaper model handles metadata gathering and planning, and the expensive one is reserved for call-chain reasoning and the edits themselves.
What the 50-flag evaluation actually showed
The evaluation set was not cherry-picked toward easy wins. It covered 41 boolean values, six strings and three integer types, split into six simple cases touching one or two files, 18 medium cases across three to five files, and 26 complex cases involving nested conditionals and cross-file dependencies.
First-pass success tracked complexity almost exactly: 100% on simple, 94% on medium, 85% on complex. So did cost. Simple cleanups averaged 7.5 minutes and $2.69, medium 10.4 minutes and $3.46, complex 17.7 minutes and $6.20. Fourteen of the 45 accepted pull requests needed one minor fix; five cases required an engineer to step in.
The failure modes are specific enough to be useful. Six revisions came from patch-coverage shortfalls and eight from incomplete removals β a leftover variable, or a reference surviving deeper in the call chain. Every one of the five human interventions was associated with call-chain depth and parameter threading across interfaces, which is precisely the reasoning that a coding agent working from a file-local view is worst at.
What this says about agent deployments that stick
Read the architecture and the pattern is familiar. Isolation per unit of work, a narrow human approval point, and mechanical gates that the agent cannot argue its way past β the same shape Cloudflare used when it cut Astro's open issues by 85% with four agents that refuse to share context. The model choice is a line item; the gates are the product.
The economics hold even at the worst tier: $6.20 and under 18 minutes against one to two engineer-hours. The caveat is sample size. Fifty flags is a pilot, not the 1,659-flag backlog, and both cost and failure concentrate in the complex tier that dominates that backlog. Whether 85% survives contact with the full queue is the number worth watching.
FAQ
Which models does DoorDash's cleanup system use?
Two, split by job. A Claude Sonnet orchestrator handles Jira retrieval, MCP queries against live rollout data, and planning. Claude Opus agents perform the code modifications, because call-chain reasoning is where the more capable model earns its cost.
What stops the agents from breaking the build?
Each agent works in an isolated git worktree and must pass the build, the test suite, a 95% JaCoCo patch-coverage threshold and Detekt static analysis before a pull request is created. A one-hour hard timeout caps any agent that stalls.
Is the cleanup fully autonomous?
No. An engineer confirms the replacement value at the end of the planning phase before any code changes begin, and the resulting pull requests still go through normal review. Five of 50 cases in the evaluation needed direct engineer intervention.
