AI Newsway

GitHub Doubled Its Commit Volume in Four Months. Then It Ran Out of Room.

The postmortem for a 7-hour August 17 failure points at capacity ceilings and retry storms, with a rival launching a competing platform the same morning

|3 min read0
AI Summary
GitHub's postmortem, attributed to CTO Vladimir Fedorov, says the 7-hour-47-minute August 17 outage was caused by capacity ceilings in its Central US facility and a Copilot client retry loop, not by any deploy or config change, as monthly commits doubled from 1.4 billion in April to 2.9 billion. Error rates hit roughly 20% for web and API traffic. GitHub has added 3 million CPU cores but is power-constrained, so watch its accelerating shift to Azure.
Server racks in a data center, the kind of infrastructure GitHub says failed to scale with peak traffic during its August 17 outage.
Server racks in a data center, the kind of infrastructure GitHub says failed to scale with peak traffic during its August 17 outage.

Two numbers explain most of what went wrong at GitHub last week. Monthly commits on the platform stood at 1.4 billion in April. By August they had reached 2.9 billion. Somewhere between those figures, a component in the company's Central US facility hit a ceiling it could not grow past, and on August 17 the largest code host on the internet spent 7 hours and 47 minutes inside a critical incident.

The company's account, attributed to chief technology officer Vladimir Fedorov, rules out the explanations engineers usually reach for first. No deploy triggered it. No configuration change triggered it. Both this failure and an Actions breakdown eleven days earlier were, in GitHub's framing, capacity problems — components that should have been scaled up before demand arrived, and were not.

The Blast Radius

Authentication was the pivot point. Once capacity pressure reached it, the damage spread across github.com, Actions, the REST and GraphQL APIs, pull requests, issues, and Copilot simultaneously. External monitoring during the window put error rates near 20% for web and API traffic and roughly 50% for archive and raw repository downloads, with the disruption beginning around 13:40 UTC.

Getting back was not a single action. Engineers rerouted traffic, walled off the failing infrastructure, and restored services in stages. Most came back the same day. Copilot did not, and the reason is instructive: failures there set off a client-side retry loop that generated additional load precisely when the platform had the least capacity to absorb it. That amplification had to be contained before traffic could safely return.

Independent analyses of the incident have pointed to a cascade involving autoscaling that watched the wrong service and downstream proxy nodes exhausting their limits, with client retry behavior multiplying token-service requests by an order of magnitude. GitHub's own writeup does not go into that level of mechanism, reserving it for a separate root cause analysis with a full technical timeline.

Hardware Against a Power Ceiling

GitHub is not short on spending. The company says it has brought online more than 3 million CPU cores, 120 petabytes of high-speed storage, and substantial additional network capacity as part of reliability commitments made earlier this year.

The constraint turned out to be electrical rather than financial. GitHub installed as much hardware as the available power in its existing data centers permitted — and then leaned harder on Azure. That migration has moved fast: Azure handled 12% of platform load in May and now carries roughly 58%, along with half of all Git operations. GitHub credits the expanded footprint for absorbing growth in Actions job runs and for accelerating work on its largest monorepos, where the next target is an architecture whose read capacity scales linearly with reader count.

Operations Lagged the Growth

Fedorov concedes a second failure alongside the capacity one: as change accelerated, operational practice did not keep up. GitHub says it has moved teams onto availability work and invested in testing, safer rollouts, observability, and alerting, while isolating critical systems and cutting shared dependencies between them.

Two concrete changes came out of August. Retry limits, retry budgets, and variable timeouts are being standardized across service-to-service calls, aimed squarely at the storm behavior that prolonged the Copilot recovery. And low-priority CPU and memory alerts are under review to flag components likely to buckle during sudden spikes.

A Competitor Was Watching

Cursor launched Origin, its own code hosting product, on the morning GitHub went dark. Whether that timing was opportunism or coincidence, it made the commercial stakes legible: reliability is the entire value proposition of a platform that everything else depends on. With commits still climbing and the Azure migration incomplete, GitHub's fixes are racing the next peak.

How do you feel about this article?

SJ

Discussion

Sign in to post
Loading...

Related articles

Cursor Launches Origin, Taking Aim at GitHub's Home Turf
Developer Tools

Cursor Launches Origin, Taking Aim at GitHub's Home Turf

Cursor's new Origin service brings repos, pull requests, and two-way GitHub sync into the AI editor, opening in beta to paid users.

Seung Jung30 days ago
GitHub's HydraFusion Stops Picking a Model. It Builds a Workflow Instead.
Developer Tools

GitHub's HydraFusion Stops Picking a Model. It Builds a Workflow Instead.

GitHub's Project HydraFusion assembles a multi-model plan for every Copilot coding request, trading single-model simplicity for sharply lower cost.

Seung Jung4 days ago
Cloudflare Cut Astro's Open Issues by 85% With Four Agents That Refuse to Share Context
Developer Tools

Cloudflare Cut Astro's Open Issues by 85% With Four Agents That Refuse to Share Context

Separate reproduction, diagnosis, verification and fix agents hand off through a report file, with the original bug reporter acting as the acceptance test.

Seung Jung25 days ago
Modular Opens Its AI Stack Across Trainium, TPUs and Qualcomm Silicon
Developer Tools

Modular Opens Its AI Stack Across Trainium, TPUs and Qualcomm Silicon

Modular put the Mojo compiler under Apache 2.0 and extended its platform to Trainium, TPUs and Qualcomm accelerators, opening the stack across rival silicon.

Seung Jung28 days ago
Microsoft's Record 966-Flaw Patch Month Moves the Bottleneck to Defenders
Developer Tools

Microsoft's Record 966-Flaw Patch Month Moves the Bottleneck to Defenders

Microsoft fixed 966 vulnerabilities in September, pushing its 2026 total near 2,750. Security teams say triage, not discovery, is now the hard part.

Seung Jung17 hours ago
Mixedbread Bets Search Should Be a Cheap Specialist, Not a Frontier Model
Developer Tools

Mixedbread Bets Search Should Be a Cheap Specialist, Not a Frontier Model

Mixedbread launched Toast 1, a specialised search agent it says matches frontier models at up to 10x lower cost and cuts legal-benchmark tokens by 3.5x.

Seung Jung33 days ago