OpenAI Cuts ChatGPT Storage Reads to 10ms

OpenAI said on Oct. 23 that it cut the median global read latency of Habitat, the storage platform behind ChatGPT and its other products, from roughly 120 milliseconds to roughly 10 milliseconds. In the same engineering post, the company disclosed it is building its own exabyte-scale storage system to sit alongside Azure Cosmos DB.

The post, by Jon Lee, Hrachya Hakobyan and Chaomin Yu, is the second of two parts. Habitat serves more than 500 petabytes of data to over 1 billion users across 40 geographic regions.

Key takeaways

• Global read p50 latency fell about 12x, from roughly 120 ms to roughly 10 ms, through data placement, selective replication, a Valkey cache and a Rust rewrite.
• After trying per-team database containers, OpenAI moved back to one shared container with hot-key throttling and tiered quality of service, and reports no noisy-neighbor incidents since.
• OpenAI is developing an in-house exabyte-scale store that combines Rockset's RocksDB Cloud core with Raft-based replication.

How OpenAI cut ChatGPT read latency to 10ms

Reads make up more than 90% of Habitat requests, so OpenAI attacked latency in layers. Data sits near users when capacity allows, mainly for data residency. Only the smallest, most latency-sensitive datasets, such as authentication metadata, are replicated broadly, some to all 40 Azure regions.

Everything else goes through a lookaside cache on Valkey servers, invalidated through the Cosmos DB change feed within seconds worldwide.

OpenAI also rewrote the service in Rust. "Together, these changes have reduced our overall global read p50 from roughly 120 ms to roughly 10 ms," the authors write. The slowest uncached reads still cross distant regions.

Why OpenAI went back to a shared Cosmos DB container

Multi-tenancy took three tries. First, teams shared a default container and chose their own partition keys; common keys such as user IDs collided, and noisy neighbors hit unrelated data. Microsoft's documentation caps Cosmos DB at 10,000 request units per second per partition.

Isolated per-team containers fixed that but caused over-provisioning and launch-day scaling problems, and pushed capacity planning onto product teams.

The third phase returned to a shared container, adding hot-key throttling, tiered QoS that throttles offline jobs first, and usage-based partition splits. "We have not had any provisioning or noisy neighbor incidents in the months since, despite accelerating product development," the authors write.

The outage that changed the Azure relationship

In late 2024, a power outage hit OpenAI's primary Azure region. Failing over the largest accounts would have taken several hours, a scale that had not been tested. About a month later, failing back caused a second serious outage when a connection stampede overwhelmed the original region's gateway.

The two companies responded with joint monitoring, shared tests and direct paging channels. "When companies grow to OpenAI's scale, managed services become less 'managed,'" the post says. Today its largest Cosmos DB account holds tens of petabytes and serves more than 20 million queries per second.

Context

Part I, published Sept. 11, covered Habitat's move from a Python library to a service and its Rust rewrite, which OpenAI said was 6x more CPU efficient and 15x more memory efficient. Part II turns to the data underneath.

What's next

OpenAI says it is building "a custom in-house exabyte-scale online storage system, optimized for bulky Habitat data types." It pairs the RocksDB Cloud core from Rockset, which OpenAI agreed to acquire in June 2024, with Raft-based replication, and uses Cosmos DB's document model. The post gives no launch date and does not say Cosmos DB will be replaced.

FAQ

What is OpenAI Habitat?
OpenAI's online storage platform, used by all its products. It is a multi-tenant layer on Azure Cosmos DB handling caching, authorization, residency, encryption and routing.

Is OpenAI leaving Azure Cosmos DB?
Not according to the post. The in-house system is still in development and aimed at bulky data types.

What made the biggest latency difference?
OpenAI does not break it down. It credits placement, replication, caching and the Rust rewrite together.

References

• Rapidly scaling online storage to serve over 1 billion ChatGPT users (Part II) — OpenAI (Oct. 23, 2026)
• Rapidly scaling online storage to serve over 1 billion ChatGPT users (Part I) — OpenAI (Sept. 11, 2026)
• Service Quotas and Default Limits - Azure Cosmos DB — Microsoft Learn
• OpenAI acquires Rockset to enhance ChatGPT real-time data processing — InfoWorld (June 25, 2024)