Caching

Caching is the practice of keeping a copy of expensive-to-produce data in fast storage so later requests can be served without redoing the work. A cache sits between a consumer and a slower source such as a database, a remote API, or a model. Every cache needs an eviction policy and an expiry rule, because stale data is the price of speed. Time-to-live settings, explicit invalidation on write, and key namespacing are the usual controls. Redis and Memcached serve as shared in-memory caches for application data, CDNs such as Cloudflare cache static assets close to users, and browsers cache responses locally. Language model providers apply the same idea. Prompt caching stores the processed prefix of a long system prompt so repeated calls skip recomputation, and semantic caching returns a stored answer when a new question is close enough in embedding space. The hard part is invalidation. A cache that is never cleared serves wrong data, and one cleared too eagerly gives no benefit, so teams pair a short TTL with targeted invalidation on the writes that actually matter.