SageMaker Routes LLM Calls by KV Cache State
AWS on Nov. 17 added KV cache-aware routing to SageMaker AI real-time endpoints. In AWS tests with Llama 3.1 70B, median time to first token fell by up to 97% and throughput rose by up to 234% versus random routing. The feature is free but needs a vLLM-based AWS container.
Amazon Web Services on Tuesday, Nov. 17, added KV cache-aware routing to Amazon SageMaker AI real-time endpoints, a strategy that sends each large language model request to the instance that already holds the longest cached copy of its prompt. In AWS benchmarks with Llama 3.1 70B, the feature cut median time to first token by up to 97% compared with random routing, according to a post on the AWS Machine Learning Blog.

Key takeaways
- A new KV_AWARE routing strategy uses cache state reported by the model container, not guesses from the request text.
- In AWS tests on seven ml.p5.48xlarge instances, median time to first token fell by up to 97% and throughput rose by up to 234% versus random routing.
- The feature is free, available in all commercial Regions with SageMaker real-time inference, and needs a vLLM-based AWS container.
How SageMaker KV-aware routing works
When an LLM processes a prompt, it stores intermediate results, the KV cache, in GPU memory. If the same opening text reaches the same instance again, the model can skip that work. On a fleet of several instances, however, requests are usually spread around, so caches rarely get reused.
With KV-aware routing, the model container publishes an event each time it stores or evicts cache entries. SageMaker keeps a managed index of that state, which AWS says holds only one-way, truncated hashes mapped to instances. It never stores prompts, token IDs, request payloads or KV tensors.
When a request arrives, the endpoint tokenizes the prompt, looks it up in the index and weighs how much of the prompt each instance has cached against how busy it is. If no instance has a useful match, or all candidates are busy, the request falls back to load-aware routing.
KV-aware vs. prefix-aware routing
The launch is the second part of a series. In September, AWS introduced prefix-aware routing, which hashes the first bytes of a request and sends matching requests to the same instance. That method cannot see when an instance has evicted its cache under memory pressure, and it can scatter short, early turns of a conversation.
KV-aware routing matches on tokens and on what is actually cached, so it needs no prefix length to tune. The trade-off is overhead: AWS measured 1.3 to 1.9 milliseconds per request for prefix-aware routing, while the new lookup takes a few milliseconds for short prompts and more for long ones, mostly for tokenization. SageMaker now offers four strategies: RANDOM, LEAST_OUTSTANDING_REQUESTS, PREFIX_AWARE and KV_AWARE.
SageMaker benchmark results
AWS ran 30-minute tests with 20 or 100 simulated users holding multi-turn chats and document Q&A sessions, with prompts of up to about 64,000 tokens. The KV cache hit rate rose from 3% to 48% under random routing to 80% to 93%. Gains were smallest with short prompts.
In a smaller test with Qwen3-4B-Instruct-2507 on two ml.g5.xlarge instances, average model latency dropped from 4,124 and 5,100 milliseconds to 1,235 and 1,314 milliseconds across two runs. SageMaker's own overhead rose by 14 to 19 milliseconds on average. These are vendor-run tests, and AWS says results depend on the workload.
Context: requirements and limits
At launch the feature works only with the AWS vLLM Deep Learning Container, vLLM 0.33.0 or later, and OpenAI-compatible Completions or Chat Completions payloads. It requires at least two instances or two inference component copies. Bidirectional streaming is not supported, and requests with a non-null cache_salt field are rejected with a 400 error. Fallbacks are not reported in the console or metrics at launch.
The launch adds to a run of recent AWS AI tooling releases, including the open-source Strands harness.
What's next
Customers turn the feature on by adding the SM_VLLM_KV_EVENTS_CONFIG environment variable to the model and setting RoutingStrategy to KV_AWARE. AWS advises teams to compare it with prefix-aware routing on their own traffic and to keep prefix-aware routing if the two perform about the same, since it is simpler.
FAQ
Does KV-aware routing cost extra?
No. AWS says it is available at no additional charge; customers pay only for endpoint instances.
Does it help single-instance endpoints?
No. With one instance, every request goes to that instance, so the setting has no effect.
Does it speed up the first request in a conversation?
No. The first request with new context has nothing cached. The gains apply to follow-up requests.
References
- Amazon SageMaker AI real-time endpoints now support KV cache aware routing — AWS Machine Learning Blog (Nov. 17, 2026)
- Reduce LLM latency with prefix-aware routing on Amazon SageMaker Inference — AWS Machine Learning Blog (Sept. 10, 2026)
Editor responsible for AI Newsway. Writes about AI, developer tools, cloud and productivity technology.
Comments 0
Comments are currently disabled.