Tsinghua AKV Cuts AI Agent Costs 27.7%

Researchers at Tsinghua University and the Chinese University of Hong Kong have open-sourced AKV, a serving engine that lets AI agents delete and re-pack parts of their own key-value (KV) cache instead of summarizing old history. In a paper posted to arXiv on Oct. 20, the team reports a 27.7% average cost cut and a 1.7-point score gain over threshold-triggered summarization. MarkTechPost covered the release on Oct. 26.

Key takeaways

• AKV adds two cache operations, "drop" and "repos," so an agent can stop attending to stale tool output without forcing a costly re-prefill of its whole history.
• Across three models and four agent benchmarks, a policy that keeps only the latest 12 tool responses cut average cost 27.7% and raised average scores 1.7 points.
• The code is on GitHub under the Apache-2.0 license, built on the SGLang serving framework with an OpenAI-compatible API.

Why agent context costs pile up

Long-running agents re-read a growing log of observations and actions on every step. The paper finds that KV cache reads made up 82.4% of total API cost in its analysis, which makes the cache, not the generated text, the main bill.

The usual fix is context engineering: trimming or summarizing old turns. But editing the start of a conversation invalidates the cached states built on it, so the server must recompute them. The authors call their alternative "cache engineering," in which the agent edits the cache directly.

How AKV drop and repos work

"Drop" excludes selected cache entries from future attention while leaving positions unchanged. "Repos" rewrites the position encodings of retained keys so they sit in a compact range. MarkTechPost, citing the paper's ablations, reported that adding repos lifted GPT-OSS-120B's BrowseComp-Plus pass rate from 55% to 62%.

Developers keep sending stateless, OpenAI-style requests. Two optional fields, drop_message and reposition, tell the server what to remove and when to compact, and AKV handles cache reuse, eviction and restoration behind the scenes. Policies can be turn-based, token-based or model-driven; a built-in tool also lets the model call drop_context on itself.

AKV benchmark results on SWE-bench and BrowseComp

The team tested GPT-OSS-120B, MiniMax-M2.7 and MiroThinker-1.7-Mini on BrowseComp-Plus, GAIA-Text, SWE-bench Verified and xbench-DeepSearch-2510. On SWE-bench Verified, GPT-OSS-120B's score rose from 40.0 to 43.2 while estimated cost per task fell from 20.0 to 9.7 U.S. cents.

Serving gains followed. On eight Nvidia A800-80GB GPUs, AKV delivered 1.18 times the output throughput, 1.53 times the input throughput and a 1.51-fold speedup in P95 request latency. At 16 concurrent requests, P95 latency dropped from 229.90 seconds to 30.63 seconds.

The authors also let a model refine its own cache policy, which they say cut cost a further 7.1% beyond the expert-designed policy. Costs in the paper are API-equivalent estimates based on published prices, not measured invoices.

Context: the cost of long-running agents

The work adds to evidence that agent bills depend heavily on how context is handled, not just on the model. This site has previously covered how LLM bug-fixers broke working code during long automated runs, another sign that agent scaffolding shapes outcomes.

AKV's reach is limited for now. Drop and repos support only Qwen, GPT-OSS and MiniMax-M2 text models, and MarkTechPost notes the system requires Nvidia GPUs, with no CPU or Apple MLX path.

What's next

Because AKV keeps the standard Chat Completions interface, teams already serving open-weight models on SGLang could test it without rewriting agent code. Independent reproductions and support for more model families will show whether the savings hold outside the authors' benchmarks.

FAQ

What is KV cache engineering for AI agents?

It means letting an agent directly remove or reposition entries in the model's key-value cache, rather than rewriting the text history. That avoids recomputing the cache each time old context is trimmed.

Is AKV free to use?

Yes. The code is released on GitHub under Apache-2.0. It is a serving system and policy library and ships no model weights.

Which models does AKV support?

Qwen-family, GPT-OSS and MiniMax-M2 text models, according to the project README.

References

• Cache Engineering for Agents — arXiv, Tianyu Fu et al. (Oct. 20, 2026)
• thu-nics/AKV — GitHub
• Tsinghua AI Open-Sources AKV, a Cache Engineering Server for LLM Agents on SGLang — MarkTechPost (Oct. 26, 2026)