Scaling KV Cache Compression to Frontier Models

Research Author: Jeff Zheng, Alex Chen

We adapted KVzip and Attention Matching to GLM 5.3 to test how much context a large sparse frontier model can discard before task quality suffers. The experiments show where shared cache allocation helps and what fitting recovers beyond eviction.

---

Intro

2026 has been an inflection point for AI token spend. From August 2025 to August 2026, monthly AI spend per employee in the Ramp AI Spend Index rose 165% at the median, 246% for the top 10%, and 198% for the top 1%. Ramp built Router to maximize the amount of intelligence per dollar of this spend, helping reduce Ramp's inference cost by roughly 30%. We want to bend the cost curve down even further by reducing the cost of running the model itself, starting with how it stores and uses context. This post kicks off a new line of research by Ramp Labs on improving model compute efficiency through KV cache compression.

---

Overview on KV Cache Compression

When a model processes a prompt, it computes keys and values at each attention layer. The inference server saves these tensors in the KV cache so the model can reuse them as it generates each new token. This cache saves computation, but it consumes GPU memory and memory bandwidth. As context grows, the server has more state to store and the KV cache often grows beyond the size of the model weights themselves.

KV cache compression attempts to find a more efficient representation of the state that a model actually needs. If we can represent the same context with a smaller cache while preserving task quality, the server can fit more concurrent requests in memory and spend less bandwidth on attention. Those efficiency gains make inference cheaper and faster, reducing the cost to serve both input and output tokens.

Our research expands on two of the most promising methods in the research literature for KV cache compression:

• KVzip is a fast compression algorithm that uses context reconstruction prompts to score cache entries, keep the top N, and evict the rest. KVzip tops the KVPress leaderboards in KV cache compression performance.

• Attention Matching implements a more computationally intensive compression algorithm that constructs a smaller cache which approximates the original cache's attention outputs and its share of attention relative to other tokens.

On top of new algorithmic improvements, we’ve scaled up these methods from their original small dense attention model implementations (e.g., Qwen3-8B) to work on large sparse attention frontier models, starting with GLM 5.3.

---

Why Sparse Attention Complicates KV Cache Compression

The goal behind our KV cache compression research was to discover improvements that could be utilized by the frontier model architectures being served in production today. The implementations of KVzip and Attention Matching both assume dense attention, where every query can score every eligible cached position and compression can operate on separate key-value heads.

Modern architectures break these assumptions. In GLM 5.3, not all tokens in the model context are used for attention. Instead, GLM 5.3 relies on DeepSeek Sparse Attention (DSA) which uses a learned indexer to select a maximum of 2,048 token positions for each query to read. It then reuses the selected token position across layers (a technique called IndexShare) within 21 layer groups in the model. These tokens are then attended to using Multi-head Latent Attention (MLA), which uses a shared latent to build each head’s keys and values.

This creates several new problems that require novel adaptations for KVzip and Attention Matching:

• Different heads weight token positions differently, exhibiting head specialization on what to attend to. KVzip and Attention Matching work best when allowed to select different token positions for each head and layer but we must choose one retained token set for every head and layer in an IndexShare group. A naive approach like taking the largest score in an IndexShare group can let a few high scoring heads dominate the budget, discarding context that other heads need.

• Sparse attention computation only scores positions selected by the indexer. A position with no observed attention may have been excluded by routing rather than judged unimportant. DSA is trained to select important token positions relevant for the current query which can conflict with our goal of selecting tokens that best represent the original context.

• GLM's shared latent requires new selection methods to handle potential adverse inter-head effects when evicting a MLA entry while in standard multi-head attention, each head’s keys and values are independent from one another.

---

Cache Compression through Eviction using KVzip

Adaptation to GLM 5.3

KVzip estimates importance by asking the model to reconstruct its context. We teacher-force the model and prefill it on the exact correct response by concatenating a copy of the context as an assistant message and measure which positions in the first copy receive attention.

For model context X, the temporary scoring request is:

[未转换内容，参见原始 JSON]

The second copy supplies query vectors used to generate attention probabilities for scoring. We use dense attention for this scoring pass so every position can receive a score. For each layer I, head h, and original position t, we record the largest dense attention probability that position receives:

[未转换内容，参见原始 JSON]

Here a is the attention probability assigned to original position t by reconstruction query q, in layer l and head h. The maximum gives each position credit if it matters strongly to any reconstruction query. We compute these scores with dense attention through all 78 layers so every eligible position can be considered.

Our GLM 5.3 implementation of KVzip generates a shared mask that prevents evicted positions from being selected by the DSA indexer. Let G contain all layer-head pairs in one IndexShare group, and let B be its retained token budget. The raw maximum selector ranks each position by its largest score anywhere in G and selects the top B positions to retain. By keeping positions with the highest attention scores, we aim to efficiently preserve the attention patterns of the original uncompressed context.

[未转换内容，参见原始 JSON]

Results

We tested the raw maximum selector first on ComplexMCP's 29,807-token prefix containing 314 tool definitions on a set of 47 tasks in the benchmark across 3 runs.

The raw maximum baseline shows a gradual tradeoff between compression and task accuracy. Removing 10% of the cached prefix reduced mean accuracy by only 1.4 percentage points but halving the cache produced a larger 9.9 percentage point drop, suggesting that more aggressive eviction increasingly removes information needed for these tasks.

Alternative KVzip Selection Strategies

We next explored different selection strategies that addressed the shortcomings of the raw maximum selector. In raw maximum, a few high scoring heads can dominate selection and retain little of another head’s useful context. We tested whether allocating the same shared budget more evenly would preserve more of the reconstruction signal. For each layer-head pair, we divide its score vector by the sum over all prefix positions to obtain normalized scores p, reducing the influence of differences in total score magnitude between heads. We create a new selector, normalized maximum, which ranks each position by its maximum p within the IndexShare group.

[未转换内容，参见原始 JSON]

We also tried another selector, log coverage, which makes selection depend on the positions already retained. For each layer-head pair, define coverage cov as the sum of its normalized scores over the retained set S. The selector greedily adds the position that most increases the sum of log coverage across the group. The logarithm gives a larger reward for improving a poorly covered head than an already well-covered head. We use ε = 10⁻⁶ so the objective is defined even at zero coverage. Protected initial positions are included in S and count toward the same budget B.

[未转换内容，参见原始 JSON]

We compared the two selectors against raw maximum on the same saved reconstruction scores at 30% compression. A layer-head pair’s score coverage is the fraction of its total reconstruction score that survives eviction. This acts as a scalar approximation of how much the compressed context approximates the uncompressed context. Mean score coverage averages that fraction over all layer-head pairs (L layers × H heads). Minimum head coverage is the lowest fraction among all pairs. In the equations below, g(l) is the IndexShare group containing layer l, and S_g(l) is that group’s retained set. The minimum helps reveal a head whose useful context the shared mask largely discarded, which an average can hide. Selection time measures only mask selection and excludes the shared 199.775 second score capture pass.

[未转换内容，参见原始 JSON]

Normalized maximum and log coverage showed coverage gains over raw maximum with noticeable improvement in minimum head coverage. Normalized maximum nearly matched log coverage’s mean at about 1% of its selection time though score capture still dominated the fast selectors’ preparation cost.

We then tested whether improvements in mean score coverage and minimum head coverage translated into better actual task performance. We evaluated all three selectors on the same 47 ComplexMCP tasks, with three runs per selector at a fixed 30% compression, measuring strict task accuracy and tool call counts.

Normalized maximum achieved the highest mean strict accuracy at 60.3%, outperforming raw maximum by 5.0 percentage points and log coverage by 4.3 percentage points. It also achieved the highest minimum head coverage at 69.6%, despite having lower mean coverage than log coverage. This suggests that protecting poorly covered heads may matter more for task performance than maximizing average reconstruction score coverage: a strong average can hide substantial losses for individual heads.

---

Fitting a Smaller Cache with Attention Matching

Adaptation to GLM 5.3

Changing KVzip’s selector changes which entries survive, but leaves their keys and values untouched. Attention Matching takes the next step: after selecting a smaller set of keys, it fits new attention biases and values so the retained entries can account for contributions from discarded tokens.

For this fitting step, we adapt the paper’s two-stage regression procedure in Fast KV Compaction via Attention Matching, Section 3.2. First, it fits a weight for each retained key to match the original context’s attention mass, preserving how strongly the context contributes relative to later tokens:

[未转换内容，参见原始 JSON]

Here A contains the retained keys’ exponentiated query-key scores, m is the original attention mass, and β supplies the attention biases. A small positive floor ε avoids taking the log of zero.

With those biases fixed, the second regression fits new values to reproduce the original attention output:

[未转换内容，参见原始 JSON]

Here X contains the compact cache’s normalized attention weights and Y contains the original outputs, both evaluated on the fitting queries. Together, these fits aim to approximate the same attention patterns produced by the larger uncompressed context in a smaller KV cache. To apply these regressions to GLM 5.3, each layer must share one set of 576-dimensional latent keys and 512-dimensional latent values across all 64 query heads. We fit a separate coefficient w for each head h and retained key j. Its logarithm becomes an attention bias that scales the key’s contribution before normalization:

[未转换内容，参见原始 JSON]

Here z is the scaled query-key score for the current query. A coefficient of two doubles that key’s unnormalized contribution for that head. Different heads can therefore assign different importance to the same retained entry. With these biases fixed, we fit the shared values jointly across heads, preserving a single physical KV cache.

Compact Cache Construction

To build a compact cache for a latent cache using Attention Matching, we follow three steps at each layer: choose which keys to retain, then fit their weights and values.

1. Select a shared set of keys. We implement two key selection methods.

• Highest-attention key selection (HAK) ranks original keys by their root mean square attention score over the fitting queries and retains the highest scoring keys to fill a budget of size *B *:

• Orthogonal matching pursuit (OMP) builds the set incrementally. It chooses a key that helps explain the attention mass the current set misses, then refits the selected keys’ weights together:

2. Fit the retained entries. For the selected keys, we fit per-head coefficients to reproduce the original article’s attention mass. Their logarithms become additive attention biases. We then fit shared latent values to reproduce the article’s attention output. The keys remain selected original keys, but their new weights and values can account for contributions from discarded entries.

3. Fit with on-policy queries. Compressing earlier layers changes the residual stream passed to later layers, so their queries may differ from those produced by the uncompressed model. To account for this shift, we construct the cache sequentially as specified in Fast KV Compaction via Attention Matching, Section 3.1. Before fitting each layer, we rerun the fitting sequences with all earlier layers already compressed and capture the current layer’s queries. We then use these on-policy queries to select and fit its retained entries, reducing the mismatch between cache construction and inference.

Results

We tested this approach on QuALITY, a benchmark that evaluates QA performance against a long article context. We compressed each article's KV cache, the model's stored representation of its source text, while keeping the chat template, questions, and answer choices uncompressed.

On this five article slice across 4 runs, Attention Matching with HAK reached 73.7% accuracy on average at 95% compression, compared with 67.8% for the KVzip baseline. OMP reached 77.8% at 95%, 81.8% at 90%, and 83.9% at 80%. The HAK vs. OMP difference at 95% is 4 questions out of 90 on average, but OMP selection takes substantially longer.

Fitting improves substantially over eviction at the same budget but none of these compressed conditions matched the uncompressed context accuracy of 91.8%.

As an exercise to measure the potential ceiling from an optimal Attention Matching process, we ran Attention Matching against an oracle query where the query is a variation of the exact question that is being tested against in the QuALITY eval.

We see accuracy improvements across the board with these focused task specific queries, showing the power of query guidance for improving usability of compressed KV caches.

---

Embracing Sparsity and the DSA Indexer

Our Attention Matching adaptations so far have relied on using attention scores. We wanted to exploit the native sparsity in GLM 5.3 more directly and use the model’s own mechanism for selecting relevant context, DSA’s learned indexer. The indexer already predicts which token positions matter for each query and we hypothesized that its scores could also help identify which entries to retain in a compressed cache.

We tested this hypothesis by replacing attention scores with indexer scores for HAK and OMP key selection. For each query, we normalized the scores across all available positions, then used the scores for article positions to select the retained keys. We evaluated this approach on the same five articles and 90 questions from QuALITY.

These results suggest that the indexer provides useful information about which entries to retain and merits further exploration of how to best integrate indexer scores into these KV cache compression algorithms.

---

What We Learned

Our experiments show that KVzip and Attention Matching can be scaled up to large sparse frontier models, retaining over 90% of the accuracy of the uncompressed baseline on QuALITY while using a KV cache that is 80% smaller in size.

For KVzip, normalizing each head’s scores before selecting a shared set of entries improved mean ComplexMCP accuracy by 5.0 percentage points at 30% compression. Normalized maximum achieved the highest minimum head coverage and the best observed accuracy, even though log coverage preserved more of the reconstruction score on average. This points to an important consideration for shared-cache compression: a high average can conceal substantial losses for individual heads.

For Attention Matching, we adapted the fitting procedure to GLM 5.3’s shared latent cache, combining head-specific attention biases with shared values fitted against the original sparse attention. On our five article, 90 question QuALITY evaluation, OMP selection reached 83.9% accuracy at 80% compression, compared with 91.8% for full context. At 95% compression, it reached 77.8%, versus 67.8% for eviction alone. Fitting the retained entries recovered behavior that selection alone could not preserve, although full context remained more accurate.

Together, these results motivate more research into better selection and fitting methods and more optimal queries to use for scoring cache positions and guiding cache eviction. We’ll share more on these topics as well as our other lines of KV cache research in future posts.

---

Want to keep up with our next AI experiments? Subscribe here and follow us on @RampLabs. We’re also hiring across roles at Ramp.

---

References

1. Ramp. Ramp AI Spend Index. Accessed September 27, 2026.

2. Jang-Hyun Kim et al. (2025). KVzip: Query-Agnostic KV Cache Compression with Context Reconstruction. arXiv:2505.23416.

3. NVIDIA. KVPress. Official repository and KVPress leaderboard. Accessed September 27, 2026.

4. Adam Zweiger, Xinghong Fu, Han Guo, and Yoon Kim. (2026). Fast KV Compaction via Attention Matching. arXiv:2602.16284.

5. Yuanyang Li et al. (2026). ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox. arXiv:2605.10787.

6. Richard Yuanzhe Pang et al. (2022). QuALITY: Question Answering with Long Input Texts, Yes! NAACL-HLT 2022, pp. 5336–5358. Benchmark and leaderboard.