Validation Report: KV Cache Quantization on Apple Silicon
This page explains what VeloxQuant-MLX does, how to measure it honestly, and
how to reproduce numbers with scripts/validate_kv_memory.py.
No marketing softeners: every claim is a definition, a formula, or a number
from a committed script. After you run the harness, cite
figures/validation/<model>/results.json.
1. What the KV cache is
During autoregressive generation, each transformer layer computes key (K) and value (V) vectors for every token. The KV cache stores those tensors so each new token only computes its own K/V and attends over history.
bytes ≈ 2 × n_layers × n_kv_heads × head_dim × seq_len × 2
seq_len is tokens currently in the cache (prompt + generated).
2. What quantization does
Quantization stores fewer bits (or codebook indices) instead of fp16 so
the same RAM can hold a longer seq_len.
Eviction drops or merges tokens (different axis). Cross-layer methods share state across depth.
Integration is three lines: build caches with KVCacheConfig /
KVCacheBuilder.for_model, pass them to mlx_lm.generate as prompt_cache.
3. Where reduction happens
| Axis | What shrinks | Examples |
|---|---|---|
| Keys only | Packed key bytes | RVQ, VecInfer keys |
| Keys + values | Full KV packed bytes | RaBitQ, KIVI |
| Token count | seq_len | H2O, StreamingLLM |
| Across layers | Shared state | XQuant, MiniCache, xKV |
Honest storage behavior (default RVQ / VecInfer)
- Quantize incoming tensors.
- Dequantize back to fp16.
- Store in the parent
mlx_lmKVCache. - Report
compressed_key_bytesas packed-format accounting.
Headline ~7.5× (RVQ-1bit, d=128) and ~16× (VecInfer-1bit,
sub_dim=8) are accounting ratios unless a packed/fused path is active and
measured.
Measured example (M4 Pro, 48 GB)
From committed figures/validation/Llama-3.2-3B-Instruct-4bit/results.json
(max_tokens=64, 120 tokens in cache):
| Arm | tok/s | Peak MB | Keys fp16 (acct) | Keys compressed (acct) | Claim |
|---|---|---|---|---|---|
| fp16 | 58.2 | 1954 | n/a | n/a | 1.00× |
| RVQ-1bit | 81.1 | 1855 | 6.562 MB | 0.872 MB | 7.53× |
| VecInfer-1bit | 49.2 | 1926 | 6.562 MB | 0.410 MB | 16.00× |
claim = fp16_key_bytes / compressed_key_bytes
4. Before / after metrics
| Metric | Meaning | JSON field |
|---|---|---|
| Tokens in cache | Context length held | tokens_in_cache_max |
| Keys fp16 size | Uncompressed key accounting | fp16_key_bytes, fp16_key_mb |
| Keys compressed size | Packed key accounting | compressed_key_bytes, compressed_key_mb |
| Compression claim | fp16 / compressed | key_compression |
| MLX peak MB | Weights + activations + temps | peak_mb |
| Throughput | tok/s | throughput_tok_s |
| Preview | Quality spot-check | output_preview |
At short generation, weights dominate peak MB. Grow prefill
(--prompt-repeat) before claiming cache-driven RAM wins.
5. Method defaults
| Goal | Method |
|---|---|
| Everyday, zero calibration | turboquant_rvq b=1 |
| Max key accounting | vecinfer 1-bit (calibrate) |
| Quality at moderate rate | spectral |
| Max context in tight RAM | rabitq or eviction |
| Constant memory | streaming_llm / h2o |
6. Reproduce
PYTHONPATH=. python scripts/validate_kv_memory.py \
--model mlx-community/Llama-3.2-3B-Instruct-4bit \
--max-tokens 128
Writes figures/validation/Llama-3.2-3B-Instruct-4bit/results.json.
Also see the fuller write-up in the repo: docs/validation-report.md.
7. Formula check
RVQ (d=128, b=1): (128×2) / (ceil(128×2×1/8)+2) ≈ 7.5×.
VecInfer (d=128, d_k=8, 8-bit indices): 256 / 16 = 16×.
Matching JSON ratios confirms counters, not resident RSS.