Measured on Apple Silicon.
Verified across 12 models.
Real causal-LM generation throughput, attention correlation accuracy, and perplexity measurements on Apple Silicon hardware (M1 through M4). Zero synthetic projections — all logs committed under figures/.
Metal GPU Kernel Acceleration
Hand-written Metal Shading Language (MSL) kernels vs. pure MLX operations across sequence lengths on Apple Silicon unified memory.
Metal threadgroup caching scales with context length: While unquantized attention stalls memory bandwidth on long sequences, our fused Metal kernels achieve 6.91× speedup at 128 tokens and scale up to 14.68× speedup at 8,192 tokens.
KV-Cache Kernel Roofline: What Actually Limits Decode
A self-calibrated bandwidth roofline (not spec-sheet numbers) across all four KV-cache quantize/dequantize/fused-attend Metal kernels — two are already within a few percent of this hardware's measured bandwidth ceiling, and one never becomes bandwidth-bound at all for a different, more interesting reason.
The fused decode+attend kernel's bottleneck isn't memory bandwidth — it's GPU occupancy. At a realistic single-token decode shape it tops out at just 6.1% of peak bandwidth, even at 16,384 cached tokens, because too few threadgroups are dispatched to fill a 10-core GPU. Routing multiple concurrent requests through that same kernel — verified end-to-end on a real model — measured a real 1.50×–3.83× decode speedup that grows with batch size.
| Dispatch shape | Threadgroups | % of peak bandwidth |
|---|---|---|
| B=1, H=8, S_q=1 (realistic single request) | 8 | 6.1% |
| B=1, H=32, S_q=1 | 32 | 21.8% |
| B=4, H=32, S_q=1 | 128 | 36.7% |
Bandwidth utilization scales with threadgroup count, not with S_kv or bytes moved per threadgroup — direct evidence this kernel is occupancy/dispatch-bound, not memory- or compute-bound, at the shapes that matter for real single-token decode.
| Concurrent requests (B) | Baseline decode tok/s | Fused-kernel decode tok/s | Speedup |
|---|---|---|---|
| 1 | 17.1 | 25.5 | 1.50× |
| 4 | 25.3 | 48.1 | 1.90× |
| 16 | 34.3 | 108.1 | 3.16× |
| 32 | 39.1 | 149.9 | 3.83× |
Real forward passes on real hardware (Apple M4, prompt_len=256, 30 decode steps) — not an isolated kernel call. Both arms decode from the identical quantized KV state and were verified to produce bit-identical tokens before timing was trusted, so this measures the fused kernel's speedup over dequantize+MLX-SDPA on the same compressed state — not a general "KIVI is faster in production" claim. The test harness behind this table also has no fp16 residual window (unlike shipping KIVI), so its generation quality isn't production-representative; only the timing comparison is.
| Approach | Result | Disposition |
|---|---|---|
| GQA head-packing (share K/V decode across query heads) | 2.7×–4.7× slower | Reduces threadgroup count, which is the actual bottleneck — kept as tested code, not the default. |
| Two-pass "decode-once" (separate decode + attend kernels) | 1.1×–3.2× faster at S_kv≤2048, 0.80×–0.85× (slower) at S_kv≥3072 | Real crossover, but this project's realistic decode targets sit past it — not adopted as the default. |
| Cross-layer batched dispatch | 1.0×–4.5× faster in isolation | Structural dead end for real single-request serving — a standard transformer's residual stream makes layer L+1's attention depend on layer L's full output, so layers can't be batched together without changing the model. |
python scripts/kv_kernel_roofline_bench.pyFull methodology, self-calibration approach, and all four follow-up experiments (including the ones summarized above) → KV_KERNEL_ROOFLINE_FINDINGS.md on GitHub.
12-Model Quality Matrix (KIVI 4-Bit vs. 2-Bit)
Long-context causal generation (~430–500 tokens, well beyond the 128-token residual cache window) tested across 12 models spanning 135M to 8B parameters.
4-bit is safe across every model tested (perplexity within ~1 point of fp16). 2-bit provides an 8× compression win on 7B–8B models, but causes significant degradation under ~3B (Llama-3.2-1B perplexity triples from 22.8 → 81.3). Choose bit-width based on parameter size!
View Complete Numerical Data Table (All 12 Models)
| Model Architecture | fp16 Perplexity | KIVI 4-bit (4×) | KIVI 2-bit (8×) | Decode tok/s |
|---|---|---|---|---|
| SmolLM2-135M | 28.87 | 29.63 | 44.85 | — |
| Qwen2.5-0.5B | n/a† | 22.22 | 49.08 | 244.3 tok/s |
| Llama-3.2-1B | 22.77 | 23.27 | 81.30 | 119.7 tok/s |
| gemma-3-4B | 16.00 | 16.37 | 17.72 | — |
| Qwen3-4B | n/a† | 15.07 | 17.21 | — |
| Llama-3.2-3B | 15.47 | 15.82 | 20.93 | 46.2 tok/s |
| Mistral-7B-v0.3 | 8.01 | 7.96 | 8.05 | 22.8 tok/s |
| Qwen2.5-7B | n/a† | 9.30 | 10.75 | 20.9 tok/s |
| Llama-3.1-8B | 10.88 | 11.02 | 13.57 | 18.5 tok/s |
| Falcon3-7B | 12.69 | 12.63 | 14.44 | 16.1 tok/s |
| Qwen3-8B | n/a† | 9.92 | 10.57 | — |
| DeepSeek-V2-Lite (MoE·MLA) | n/a† | 15.34 | 17.87 | 82.5 tok/s |
Attention Scorer Correlation (Q-Filters)
Spearman rank-order correlation with genuine model attention weights. Tests whether the scorer correctly identifies which tokens are critical to retain.
100% sign-accuracy across all models tested. Q-Filters achieves roughly 2× the correlation of naive K-norm / L2Norm baselines (+0.863 vs +0.410 on Llama-3.2-3B).
Generation Perplexity & Quality Gap Closed
Percentage of the fp16 degradation gap closed by Q-Filters calibration relative to naive eviction at varying memory compression budgets.
Measurements on Llama-3.2-1B (fp16 baseline 4.050 ppl). Llama-3.2-3B closes 55% / 42% at Budgets 256 / 128.
Real KV Cache Memory Scaling (RaBitQ on Falcon3-7B)
Measured RAM footprint of the KV cache across context lengths from 256 to 4,096 tokens.
| Sequence Length | Standard fp16 Cache | RaBitQ MSE-4V Cache | Memory Saved | Compression Ratio |
|---|---|---|---|---|
| 256 tokens | 29.4 MB | 4.9 MB | 24.5 MB freed | 5.95× |
| 512 tokens | 58.7 MB | 9.9 MB | 48.8 MB freed | 5.95× |
| 1,024 tokens | 117.4 MB | 19.7 MB | 97.7 MB freed | 5.95× |
| 2,048 tokens | 234.9 MB | 39.5 MB | 195.4 MB freed | 5.95× |
| 4,096 tokens | 469.8 MB | 78.9 MB | 390.9 MB freed | 5.95× |
python -m veloxquant_mlx benchmark --all