Measured on Apple Silicon.
Verified across 12 models.

Reproduce: pytest veloxquant_mlx/tests/metal/test_h2o_evict.py veloxquant_mlx/tests/metal/test_keyformer_evict.py -k benchmark -s. Both eviction paths replace a per-token Python score-and-compact loop with a single fused dispatch.

Medians of interleaved samples over a rotating input pool. The win grows with block size — these kernels replace several full-size MLX intermediates with one pass, so they scale with bytes moved. Prefill (one large group-aligned flush) is where this matters; decode flushes only group_size tokens at a time. Reproduce: pytest veloxquant_mlx/tests/metal/test_kivi_quant.py -k benchmark -s.

GB/s is nominal traffic (input read once + packed streams written), not a hardware counter; the MLX baseline additionally charges the two full-size uint8 index intermediates the fused kernel never materializes. Reproduce: pytest veloxquant_mlx/tests/metal/test_rvq_quant_pack.py -k benchmark -s.

Synchronized median timings from 20 measured repetitions after five warmups. This is a synthetic complete-cache update, not end-to-end generation throughput. Selection-only Metal medians were 0.246 / 0.249 / 0.783 ms versus MLX 0.390 / 0.410 / 1.140 ms at N=512 / 2,048 / 8,192. Raw data: snapkv_selection_after_fused.json.

One warmup plus three measured trials per route in the post-BF16-fix follow-up. “Exact” means zero measured first-prefill logit error against the SnapKV reference and identical greedy tokens. Metal is mixed at model level and remains opt-in. Full methodology and raw runs: SNAPKV_8B_9B_MODEL_RESULTS.md.

Ranges span three run medians, each with 10 warmups and 100 alternating-order trials. Synchronized host-wall timing includes dispatch and evaluates all outputs. Resident random finite scores; exact output parity passed. Scoring, append, and model execution are excluded. These ranges are not confidence intervals.

One warmup and three measured trials per backend, fresh caches, alternating backend order, and 16 greedy decode steps. Timings include Python dispatch and decode token reads. All accelerated runs matched reference final-prefill logits exactly and generated identical tokens. Metal invocation counts confirmed execution. This single repeated-prompt workload establishes backend parity, not general model quality or long-context mask correctness. Qwen Metal prefill was about 1% slower; GLM Metal decode was about 1% slower.

Reproduction, exact model revisions, validation, and limitations: PyramidKV findings. Raw latest runs: Qwen JSON · GLM JSON.

Bandwidth utilization scales with threadgroup count, not with S_kv or bytes moved per threadgroup — direct evidence this kernel is occupancy/dispatch-bound, not memory- or compute-bound, at the shapes that matter for real single-token decode.

Real forward passes on real hardware (Apple M4, prompt_len=256, 30 decode steps) — not an isolated kernel call. Both arms decode from the identical quantized KV state and were verified to produce bit-identical tokens before timing was trusted, so this measures the fused kernel's speedup over dequantize+MLX-SDPA on the same compressed state — not a general "KIVI is faster in production" claim. The test harness behind this table also has no fp16 residual window (unlike shipping KIVI), so its generation quality isn't production-representative; only the timing comparison is.

The head-packing numbers above were revised after this table's original measurement (before issue #317's occupancy fix changed the packed kernel's threadgroup-memory footprint) turned out to be superseded, not simply wrong — full re-measurement and methodology → re-measured after nsg autotune addendum.

At this shape and ratio (heads_per_kv=8), packing alone already removes most of the redundant KV re-decode unpacked pays for at every S_kv tested.

The two-pass margin over unpacked widens monotonically with S_kv at this prefill shape — the reverse of the decode-shape crossover in the Decode Occupancy section, where two-pass's advantage shrinks and flips negative past a few thousand cached tokens for the same heads_per_kv=8 ratio. The two kernels dispatch different threadgroup counts for the same S_kv (one query token vs. many), which is why the same code change lands on opposite sides of its occupancy crossover depending on whether it runs during prefill or decode. These figures characterize the prefill path only; do not read them as superseding the decode-step verdict in KV_KERNEL_ROOFLINE_FINDINGS.md.

Measurements on Llama-3.2-1B (fp16 baseline 4.050 ppl). Llama-3.2-3B closes 55% / 42% at Budgets 256 / 128.