Tagged "kv-cache-quantization"
6 articles tagged kv-cache-quantization, 28 February 2026 to 14 October 2026. Newest first.
-
NVIDIA AutoQuant KV Reduces Memory via Automatic Mixed-Precision KV Cache Quantisation
NVIDIA introduces AutoQuant KV, enabling automatic mixed-precision quantisation of KV caches to reduce memory requirements and improve inference efficiency for local LLM deployments.
-
vLLM v0.31.0 Optimizes DeepSeek-V4.1-Flash with FlashMLA and NVFP4 Compression
vLLM v0.31.0 brings major inference optimizations including FlashMLA mega attention support for DeepSeek-V4.1-Flash and NVFP4 compressed KV cache as SM100 default. This release from 717 commits by 307 contributors significantly improves throughput and memory efficiency for production local inference.
-
llama.cpp b10256 – SYCL SDPA Extended to Quantized KV Caches
Major optimization extending Intel SYCL oneDNN scaled dot-product attention to support quantized key-value caches, significantly reducing memory overhead on Intel hardware.
-
Mixed KV Cache Quantization: Performance Risks and Pitfalls
A technical deep-dive warning against mixed-precision KV cache quantization, revealing accuracy degradation that contradicts common optimization assumptions.
-
Community Converges on Optimal KV Cache Quantization Strategies for Qwen 3.5 Models
The local LLM community is establishing practical guidelines for KV cache quantization with Qwen 3.5, balancing memory savings against accuracy loss to optimize inference on consumer hardware.
-
Qwen3.5-35B RTX 5080 Experiments Confirm KV q8_0 as Free Lunch, Q4_K_M Remains Optimal
Follow-up benchmarking of Qwen3.5-35B-A3B on RTX 5080 16GB validates community-requested configurations, achieving 74.7 tokens/second and confirming KV cache quantisation strategies.