Tagged "kv-cache-optimization"

23 articles tagged kv-cache-optimization, 14 March 2026 to 20 October 2026. Newest first.

• vLLM Optimises Disaggregated Inference with Push-Based KV Transfer 20 October 2026

  AWS and vLLM introduce push-based KV cache transfer to improve time-to-first-token and time-per-output-token for disaggregated inference deployments. This optimization technique reduces latency in multi-node local inference setups.

• 30B-Class Models Squeezed Into Phones: On-Device AI Arms Race Ignites New Memory Battleground 19 October 2026

  The industry is pushing larger 30B-parameter models onto mobile devices, creating a new competitive frontier around memory optimization and compression techniques for on-device inference.

• vLLM Architecture, Memory and Benchmarks Deep Dive 23 September 2026

  An in-depth technical analysis of vLLM's architecture, memory management, and throughput characteristics, providing concrete benchmarks and optimization strategies for local LLM inference.

• vLLM v0.30.0 Released With DeepSeek-V4.1 and Advanced Optimizations 22 September 2026

  vLLM v0.30.0 brings 762 commits including support for DeepSeek-V4.1-Flash with MXFP8 quantization and async prefetch optimizations for improved throughput on local hardware.

• Running 104GB Qwen3.8-Flash-Next on 48GB Mac with Slotstream at ~12 tok/s 2 September 2026

  A breakthrough demonstration of running a 104GB model on a 48GB Mac using adaptive KV streaming techniques, achieving practical inference speeds of ~12 tokens/second. This showcases innovative memory optimization for consumer hardware.

• Llama.cpp Fork Enables Qwen 3.8 27B with Large Contexts on 16GB VRAM 1 September 2026

  A specialized llama.cpp implementation adds adaptive KV-cache streaming to run Qwen 3.8 27B with large context windows on 16GB GPUs, demonstrating significant memory optimization advances.

• llama.cpp Optimizes DFlash Encoder with KV Cache Injection 31 August 2026

  Recent llama.cpp builds include performance improvements for DFlash models by fusing encoder operations into KV cache injection, reducing computational overhead for local inference.

• K-EXAONE 2.0 Brings 262K Context to Frontier AI 4 August 2026

  K-EXAONE 2.0 introduces a 262K token context window, significantly expanding the capabilities of frontier-class models for local deployment and extended reasoning tasks. This represents a major advancement in practical context window management.

• The KV Cache Survival Guide: Why Your GPU Runs Out of Memory with Local LLMs 2 August 2026

  Deep dive into KV cache management and practical strategies to prevent GPU out-of-memory errors when running local LLMs, a critical bottleneck for on-device inference.

• The KV Cache Survival Guide: Why Your GPU Runs Out of Memory with Local LLMs 2 August 2026

  A comprehensive guide addressing one of the most critical bottlenecks in local LLM deployment: KV cache memory consumption. Learn practical strategies to manage GPU memory constraints when running LLMs on-device.

• TriAttention Solves KV Cache Memory Bottleneck in Local LLM Inference 28 June 2026

  TriAttention presents a solution to the KV cache memory bottleneck that constrains local LLM inference speed and hardware requirements. This breakthrough addresses one of the most significant performance limitations in on-device language model deployment.

• CacheWise Optimizes KVCache Reuse for LLM Coding Agents 16 June 2026

  CacheWise improves inference efficiency by optimizing KVCache reuse in language models used for coding tasks. This memory optimization technique reduces computational overhead and latency for agent-based LLM applications.

• Prefill Once, Fan Out: KV Snapshot Sharing for Multi-Agent LLM Pipelines 10 June 2026

  Towards Data Science published research on KV snapshot sharing optimization that enables efficient multi-agent LLM pipelines by reusing computed key-value caches across multiple agents. This technique significantly reduces compute requirements for local deployment scenarios.

• Apple Unveils AFM 3 Core Advanced with 20 Billion Parameters for On-Device AI 10 June 2026

  Apple introduced the AFM 3 Core Advanced architecture at WWDC26, featuring a 20 billion parameter model optimized for on-device inference. This represents a significant milestone in local LLM deployment on consumer hardware with architectural innovations to overcome memory constraints.

• Elastic KV Cache Memory Breakthrough Enables Efficient Bursty LLM Serving and GPU Sharing 26 April 2026

  A new coding implementation on elastic KV cache memory optimization allows more efficient handling of variable-load LLM serving patterns and multi-model GPU sharing scenarios.

• Gemma 4 Support Stabilized in Llama.cpp 9 April 2026

  Major fixes for Gemma 4 models have been merged into Llama.cpp, resolving known issues and enabling stable inference. Users report successful deployments of Gemma 4 31B on Q5 quantizations without problems.

• Gemma 4 GGUF Models Updated with Critical Quantization Fixes 9 April 2026

  Unsloth has released updated Gemma 4 GGUF quantizations addressing kv-cache issues and other inference problems. New versions are available for both 26B and 31B model sizes.

• TurboQuant in Llama.cpp Achieves 6X Smaller KV Cache 6 April 2026

  A new implementation of TurboQuant in llama.cpp reduces KV cache size by 6x, significantly improving memory efficiency for local LLM inference. This breakthrough enables running larger models on resource-constrained devices.

• Gemma 4 KV Cache Memory Issues Fixed in llama.cpp 4 April 2026

  llama.cpp has released critical fixes for Gemma 4's KV cache implementation, dramatically reducing VRAM consumption and making the model practical for local deployment on consumer hardware.

• VRAM Optimization Technique Cuts Gemma 4 Memory Usage by 3x 3 April 2026

  A simple llama.cpp parameter adjustment (-np 1) significantly reduces Sliding Window Attention cache VRAM requirements for Gemma 4, enabling deployment on systems with limited GPU memory.

• TurboQuant KV Cache Compression Achieves 22.8% Faster Decoding at 32K Context 28 March 2026

  Google's TurboQuant compression method has been successfully integrated into llama.cpp, enabling 4.6x KV cache compression and 22.8% decode speedup at 32K context length by skipping 90% of dequantization work. This breakthrough makes long-context inference practical on consumer hardware like MacBook Air M4.

• LMCache Dramatically Accelerates LLM Inference on Oracle Data Science Platform 20 March 2026

  Oracle integrates LMCache, a cutting-edge prompt caching and KV cache optimization technique, into their cloud data science platform to accelerate LLM inference and reduce computational overhead.

• 3-Path Agent Memory: 8 KB Recurrent State vs. 156 MB KV Cache at 10K Tokens 14 March 2026

  A new memory architecture demonstrates significant efficiency gains for local LLM agents, reducing memory footprint from 156 MB to just 8 KB while maintaining performance at 10K token contexts. This breakthrough is critical for deploying agents on resource-constrained devices.