Tagged "long-context-inference"

5 articles tagged long-context-inference, 28 March 2026 to 15 September 2026. Newest first.

• Ollama v0.34.1 releases with MLX improvements and memory optimizations 15 September 2026

  The latest Ollama release brings MLX runner enhancements including prefix cache eviction, improved system memory management, and higher token repeat limits for more stable inference.

• LLM Memory Doesn't Only Get Written Wrong, It Goes Wrong Later 4 August 2026

  Research on how LLM memory degrades and becomes corrupted over time during inference. Understanding memory behavior is critical for reliable local deployment.

• Exploiting Sparsity for Long Context Inference: Million Token on Commodity GPUs 10 July 2026

  A new technique enables million-token context windows on standard consumer GPUs by leveraging sparsity optimizations. This breakthrough makes long-context LLM inference practical and affordable for self-hosted deployments.

• Gemma 4 31B vs Qwen 3.5 27B: Comprehensive Long Context Benchmark 11 April 2026

  Community benchmark comparing Gemma 4 31B and Qwen 3.5 27B for long context workloads on 24GB VRAM, establishing these as the top local models for mid-range GPU setups.

• TurboQuant KV Cache Compression Achieves 22.8% Faster Decoding at 32K Context 28 March 2026

  Google's TurboQuant compression method has been successfully integrated into llama.cpp, enabling 4.6x KV cache compression and 22.8% decode speedup at 32K context length by skipping 90% of dequantization work. This breakthrough makes long-context inference practical on consumer hardware like MacBook Air M4.