Tagged "speculative-decoding"

37 articles tagged speculative-decoding, 28 February 2026 to 18 October 2026. Newest first.

• Ollama v0.40.7 Release: OpenAI API Improvements and MLX Enhancements 18 October 2026

  Latest Ollama release adds OpenAI-compatible API features for non-thinking models and improves MLX support, enhancing compatibility and performance for local inference deployments.

• llama.cpp v0.6.0 Introduces Extended Batch API and Decision Model Support 6 October 2026

  llama.cpp v0.6.0 adds a new llama_batch_ext API supporting mixed token/embedding inputs, introduces support for decision models like Clef, and includes GLM-5.3-Flash hybrid model support with MTP speculative decoding. These changes expand the framework's capability for complex inference patterns.

• vLLM v0.31.0: DeepSeek-V4.1-Flash with FlashMLA Mega Attention and Sparse MQA Logits 5 October 2026

  vLLM releases v0.31.0 with major performance optimizations for DeepSeek-V4.1-Flash including FlashMLA mega attention with NVFP4 compressed KV cache and sparse MQA logits, contributed by 307 contributors across 717 commits.

• Forlinx Launches 20-TOPS M.2 AI Accelerator With PCIe Cascading Support 29 September 2026

  Forlinx announces a new M.2-form-factor AI accelerator offering 20 TOPS of inference performance with support for PCIe cascading, enabling scalable local LLM inference on edge devices. The compact form factor and cascading capability make it suitable for heterogeneous edge computing deployments.

• Faster Prompt Lookup Drafting in llama.cpp 28 September 2026

  A new optimization technique for prompt lookup drafting has been implemented in llama.cpp, significantly improving inference speed for local LLM deployments. This speculative decoding method accelerates token generation without sacrificing quality.

• 42x Faster Prompt Lookup Drafting in llama.cpp 27 September 2026

  A new optimization in llama.cpp achieves 42x speedup for prompt lookup drafting, significantly improving inference performance for local LLM deployment. This speculative decoding technique dramatically reduces time-to-first-token and overall generation latency.

• Oh My Pi Adds Custom Model Support via vLLM, Llama.cpp, and SGLang 26 September 2026

  A new guide demonstrates running custom quantized models on Raspberry Pi using multiple inference engines including vLLM, Llama.cpp, and SGLang. This enables practical multi-engine inference workflows on edge devices with detailed configuration examples.

• Speculative Decoding in vLLM on AMD GPUs 7 September 2026

  vLLM now supports speculative decoding on AMD GPUs, enabling significant inference speed improvements for local LLM deployment on AMD hardware.

• DSpark Speculative Decoding: Speeding Up LLM Inference 1 September 2026

  New speculative decoding technique accelerates LLM inference by predicting and validating multiple tokens ahead, reducing latency in local deployment scenarios.

• vLLM Becomes Production Infrastructure at PyTorch Conference 2026 30 August 2026

  vLLM elevated to production status at PyTorch Conference, signaling maturity of the inference engine for scaling local LLM deployments from single-device to multi-GPU setups.

• How to Run Qwen3.8-27B on a Single 16GB Card 29 August 2026

  Practical guide demonstrating techniques to fit the 27-billion parameter Qwen3.8 model within 16GB VRAM constraints using llama.cpp, quantization, and RTX 3080 optimizations.

• vLLM v0.28.0 Features Major Kimi-K3 Optimization and Decode Context Parallel Support 27 August 2026

  vLLM 0.28.0 introduces Decode Context Parallel (DCP) support and optimized kernels for Kimi-K3, alongside improvements for 270+ contributors. The release enables faster multi-sequence inference on both datacenter and edge hardware.

• Liquid AI Releases LFM2.5-DSpark Draft Models with 3.18x Faster Decoding 21 August 2026

  Liquid AI introduces speculative decoding models that achieve up to 3.18x faster inference without changing model outputs, significantly improving local LLM performance.

• HackerNoon Compares 7 Best Self-Hosted Inference Servers for Open-Source Models 16 August 2026

  A comprehensive 2026 comparison of leading self-hosted inference servers evaluates deployment options for running open-source models locally, covering performance, ease of use, and feature parity across major frameworks.

• 7 Best Self-Hosted Inference Servers for Open-Source Models Compared (2026) 15 August 2026

  Comprehensive comparison of leading self-hosted inference server solutions, evaluating performance, features, and deployment characteristics for local LLM inference.

• Ollama 0.32.10: 7-8% Prefill Speed Gains on NVFP4 Models 14 August 2026

  Ollama 0.32.10 delivers significant prefill performance improvements for NVFP4 quantized models through kernel fusion optimizations, alongside updated default repeat penalty settings for improved speculative decoding.

• vLLM v0.27.0 – Kimi K3 Support and 561 Commits from 242 Contributors 11 August 2026

  vLLM releases v0.27.0 with comprehensive Kimi K3 model support including core kernels, Python and Rust frontends, and optimized attention mechanisms. The release represents major performance and compatibility improvements across serving infrastructure.

• Ollama v0.32.6: Faster Apple GPU Inference with Speculative Decoding 7 August 2026

  Ollama releases v0.32.6 with significant performance improvements for Apple Silicon users, including automatic speculative decoding via MLX engine's MTP head and improved OpenAI-compatible streaming format.

• Theoretical Bottlenecks for Scaling LLM Inference to Achieve Higher Token per Second 2 July 2026

  A technical discussion exploring the fundamental performance limits and bottlenecks when scaling local LLM inference throughput. This analysis helps practitioners understand optimization trade-offs and realistic performance ceilings.

• NVIDIA DFlash Block Diffusion Accelerates Autoregressive LLM Inference 25 June 2026

  NVIDIA's new DFlash block diffusion technique promises to significantly speed up inference for autoregressive language models. The optimization targets the memory and compute bottlenecks that limit throughput in local LLM deployments.

• Boost Inference Performance up to 15x on NVIDIA Blackwell Using DFlash Speculative Decoding 24 June 2026

  NVIDIA introduces DFlash speculative decoding technique achieving up to 15x inference speedup on Blackwell GPUs, a major breakthrough for accelerating local LLM deployments on enterprise hardware.

• Prefill Once, Fan Out: KV Snapshot Sharing for Multi-Agent LLM Pipelines 10 June 2026

  Towards Data Science published research on KV snapshot sharing optimization that enables efficient multi-agent LLM pipelines by reusing computed key-value caches across multiple agents. This technique significantly reduces compute requirements for local deployment scenarios.

• Meet EAGLE 3.1: The Speculative Decoding Algorithm That Fixes Attention Drift in LLM Inference 27 May 2026

  EAGLE 3.1 introduces an improved speculative decoding approach that addresses attention drift, significantly improving inference speed and efficiency for local LLM deployment.

• DFlash Speculative Decoding Delivers 8.5x Speed Improvement for LLM Inference 11 May 2026

  A new speculative decoding technique achieves dramatic speedups in local LLM inference without sacrificing output quality. This optimization is particularly impactful for latency-sensitive applications and resource-constrained deployments.

• Google Releases Gemma 4 Multi-Token Prediction Drafters To Accelerate AI Inference 8 May 2026

  Google has released new multi-token prediction drafters for Gemma 4, providing significant inference acceleration capabilities for local LLM deployment. This optimization technique enables faster token generation while maintaining output quality.

• Google Accelerates Gemma 4 Inference Speed 3x With Multi-Token Prediction Drafters 6 May 2026

  Google announced significant performance improvements for Gemma 4 through multi-token prediction drafters, achieving 3x faster inference. This optimization technique is directly applicable to local LLM deployments and represents a major breakthrough in edge inference efficiency.

• Supercharging LLM Inference on Google TPUs: Achieving 3X Speedups With Diffusion-Style Speculative Decoding 5 May 2026

  Google researchers have demonstrated 3x inference speedups on TPUs using diffusion-style speculative decoding, a novel optimization technique that could influence local inference strategies. The breakthrough shows how advanced decoding methods can dramatically reduce latency on specialized hardware.

• Prefill Is Compute-Bound, Decode Is Memory-Bound: Optimizing GPU Utilization for LLM Inference 16 April 2026

  A deep dive into why GPUs shouldn't handle both prefill and decode phases equally, and how understanding this fundamental bottleneck can dramatically improve local LLM inference performance.

• DFlash Doubles Token Generation Speed of Qwen3.5 27B on Mac M5 Max 15 April 2026

  New DFlash support in oMLX 0.3.5 RC1 achieves 2x speedup for Qwen3.5 27B inference on Apple Silicon, reaching 22 T/S from 9 T/S using speculative decoding with draft models.

• Speculative Decoding Achieves 29% Speed Boost for Gemma-4 31B 13 April 2026

  Benchmarks show speculative decoding with Gemma-4 E2B draft model delivers 29% average throughput improvement and 50% gains on code tasks. This practical optimization technique significantly accelerates local inference on consumer GPUs.

• DFlash Speculative Decoding Achieves 3.3x Speedup on Apple Silicon 12 April 2026

  A native MLX implementation of DFlash speculative decoding reaches 85 tokens/second on Qwen 3.5-9B running on Apple M5 Max, delivering a 3.3x performance boost through parallel draft token generation and single-pass verification.

• Speculative Decoding Made My Local LLM Actually Usable 9 April 2026

  A practitioner shares how implementing speculative decoding techniques dramatically improved inference speed on local LLM deployments, making previously unusable models practical for daily use.

• I Replaced My Local LLM With a Model Half Its Size and Got Better Results — and It Wasn't About the Parameters 9 April 2026

  A detailed account of how switching to a smaller, better-optimized model outperformed a larger predecessor on local hardware, challenging assumptions about model scaling and practical performance.

• Qwen 3.5 27B Achieves 1.1M Tokens/Second on B200 GPUs with Optimized vLLM Config 27 March 2026

  A developer optimized Qwen 3.5 27B to reach 1.1 million tokens per second on 96 B200 GPUs using vLLM, with detailed configurations and all settings published on GitHub. Key optimizations included distributed parallelism, reduced context windows, FP8 KV cache, and speculative decoding.

• P-EAGLE: Faster LLM Inference with Parallel Speculative Decoding in vLLM 14 March 2026

  AWS introduces P-EAGLE, a parallel speculative decoding technique integrated into vLLM that significantly accelerates LLM inference speed. This advancement is crucial for practitioners deploying local LLMs who need to optimize throughput and reduce latency.

• Qwen 3.5 27B on Dual RTX 3090s: 170K Context Holds, 100+ Tokens/s Claim Disputed 2 March 2026

  A widely shared r/LocalLLaMA video reported Qwen 3.5 27B running at 100+ tokens/second decode with a 170K context window on dual RTX 3090s. The context claim holds and is in fact understated — 262K fits. The decode figure is contradicted by independent benchmarks measuring 41.4 t/s on the same model and hardware, and the original video has never been independently verified.

• Qwen 3.5-27B Demonstrates Exceptional Performance with Thoughtful Prompt Engineering 28 February 2026

  Users report that Qwen 3.5-27B significantly exceeds expected performance for its size when paired with effective prompting strategies, suggesting prompt engineering can bridge the capability gap between model sizes.