Tagged "inference-engine"

19 articles tagged inference-engine, 12 February 2026 to 11 October 2026. Newest first.

• ML Drift: Google's Open-Source On-Device GPU Inference Engine 11 October 2026

  Google releases ML Drift, a new open-source GPU inference engine optimized for on-device AI/ML workloads with focus on edge deployment scenarios.

• ML Drift: Google's Open-Source On-Device GPU Inference Engine 10 October 2026

  Google has released ML Drift, a new open-source GPU inference engine specifically optimized for edge and on-device deployment scenarios. The framework focuses on GPU efficiency for local inference workloads.

• llama.cpp B11521: Server Infrastructure Improvements and Performance Optimisations 9 October 2026

  Latest llama.cpp release includes default port configuration changes and continued optimisations across CUDA, Metal, and other acceleration backends.

• A Wave of Narrow AI Inference Engines Is Beating vLLM and llama.cpp at Their Own Game 4 October 2026

  Specialized inference engines optimized for specific tasks are emerging as stronger competitors to general-purpose frameworks like vLLM and llama.cpp, offering superior performance for local LLM deployment.

• Q4 vs Q6 vs Q8: The Quantization Decision Framework for Local LLMs 2 August 2026

  A detailed comparison framework for choosing the right quantisation level (Q4, Q6, Q8) when running local LLMs, balancing model quality, inference speed, and memory requirements.

• How To Build Your Own LLM Runtime From Scratch 23 July 2026

  A comprehensive guide on constructing custom LLM inference runtimes, providing practitioners with deep knowledge to optimize and control local model deployment without relying on black-box frameworks.

• Developer Ditches Ollama for llama.cpp's WebUI: A Practical Comparison 11 July 2026

  An experienced practitioner switched from Ollama to llama.cpp's WebUI after preferring its control, performance, and flexibility for local model inference. The shift highlights ongoing competition between local inference frameworks and the importance of evaluating tools for specific use cases.

• Developer Switches from LM Studio to llama.cpp, Citing Performance and Simplicity 9 June 2026

  A How-To Geek article documents why developers are moving away from heavier LM Studio implementations toward the leaner llama.cpp inference engine for local LLM deployment.

• Tether AI Upgrades QVAC SDK With TurboQuant for Data Center-Sized Memory on Everyday Devices 2 June 2026

  Tether AI has released TurboQuant, a quantization advancement in their QVAC SDK that enables everyday devices to run local AI with memory efficiency comparable to data center deployments. The upgrade focuses on reducing memory requirements while maintaining inference quality.

• DwarfStar 4: Native Inference Engine Optimized for DeepSeek V4 Flash 16 May 2026

  DwarfStar 4 is a compact native inference engine specifically designed for DeepSeek V4 Flash, enabling efficient local deployment of advanced language models on resource-constrained devices.

• Hipfire: A Rust-Native AMD Inference Engine That Outperforms llama.cpp 28 April 2026

  Hipfire, a new Rust-native inference engine optimized for AMD consumer GPUs, demonstrates performance improvements over the widely-used llama.cpp framework. This breakthrough offers local LLM practitioners a faster alternative for AMD-based setups.

• DotLLM – Building an LLM Inference Engine in C# 15 April 2026

  A new LLM inference engine implementation in C# provides .NET developers with native capabilities for running language models locally. This expands the ecosystem of local inference frameworks beyond Python-dominant tooling.

• Llamafile 0.10 Released with GPU Support and Rebuilt Core 20 March 2026

  Mozilla's Llamafile, the portable single-file LLM runner, reaches version 0.10 with enhanced GPU acceleration and a completely rebuilt inference core. This update makes it easier than ever to run large language models locally without complex dependencies.

• Llama.cpp Celebrates Major Milestone: From Leak to Industry Standard 11 March 2026

  The llama.cpp project marks a significant birthday, reflecting its evolution from a hobbyist experiment running leaked models to the foundational inference engine for local LLM deployment.

• Llama.cpp Merges Automatic Parser Generator to Mainline 7 March 2026

  After months of testing, llama.cpp has merged its new automatic parser generator solution into the main codebase, building on improved Jinja templating and native parsing infrastructure. This enhancement streamlines model deployment and reduces manual configuration overhead for local inference.

• Critical: Qwen 3.5 Requires BF16 KV Cache, Not FP16 for Accurate Inference 2 March 2026

  Community member Daniel Han alerts users that Qwen 3.5 models require bfloat16 KV cache precision instead of the default float16, with perplexity measurements demonstrating the accuracy impact when using incorrect cache formats.

• I Thought I Needed a GPU to Run AI Until I Learned About These Models 21 February 2026

  A practical guide demonstrating that modern optimized models and inference engines enable effective LLM deployment on CPU-only hardware, removing a major perceived barrier to local AI.

• LayerScale Launches Inference Engine Faster Than vLLM, SGLang, and TRT-LLM 19 February 2026

  A new inference engine claims to outperform established LLM serving platforms including vLLM, SGLang, and TensorRT-LLM. This breakthrough in inference speed could significantly improve local LLM deployment efficiency.

• OpenClaw with vLLM Running for Free on AMD Developer Cloud 12 February 2026

  AMD launches free cloud access to run OpenClaw and vLLM inference workloads, providing developers with no-cost GPU resources for local LLM development.