Tagged "gpu-optimization"
20 articles tagged gpu-optimization, 28 March 2026 to 11 October 2026. Newest first.
-
ML Drift: Google's Open-Source On-Device GPU Inference Engine
Google releases ML Drift, a new open-source GPU inference engine optimized for on-device AI/ML workloads with focus on edge deployment scenarios.
-
Llama.cpp Optimizes Kernel Execution with RMS_NORM and SCALE Fusion
The latest llama.cpp release fuses RMS_NORM and SCALE operations into a single kernel, eliminating 96 extra kernel launches per batch on large models like Qwen3.8-27B. This optimization reduces computational overhead without sacrificing accuracy.
-
vLLM v0.29.0 Advances with Model Runner V2 as Default
vLLM's latest release makes Model Runner V2 the default for all models, featuring CUDA graph memory profiling and improved performance across deployment scenarios.
-
llama.cpp Build 10605: Mamba2 GEMM Optimization Improves State-Space Model Performance
The latest llama.cpp release optimizes Mamba2 models by flattening input/output projections to dispatch GEMM operations instead of GEMV, delivering better GPU utilization and inference speed for state-space architectures.
-
vLLM's Disaggregated Serving Cuts GPU Interference, Delivering 2.5x Higher Goodput
vLLM introduces disaggregated serving architecture that significantly reduces GPU memory interference, achieving 2.5x improvement in goodput on the same hardware. This breakthrough enables more efficient batch processing and higher throughput for local and self-hosted LLM deployments.
-
Llama.cpp Fixes Metal NORM Operations for Apple Silicon
Llama.cpp B10321 resolves critical issues with NORM and RMS_NORM operations on Apple Silicon, fixing threadgroup synchronization for row lengths that don't align with SIMD group boundaries. This ensures reliable inference on M-series chips.
-
GPU Half-Idle: The Hundred-Billion-Dollar Race to Squeeze 10x Efficiency from Silicon
An analysis of the hardware and software optimization challenge driving the race for inference efficiency, directly impacting the feasibility of local model deployment.
-
AI Inference is Rewriting the GPU Buying Playbook
A comprehensive analysis of how the emergence of local AI inference is fundamentally changing GPU purchasing decisions and hardware optimization priorities.
-
Critical GPU Memory Leak Vulnerability Discovered in vLLM
A severe security vulnerability (CVE-2026-53923) in vLLM allows attackers to leak GPU memory through a 32-bit integer overflow, potentially exposing sensitive data from neighboring processes during local inference.
-
NVIDIA Levels Up Local AI Agents Across RTX PCs and DGX Spark
NVIDIA introduces RTX Spark, enabling local AI agent deployment on consumer RTX PCs and enterprise DGX systems. Eight major PC brands commit to shipping RTX Spark-powered AI agent laptops in fall 2026.
-
Intel llm-scaler-vllm 1.4 Released With Updated Components and Arc Pro B70 Support
Intel releases version 1.4 of its llm-scaler-vllm toolkit with improved components and support for Arc Pro B70 GPUs, enabling optimized local LLM inference on Intel hardware.
-
Running a Serious AI Model on a Consumer GPU Just Got Easier and That Matters More Than the Benchmark
Recent advances in optimization techniques and frameworks have made it significantly easier to run production-quality large language models on consumer-grade GPUs, democratizing access to capable local AI inference. Performance improvements go beyond raw speed gains to include better memory efficiency and developer experience.
-
GPU Passthrough to LXCs in Proxmox Simplifies Local Inference Infrastructure
System administrators discover that GPU passthrough to Linux containers in Proxmox offers simpler and more efficient deployment for local LLM inference compared to traditional virtual machines. This reduces operational complexity for self-hosted inference setups.
-
MiniMax M2.7 Advances Scalable Agentic Workflows on NVIDIA Platforms for Complex AI Applications
MiniMax releases M2.7, optimized for NVIDIA hardware platforms to support complex agentic workflows at scale. The model demonstrates improved performance and efficiency for self-hosted deployment scenarios requiring advanced reasoning capabilities.
-
Qwen 3.5 122B Achieves 198 Tokens/sec on Dual RTX PRO 6000 Blackwell GPUs
A detailed optimization case study demonstrates running Qwen 3.5 122B at impressive inference speeds on a budget dual-GPU Blackwell setup. The community shares verified benchmarks with full methodology and reproducible results for large-scale local deployment.
-
TurboQuant-Optimized llama.cpp Fork Delivers GFX906 GPU Acceleration
Community developer releases optimized llama.cpp fork featuring TurboQuant quantization and specialized GFX906 GPU optimizations with Gemma 4 architecture support coming soon.
-
NVIDIA Accelerates Gemma 4 for Local Agentic AI on RTX GPUs
NVIDIA provides day-one optimizations for Google's Gemma 4 models across its RTX GPU lineup, enabling accelerated local inference for agentic AI workflows on consumer and enterprise graphics cards.
-
AMD Provides Day 0 Support for Gemma 4 on Ryzen AI Processors and GPUs
AMD announces immediate optimizations for Gemma 4 across its Ryzen AI and RDNA GPU lineup, enabling accelerated local inference on AMD-based laptops, desktops, and edge devices.
-
TurboQuant Enables Qwen 3.5-27B on 16GB Consumer GPUs
Advanced quantization technique TurboQuant achieves near-Q4_0 quality at 10% smaller size, allowing high-performance models to fit on consumer-grade graphics cards.
-
GPU Passthrough to LXCs in Proxmox Simplifies Local LLM Deployment
GPU passthrough to Linux containers in Proxmox offers superior performance and simplicity compared to virtual machines for running local LLMs, enabling efficient on-device inference without virtualization overhead.