Tagged "datacenter-gpu"
81 articles tagged datacenter-gpu, 12 February 2026 to 11 October 2026. Newest first.
-
vLLM on Vera Rubin Posts 7.8x GB200 Throughput per Chip
vLLM achieves 7.8x throughput improvements on NVIDIA's GB200 hardware through Vera Rubin optimizations, demonstrating significant performance gains for local inference clusters.
-
Nvidia's DGX Spark Gets a 64GB Model at $4,999
NVIDIA announces a more affordable 64GB variant of its DGX Spark system, enabling accessible high-performance local AI development with support for memory pooling across multiple units.
-
vLLM v0.29.0 Advances with Model Runner V2 as Default
vLLM's latest release makes Model Runner V2 the default for all models, featuring CUDA graph memory profiling and improved performance across deployment scenarios.
-
IFM Releases K2 Horizon: Six Apache 2.0 Models From 0.9B to 375B
IFM has released the K2 Horizon series with six openly-licensed models spanning 0.9B to 375B parameters, providing diverse options for local deployment across different hardware constraints.
-
Speculative Decoding in vLLM on AMD GPUs
vLLM now supports speculative decoding on AMD GPUs, enabling significant inference speed improvements for local LLM deployment on AMD hardware.
-
Lemonade 11.9 Local AI Server Released With AMD ROCm HRX Backend
Lemonade AI server reaches version 11.9 with new AMD ROCm HRX backend support, expanding local inference capabilities to AMD GPU hardware and providing an alternative to NVIDIA-focused deployment stacks.
-
vLLM Becomes Production Infrastructure at PyTorch Conference 2026
vLLM elevated to production status at PyTorch Conference, signaling maturity of the inference engine for scaling local LLM deployments from single-device to multi-GPU setups.
-
vLLM v0.28.0 Features Major Kimi-K3 Optimization and Decode Context Parallel Support
vLLM 0.28.0 introduces Decode Context Parallel (DCP) support and optimized kernels for Kimi-K3, alongside improvements for 270+ contributors. The release enables faster multi-sequence inference on both datacenter and edge hardware.
-
vLLM's Disaggregated Serving Cuts GPU Interference, Delivering 2.5x Higher Goodput
vLLM introduces disaggregated serving architecture that significantly reduces GPU memory interference, achieving 2.5x improvement in goodput on the same hardware. This breakthrough enables more efficient batch processing and higher throughput for local and self-hosted LLM deployments.
-
HackerNoon Compares 7 Best Self-Hosted Inference Servers for Open-Source Models
A comprehensive 2026 comparison of leading self-hosted inference servers evaluates deployment options for running open-source models locally, covering performance, ease of use, and feature parity across major frameworks.
-
Building Local LLM Rigs with Used Server GPUs: 32GB VRAM for €220
Practical guide to sourcing used server-grade GPUs for local LLM inference, achieving 32GB of VRAM at fraction of consumer GPU costs, making large model deployment accessible.
-
NVIDIA Enables Local Agentic AI Workflows with Meta's Muse Glimmer
NVIDIA's technical documentation and optimization work demonstrates how to effectively deploy Meta's Muse Glimmer for agentic workloads on NVIDIA GPUs, providing practical guidance for enterprise and developer deployments. The guide covers performance optimization and multi-GPU configurations.
-
Runware Demonstrates Compact 1MW AI Data Center in 20-Foot Container
Runware achieves remarkable density by fitting a 1MW AI data center into a standard 20-foot shipping container, demonstrating efficient thermal management and hardware provisioning for scalable local inference infrastructure.
-
vLLM v0.27.0rc1: Latest Release Candidate for High-Performance Inference
vLLM announces v0.27.0rc1, the latest release candidate bringing continued improvements to the popular open-source LLM serving engine optimized for local and distributed deployments.
-
DeepSeek V4 Flash Optimized for Single AMD MI300X GPU
DeepSeek V4 Flash model now runs efficiently on a single AMD MI300X accelerator, demonstrating practical local deployment of advanced models on consumer-grade AMD hardware.
-
Ask HN: How Are You Operating OSS AI Infrastructure?
Community discussion on practical approaches to running and maintaining open-source AI infrastructure. Direct insights from practitioners deploying LLMs locally.
-
AMD's MI355X Undercuts Nvidia's B300 on Cost to Run China's Kimi K3
AMD's MI355X GPU offers competitive pricing advantages over NVIDIA's B300 for running large language models, providing cost-conscious practitioners with viable alternatives for local inference hardware.
-
Building a Dual V100 AI Workstation for Local LLMs
A practical guide to constructing a high-performance local LLM inference workstation using dual NVIDIA V100 GPUs, providing both cost-effective and capable hardware for serious local deployment work.
-
Triton Control: Open-Source Control Plane for Nvidia Triton on Kubernetes
A new open-source project providing a control plane for managing Nvidia Triton Inference Server deployments on Kubernetes, streamlining multi-model serving infrastructure.
-
Can a 2.8T Model Run on a Single Node of Nvidia B300 X8?
A practical deployment analysis examining whether ultra-large trillion-parameter models can be efficiently served on a single high-end GPU node, providing real-world benchmarks for modern hardware.
-
Titan Transients and LLM Scalability
An ACM Queue article examining scalability challenges and solutions for large language models, relevant to understanding infrastructure requirements for local deployment scenarios.
-
Netflix Details Its In-House LLM Serving Platform with Triton and vLLM
Netflix has published details about its production LLM serving infrastructure, combining NVIDIA Triton and vLLM for efficient model deployment. This real-world case study demonstrates battle-tested patterns for scaling LLM inference at enterprise scale.
-
AI Inference is Rewriting the GPU Buying Playbook
A comprehensive analysis of how the emergence of local AI inference is fundamentally changing GPU purchasing decisions and hardware optimization priorities.
-
AMD Acquires FastFlowLM to Accelerate On-Device AI Inferencing
AMD's acquisition of the FastFlowLM team signals major investment in optimizing AI inference on AMD hardware, particularly for edge and local deployment scenarios.
-
Qwen 3.8 with 2.4T Parameters Going Open-Weight Soon
Alibaba announced Qwen 3.8, a massive 2.4 trillion parameter model that will be released as open-weight, significantly expanding options for self-hosted large-scale LLM deployment.
-
Host Private Local AI on NVIDIA DGX Spark Using Ollama and Open WebUI
A technical deep-dive on deploying private LLM infrastructure using NVIDIA's hardware with Ollama and Open WebUI for complete control and data privacy. Ideal for enterprises managing sensitive workloads.
-
Critical GPU Memory Leak Vulnerability Discovered in vLLM
A severe security vulnerability (CVE-2026-53923) in vLLM allows attackers to leak GPU memory through a 32-bit integer overflow, potentially exposing sensitive data from neighboring processes during local inference.
-
A Guide on How to Run Nemotron 3 Super 120B Thinking on 2 Nvidia DGX Spark
Practical deployment guide for running NVIDIA's large reasoning model (120B parameters) on a two-node DGX Spark cluster with distributed inference techniques.
-
NVIDIA DFlash Block Diffusion Accelerates Autoregressive LLM Inference
NVIDIA's new DFlash block diffusion technique promises to significantly speed up inference for autoregressive language models. The optimization targets the memory and compute bottlenecks that limit throughput in local LLM deployments.
-
Boost Inference Performance up to 15x on NVIDIA Blackwell Using DFlash Speculative Decoding
NVIDIA introduces DFlash speculative decoding technique achieving up to 15x inference speedup on Blackwell GPUs, a major breakthrough for accelerating local LLM deployments on enterprise hardware.
-
Show HN: LiveHere – AI Videos with Self-Hosted Nvidia Cosmos on H200 GPUs
A project demonstrates self-hosted video generation using Nvidia Cosmos running on H200 GPUs, showcasing practical infrastructure for local large-scale AI model deployment. This bridges the gap between consumer-grade local inference and enterprise-scale self-hosted systems.
-
DeepSeek V4 Performance Analysis: 1.6T Day 0 to Day 43 Scaling Trends
SemiAnalysis published detailed performance tracking of DeepSeek V4's 1.6T parameter model across different hardware platforms including Huawei, MI355X, and NVIDIA GPUs. The analysis reveals scaling trends and optimization patterns relevant to large model deployment on varied infrastructure.
-
NVIDIA Levels Up Local AI Agents Across RTX PCs and DGX Spark
NVIDIA introduces RTX Spark, enabling local AI agent deployment on consumer RTX PCs and enterprise DGX systems. Eight major PC brands commit to shipping RTX Spark-powered AI agent laptops in fall 2026.
-
Real-time LLM Inference on Standard GPUs: 3k tokens/s per request
A breakthrough in LLM inference optimization achieves 3,000 tokens per second on standard GPUs, significantly improving real-time inference performance for local deployments.
-
GPUs and RAM Are in Short Supply, but the Real Bottleneck for AI Is Electricians
Infrastructure analysis reveals that electrical capacity and specialized technicians are becoming the critical constraint for scaling AI inference, not hardware components themselves.
-
Alibaba Cloud Joins PyTorch Foundation as Platinum Member
Alibaba Cloud's elevation to PyTorch Foundation Platinum membership indicates major enterprise backing for the deep learning framework, with implications for distributed training and on-device optimization tooling.
-
Deploying Hermes Agent for Free on AMD Developer Cloud with Open Models and vLLM
AMD and the open-source community demonstrate practical deployment of sophisticated agents using vLLM on AMD hardware, showcasing free compute access for local AI development.
-
ROCm 7.2.3 Delivers Performance Improvements Over 7.0.0 on AMD Radeon AI PRO
Phoronix benchmarks show measurable performance gains with ROCm 7.2.3 compared to version 7.0.0 on AMD's Radeon AI PRO R9700 GPU. The improvements highlight the importance of staying current with driver and runtime updates for optimal local inference performance.
-
Kog AI – Building a Real-Time Inference Stack on AMD Instinct GPUs
A technical presentation on building production inference systems using AMD Instinct GPUs, expanding the hardware ecosystem for local LLM deployment beyond NVIDIA dominance. The talk covers real-time inference optimization techniques applicable to on-device deployments.
-
AMD's vLLM-ATOM Plugin Supercharges DeepSeek-R1 and Kimi-K2 Inference on MI350/MI400
AMD has released a vLLM-ATOM plugin optimizing inference for DeepSeek-R1, Kimi-K2, and gpt-oss-120B models on Instinct MI350 and MI400 accelerators, delivering significant performance gains for local deployment.
-
$200 NVIDIA V100 Server GPU Mod Beats RTX 3060 in Local LLM Test
A creative hardware modification using refurbished NVIDIA V100 server GPUs demonstrates strong price-to-performance for local LLM inference, outperforming newer consumer-grade GPUs at a fraction of the cost.
-
Locked, stocked, and losing budget: AI vendor lock-in bites back
Analysis of how proprietary AI services create vendor lock-in, making the case for self-hosted and local LLM deployment as a cost-effective alternative.
-
NVIDIA Adds Day-0 DeepSeek V4 Blackwell Support
NVIDIA has announced immediate support for DeepSeek V4 on Blackwell GPUs, enabling optimized local inference for one of the latest high-performance language models on cutting-edge hardware.
-
Elastic KV Cache Memory Breakthrough Enables Efficient Bursty LLM Serving and GPU Sharing
A new coding implementation on elastic KV cache memory optimization allows more efficient handling of variable-load LLM serving patterns and multi-model GPU sharing scenarios.
-
Show HN: We built an OCR server that can process 270 dense images/s on a 5090
A high-performance OCR inference server achieving 270 dense images per second on a single GPU, demonstrating practical edge inference optimization techniques.
-
DGX Spark Setup Guide: Running vLLM and PyTorch for Local LLM Inference Backend
A developer details their setup process for NVIDIA DGX Spark hardware running vLLM with Hugging Face models as a local API backend for education and analytics applications while maintaining privacy.
-
Researchers Achieve 1-Bit Quantization of OLMo-3 7B Using Distillation
A novel approach using quantization-aware distillation successfully compressed OLMo-3 7B Instruct to 1-bit precision, enabling ultra-efficient inference on severely resource-constrained devices.
-
MiniMax-M2.7 Delivers Exceptional Performance on Consumer Hardware
MiniMax-M2.7 benchmarks show strong throughput (127.7 tok/s on dual RTX PRO 6000 Blackwell) and efficient VRAM utilization, positioning it as a practical alternative to larger models for resource-constrained deployments.
-
MiniMax M2.7 Advances Scalable Agentic Workflows on NVIDIA Platforms for Complex AI Applications
MiniMax releases M2.7, optimized for NVIDIA hardware platforms to support complex agentic workflows at scale. The model demonstrates improved performance and efficiency for self-hosted deployment scenarios requiring advanced reasoning capabilities.
-
Qwen 3.5 122B Achieves 198 Tokens/sec on Dual RTX PRO 6000 Blackwell GPUs
A detailed optimization case study demonstrates running Qwen 3.5 122B at impressive inference speeds on a budget dual-GPU Blackwell setup. The community shares verified benchmarks with full methodology and reproducible results for large-scale local deployment.
-
AMD Announces Day 0 Support for Google Gemma 4 Across Processors and GPUs
AMD has delivered immediate support for Google's Gemma 4 model across its processor and GPU lineup, enabling optimized local inference on AMD hardware. This expands accessibility for running powerful open-weight models on-device.
-
Qwen 3.5 397B Reduced to 35% Parameters With Usable Quality on 96GB GPU
A community researcher successfully compressed Qwen 3.5 397B to 35% of its original size while maintaining practical quality, enabling the model to run on dual GPU setups. The REAP35 variant demonstrates advanced parameter reduction techniques for enterprise-scale model deployment.
-
DGX Spark Hardware Limitations: Missing NVFP4 Support Undermines Local AI Value Proposition
User experience reports reveal that NVIDIA's DGX Spark lacks critical NVFP4 (NV Tensor Float 32) support six months after launch, significantly limiting its utility for cost-effective local model inference despite Blackwell GPU capabilities.
-
NVIDIA and Google Optimize Gemma 4 AI Models for Local RTX Deployment
NVIDIA and Google have collaborated to optimize Gemma 4 models specifically for NVIDIA RTX GPUs, enabling high-performance local inference. The optimization work ensures efficient utilization of consumer and professional GPUs for on-device AI workloads.
-
GPUs vs. TPUs: Decoding the Powerhouses of AI
A comprehensive comparison of GPU and TPU architectures for AI workloads, examining trade-offs between general-purpose graphics processors and tensor-optimized units for local and edge LLM deployment scenarios.
-
Google Launches Gemma 4 For Advanced On-Device AI
Google has released Gemma 4, an open model family designed for on-device AI inference across phones, tablets, and GPUs. The new models target efficient local deployment with improved capabilities for edge computing scenarios.
-
AMD Rolls Out Gemma 4 Model Support Across Full Range of GPUs & CPUs
AMD has announced comprehensive support for Gemma 4 across its entire lineup of GPUs and CPUs, enabling local inference on AMD-based systems. The support extends from consumer Ryzen processors to professional EPYC servers and RDNA GPUs.
-
NVIDIA Accelerates Gemma 4 for Local Agentic AI on RTX GPUs
NVIDIA provides day-one optimizations for Google's Gemma 4 models across its RTX GPU lineup, enabling accelerated local inference for agentic AI workflows on consumer and enterprise graphics cards.
-
Intel's $949 GPU Has 32GB of VRAM for Local AI, but Software is Why Nvidia Keeps Winning
Intel's new GPU offers impressive hardware specs with 32GB of VRAM at a competitive price point, yet software ecosystem maturity and optimization remain the deciding factor favoring Nvidia for local LLM deployment.
-
GPU Passthrough to LXCs in Proxmox Simplifies Local Inference Infrastructure
GPU passthrough to LXC containers in Proxmox offers a simpler and more efficient alternative to virtual machines for local LLM deployment, improving resource utilization and reducing complexity.
-
Linux Significantly Outperforms Windows for Local LLM Inference
A detailed comparison shows inference running substantially faster on Linux versus Windows on identical hardware, with implications for local deployment optimization.
-
Hold on to Your Hardware: Implications for Local LLM Deployment
An article examining hardware longevity and sustainability raises important considerations for practitioners investing in local inference infrastructure.
-
Qwen 3.5 27B Achieves 1.1M Tokens/Second on B200 GPUs with Optimized vLLM Config
A developer optimized Qwen 3.5 27B to reach 1.1 million tokens per second on 96 B200 GPUs using vLLM, with detailed configurations and all settings published on GitHub. Key optimizations included distributed parallelism, reduced context windows, FP8 KV cache, and speculative decoding.
-
Llama.cpp Benchmark: RTX 5090 vs Enterprise Systems Compared
Comprehensive llama-bench benchmarks comparing RTX 5090 consumer GPU against DGX Spark and AMD AI395 in real-world local inference scenarios, with ROCm and Vulkan results included.
-
Llama.cpp ROCm 7 vs Vulkan Performance Benchmarks on AMD Mi50
Performance benchmarks comparing ROCm 7 and Vulkan backends on AMD Mi50 GPUs provide crucial data for optimizing local inference on AMD hardware. These results help practitioners select the best acceleration backend for their specific AMD GPU configurations.
-
Rust Project Perspectives on AI
The Rust project team discusses how AI intersects with systems programming and language design, with implications for building efficient local LLM infrastructure.
-
ik_llama.cpp Fork Delivers 26x Faster Prompt Processing on Qwen 3.5 27B
A fork of llama.cpp called ik_llama.cpp is delivering dramatic 26x speed improvements for prompt processing on Qwen 3.5 27B models. Real-world benchmarks on Blackwell RTX PRO GPUs show tangible performance gains for production agentic workloads.
-
Custom GPU Multiplexer Achieves 0.3ms Model Switching on Legacy Hardware
A developer built a custom Linux kernel module that multiplexes six GPUs through a single PCIe slot, enabling model hot-swapping in under 0.3 milliseconds using repurposed Bitcoin mining hardware.
-
Qwen3.5-397B Achieves 282 tok/s on 4x RTX PRO 6000 Blackwell Through Custom CUTLASS Kernel
A developer achieved a 5x performance improvement on the massive Qwen3.5-397B model by building a custom CUTLASS kernel to fix SM120's broken MoE GEMM tiles, reaching 282 tokens/second on Blackwell GPUs. This breakthrough demonstrates significant optimization potential for running large models locally with multi-GPU setups.
-
Nvidia's Nemotron 3 Super: Understanding the Significance for Local LLM Deployment
NVIDIA's Nemotron 3 Super release carries broader implications for local LLM deployment and optimization than initially apparent, with the model designed for efficient inference on consumer and professional GPUs. The community is recognizing its importance for self-hosted LLM practitioners.
-
Sarvam Open-Sources 30B and 105B Reasoning Models
Sarvam has released open-source reasoning models in 30B and 105B sizes, expanding the landscape of locally-deployable reasoning capabilities beyond the dominant players.
-
Cutile.jl Brings Nvidia CUDA Tile-Based Programming to Julia
Cutile.jl enables tile-based CUDA programming in Julia, offering improved GPU utilization and performance optimization capabilities for compute-intensive workloads including LLM inference.
-
Comprehensive MoE Backend Benchmarks for Qwen3.5-397B: Real Numbers vs Hype
A detailed benchmark of every major MoE backend for Qwen3.5-397B NVFP4 on workstation GPUs reveals actual sustained performance of 50.5 tok/s, significantly lower than commonly cited claims. The analysis uncovers kernel issues in Nvidia's own CUTLASS implementation.
-
Sarvam Open-Sources 30B and 105B Reasoning Models
Indian AI startup Sarvam has released open-source reasoning models in 30B and 105B parameter sizes, providing locally-deployable alternatives for reasoning tasks without reliance on proprietary APIs.
-
Qwen 3.5 Family Benchmark Comparison Shows Strong Performance Across Smaller Models
New benchmarks reveal that Qwen 3.5's 27B, 35B, and 122B variants retain most of the flagship model's performance, while smaller 2B and 0.8B models show steeper degradation on long-context and agent tasks.
-
Intel Arc Pro B70 Workstation GPU Confirmed via vLLM AI Release Notes
Intel's Arc Pro B70 discrete GPU receives official support in vLLM release notes, expanding local LLM inference options for professional workstations. The BMG-G31 architecture targets professional AI computing workflows.
-
Google Is Exploring Ways to Use Its Financial Might to Take on Nvidia
Google explores strategic investments and partnerships to compete with Nvidia's dominance in AI accelerator chips, potentially enabling more accessible hardware options for local LLM deployment. This shift could significantly impact the economics of on-device inference infrastructure.
-
NVIDIA Releases Dynamo v0.9.0: Infrastructure Overhaul With FlashIndexer and Multi-Modal Support
NVIDIA's Dynamo v0.9.0 update introduces significant infrastructure improvements including FlashIndexer and multi-modal support, advancing the capabilities of local inference frameworks on NVIDIA hardware.
-
AMD Announces Day 0 Support for Qwen 3.5 LLM on Instinct GPUs
AMD has enabled immediate support for the Qwen 3.5 model on its Instinct GPU lineup, providing optimized inference performance for local deployments on AMD hardware accelerators.
-
High Bandwidth Flash Memory Could Alleviate VRAM Constraints in Local LLM Inference
A technical discussion explores how high-bandwidth flash (HBF) storage could supplement GPU VRAM for local inference, potentially enabling 256GB+ effective memory pools from consumer hardware at 10x lower cost than traditional VRAM.
-
OpenClaw with vLLM Running for Free on AMD Developer Cloud
AMD launches free cloud access to run OpenClaw and vLLM inference workloads, providing developers with no-cost GPU resources for local LLM development.