Tagged "inference-throughput"
16 articles tagged inference-throughput, 20 February 2026 to 20 October 2026. Newest first.
-
Ollama vs vLLM: Comparing Simplicity and Throughput for Local Inference
A comprehensive comparison between Ollama and vLLM explores the trade-offs between ease of use and production-scale throughput for local LLM deployment. The analysis helps practitioners choose the right framework for their specific use case.
-
Deploying LLMs Locally: Ollama, llama.cpp, and vLLM Compared
A comprehensive comparison of three major frameworks for local LLM deployment, examining their strengths, trade-offs, and use cases for practitioners seeking to run models on-device.
-
Akamai Doubles GPU Throughput with Single vLLM Flag Change
Akamai's Khaja Omer demonstrates a simple vLLM configuration flag that doubles inference throughput on a single GPU, highlighting practical performance optimization for local LLM deployment.
-
vLLM on Vera Rubin Posts 7.8x GB200 Throughput per Chip
vLLM achieves 7.8x throughput improvements on NVIDIA's GB200 hardware through Vera Rubin optimizations, demonstrating significant performance gains for local inference clusters.
-
Local LLM Inference at Scale with vLLM
Deep dive into using vLLM for high-throughput local LLM inference, covering optimization techniques and scaling strategies for self-hosted deployments.
-
vLLM v0.28.0 Released
The latest version of vLLM, a popular high-throughput LLM serving framework, has been released with performance improvements and new features for local and distributed inference.
-
vLLM's Disaggregated Serving Cuts GPU Interference, Delivering 2.5x Higher Goodput
vLLM introduces disaggregated serving architecture that significantly reduces GPU memory interference, achieving 2.5x improvement in goodput on the same hardware. This breakthrough enables more efficient batch processing and higher throughput for local and self-hosted LLM deployments.
-
Squeezing Silicon Limits: Effective Strategies to Eliminate GPU Idle Time and Maximize GPU Utilization
Practical techniques for maximising GPU utilisation during local LLM inference, addressing idle time and throughput bottlenecks that waste expensive compute resources.
-
Intel-Scaler-vLLM 0.21.0-b1 Brings Latest Features for vLLM on Intel GPUs
vLLM, the high-performance LLM inference engine, has released version 0.21.0-b1 with optimized support for Intel GPUs. This update enables developers to leverage Intel's discrete graphics for efficient local model serving.
-
Snapdragon C Specs Revealed: 6nm Process, On-Device AI Engine for Budget Laptops
Qualcomm has unveiled detailed specifications for the Snapdragon C processor featuring a 6nm process and dedicated on-device AI engine. The 1+3+4 core configuration and LPDDR5 memory support make it particularly relevant for running local LLMs on affordable edge devices.
-
vLLM vs Ollama 2026: Performance Benchmark Reveals 9x Throughput Gap
A comprehensive benchmark comparison shows vLLM significantly outperforming Ollama in throughput metrics, with implications for choosing the right inference framework for local deployments.
-
Show HN: We built an OCR server that can process 270 dense images/s on a 5090
A high-performance OCR inference server achieving 270 dense images per second on a single GPU, demonstrating practical edge inference optimization techniques.
-
GPU Memory for LLM Inference (Part 1)
A detailed technical guide exploring GPU memory optimization strategies for running large language models efficiently during inference, critical knowledge for anyone deploying LLMs locally with limited VRAM.
-
Samsung Galaxy Book6 Brings Consumer-Grade On-Device AI Hardware to Market
Samsung's new Galaxy Book6 series with Nvidia RTX 5070 graphics represents a maturation of consumer hardware specifically optimised for on-device AI inference and local LLM deployment.
-
Llama.cpp Benchmark: RTX 5090 vs Enterprise Systems Compared
Comprehensive llama-bench benchmarks comparing RTX 5090 consumer GPU against DGX Spark and AMD AI395 in real-world local inference scenarios, with ROCm and Vulkan results included.
-
The Path to Ubiquitous AI (17k tokens/sec)
A technical analysis of achieving 17,000 tokens per second inference throughput, demonstrating the performance milestones required for truly practical local LLM deployment at scale.