Tagged "inference-latency"
17 articles tagged inference-latency, 28 March 2026 to 20 October 2026. Newest first.
-
vLLM Optimises Disaggregated Inference with Push-Based KV Transfer
AWS and vLLM introduce push-based KV cache transfer to improve time-to-first-token and time-per-output-token for disaggregated inference deployments. This optimization technique reduces latency in multi-node local inference setups.
-
Local LLMs vs Cloud API Cost Analysis 2026
A comprehensive cost-benefit analysis comparing on-device LLM inference against cloud API consumption, with practical data on total cost of ownership, latency, and privacy tradeoffs.
-
Can a 2.8T Model Run on a Single Node of Nvidia B300 X8?
A practical deployment analysis examining whether ultra-large trillion-parameter models can be efficiently served on a single high-end GPU node, providing real-world benchmarks for modern hardware.
-
Grok Launches Excel AI Add-in for Integrated Model Access
Grok introduces an AI add-in for Excel, bringing LLM capabilities directly into a productivity tool interface. This represents growing integration of AI inference into mainstream software ecosystems.
-
Samsung Presents UFS 5.0 Storage Targeted at On-Device AI Performance
Samsung's next-generation storage interface optimizes for the intensive I/O patterns required by on-device AI inference, addressing a critical bottleneck in local LLM deployment.
-
Liquid AI Launches Edge-Focused LFM2.5 Model to Power On-Device AI Agents
Liquid AI has released the LFM2.5 model specifically optimized for edge deployment and on-device AI agents. This new model represents a significant development for practitioners looking to run capable language models locally with reduced resource requirements.
-
Samsung's Exynos 2800 Brings HBM Memory to Mobile AI, Enabling Faster Local Model Inference
Samsung's next-generation Exynos 2800 processor will feature high-bandwidth memory (HBM) integration, significantly improving on-device AI performance and memory throughput for local model execution on smartphones.
-
Google Makes Gemini 3.5 Flash the Default AI Model for Billions of Users
Google's decision to make Gemini 3.5 Flash the default model for billions of users signals industry trends toward smaller, faster models optimized for on-device and edge inference. This shift has implications for local LLM development and deployment strategies.
-
Supercharging LLM Inference on Google TPUs: Achieving 3X Speedups With Diffusion-Style Speculative Decoding
Google researchers have demonstrated 3x inference speedups on TPUs using diffusion-style speculative decoding, a novel optimization technique that could influence local inference strategies. The breakthrough shows how advanced decoding methods can dramatically reduce latency on specialized hardware.
-
Using a Local LLM as a Zero-Shot Classifier
Detailed guide demonstrating how to leverage locally-running language models for zero-shot text classification tasks without fine-tuning, reducing infrastructure costs and inference latency.
-
Tesseron: New API Framework for AI Agents with Developer-Defined Configuration
BrainBlend-AI releases Tesseron, an API framework allowing app developers to define AI agent behavior and configuration. The framework is designed to simplify local agent deployment and orchestration.
-
The AI-Ready Product Data Framework for B2B Commerce
A framework for structuring product data to enable efficient local and edge-based AI processing in B2B commerce applications.
-
Build a More Secure, Always-On Local AI Agent with OpenClaw and NVIDIA NemoClaw
NVIDIA releases OpenClaw and NemoClaw, new frameworks for building secure, always-on local AI agents with enhanced privacy and reduced latency. This represents a significant step forward in production-ready on-device AI deployment.
-
DMax: New Parallel Decoding Paradigm for Diffusion Language Models
National University of Singapore researchers present DMax, a novel approach enabling aggressive parallel decoding in diffusion language models through progressive self-refinement, potentially revolutionizing inference speed.
-
NVIDIA Accelerates Gemma 4 for Local Agentic AI on RTX GPUs
NVIDIA provides day-one optimizations for Google's Gemma 4 models across its RTX GPU lineup, enabling accelerated local inference for agentic AI workflows on consumer and enterprise graphics cards.
-
Is Anyone Working on an AI Operating System?
An active Hacker News discussion exploring whether anyone is building operating systems designed from the ground up for AI workloads and inference, addressing questions about architecture, scheduling, and optimization for local LLM deployment infrastructure.
-
GPU Passthrough to LXCs in Proxmox Simplifies Local LLM Deployment
GPU passthrough to Linux containers in Proxmox offers superior performance and simplicity compared to virtual machines for running local LLMs, enabling efficient on-device inference without virtualization overhead.