Tagged "rag"
85 articles tagged rag, 11 February 2026 to 15 October 2026. Newest first.
-
llama.cpp Release b11643: Embedding-Only Graph Optimization and Performance Improvements
llama.cpp continues rapid development with release b11643, introducing critical optimisations for embedding workflows and graph handling. This release reflects ongoing improvements to the most widely-used local inference runtime for consumer hardware.
-
Modular's MAX Framework Delivers 5.5x Speedup Over vLLM on Kimi Models
Modular's MAX inference framework achieves 5.5x performance improvement over vLLM when running Kimi K2.5 models, demonstrating significant optimisations for local LLM inference speed and efficiency.
-
EpochGo: Multi-Agent AutoML with Ollama Reaches Zero API Costs
EpochGo demonstrates local-first multi-agent AutoML running on Ollama, eliminating cloud API dependencies and costs while automating model selection and hyperparameter optimization.
-
Google Releases Local-First Embedding Model for Photo and Audio Search
Google introduced a new embedding model capable of searching photos and audio without requiring cloud connectivity, enabling on-device multimodal search.
-
Ollama v0.40.0 Brings MLX Runtime to Apple Silicon and Decision Models
Ollama's latest release automatically routes supported model architectures to the MLX runtime on Apple Silicon devices, improving inference performance. The release also adds support for decision models and multimodal embeddings including EmbeddingGemma2.
-
Ollama v0.40.0 Brings Multimodal Embeddings and MLX Default on Apple Silicon
Ollama's latest release makes MLX the default runtime for supported models on Apple Silicon and implements native multimodal embedding support through EmbeddingGemma2Model architecture. This brings efficient text, audio, and video embeddings to on-device deployment with significantly reduced memory footprint.
-
Google DeepMind Releases EmbeddingGemma 2: A 740M Multimodal Embedding Model for On-Device AI
Google DeepMind has released EmbeddingGemma 2, an open-source multimodal embedding model that processes text, audio, and video while running in just 567MB of memory. Built on Gemma 4, this 740M parameter model enables efficient semantic search and retrieval on-device without cloud dependencies.
-
GridCore: A GPU Scheduler for Running Multiple Local LLMs on One GPU
GridCore is a new scheduler that enables multiple local LLMs to share a single GPU efficiently. This tool addresses a critical constraint for local deployment: allowing developers to run multiple models simultaneously without requiring multiple GPUs.
-
How to Set Up GraphRAG Locally: 13 Steps, 90 Minutes Deployment Guide
A detailed practical guide for deploying GraphRAG locally in approximately 90 minutes, providing step-by-step instructions for practitioners building knowledge graph-augmented retrieval systems on-device.
-
Perplexity Brings On-Device AI to AMD Ryzen AI Max PCs, Running 27B Models Locally
Perplexity has enabled on-device AI inference on AMD Ryzen AI Max processors, allowing users to run 27-billion parameter models entirely locally. This demonstrates practical large-model deployment on consumer-grade hardware without cloud dependencies.
-
4-Bit Rotational Quantization: -45% RAM, <1% Recall Drop vs. TurboQuant
Weaviate presents a 4-bit rotational quantization technique achieving 45% RAM reduction with less than 1% recall degradation, advancing the state of memory-efficient inference.
-
4-Bit Rotational Quantization: -45% RAM, <1% Recall Drop vs. TurboQuant
Weaviate introduces a new quantization technique achieving 45% RAM reduction with negligible accuracy loss, advancing memory-efficient local model deployment.
-
Local LLMs Replace Google NotebookLM Functionality for Privacy-Conscious Researchers
An XDA comparison shows that self-hosted LLMs can replicate NotebookLM's features without uploading sensitive research to Google's servers, providing a privacy-preserving alternative for document analysis.
-
What Can You Do with a Local LLM?
A comprehensive exploration of practical use cases and capabilities enabled by running large language models locally. This guide helps practitioners understand where local LLMs provide genuine advantages over cloud-based alternatives.
-
Llama.cpp Fork Enables Qwen 3.8 27B with Large Contexts on 16GB VRAM GPUs
A specialized llama.cpp fork implements adaptive KV streaming to run Qwen 3.8 27B with large context windows on 16GB VRAM GPUs, significantly reducing hardware requirements for production-grade inference.
-
Efficient Decode Context Parallelism with vLLM for Long Context Workloads
vLLM introduces decode context parallelism technique to handle long-context inference efficiently, reducing memory overhead and latency for local deployments processing large documents and extended conversations.
-
Ollama 0.32.10: 7-8% Prefill Speed Gains on NVFP4 Models
Ollama 0.32.10 delivers significant prefill performance improvements for NVFP4 quantized models through kernel fusion optimizations, alongside updated default repeat penalty settings for improved speculative decoding.
-
Liquid AI Releases LFM2.5-VL-3B: Compact Vision-Language Model for Edge Inference
Liquid AI unveiled LFM2.5-VL-3B, a 3 billion parameter vision-language model designed for on-device deployment with capabilities for screen reading, object grounding, and tool calling without server dependencies.
-
Liquid AI LFM2.5-2.6B: Open-Weights Agentic Model With 128K Context and Tool Calling
Liquid AI releases an open-weights agentic model optimized for on-device deployment with 128K context window, tool calling capabilities, and support for extremely low-resource edge hardware.
-
K-EXAONE 2.0 Brings 262K Context to Frontier AI
K-EXAONE 2.0 introduces a 262K token context window, significantly expanding the capabilities of frontier-class models for local deployment and extended reasoning tasks. This represents a major advancement in practical context window management.
-
Stop Paying for Search APIs—This Self-Hosted Tool Lets Your Local LLM Search the Web for Free
A new self-hosted tool enables local LLMs to perform web searches without relying on paid search APIs, eliminating subscription costs while maintaining privacy. This development makes it practical to build retrieval-augmented generation (RAG) applications entirely on-premise.
-
Onemind.md – Adding Repository Memory to LLMs Without Extra Tooling
Simple approach to augmenting local LLM context with project-specific knowledge, enabling better code understanding without external infrastructure.
-
SigMap: 97% Token Reduction for AI Coding Sessions
SigMap achieves significant token efficiency improvements for AI coding workflows, reducing context size by 97% while maintaining functionality. This breakthrough in token optimization has direct implications for running LLMs locally with constrained memory and compute resources.
-
Building a Personal Ebook Librarian with Local LLMs for Better Recommendations
A user developed a local LLM-based system to manage and recommend ebooks from their personal library, achieving better results than traditional recommendation services like Goodreads.
-
code-on-incus: Isolated Machine Environments for AI Agents
A new tool that provisions isolated container environments with root access for each AI agent, enabling safer sandboxed execution of agent code on local infrastructure. This addresses a critical security concern for deploying autonomous AI systems locally.
-
LongCat-2.0 Released
LongCat-2.0 represents an advancement in handling long-context sequences locally. While limited details are available, this release is relevant to local LLM practitioners seeking models optimized for extended context windows on consumer hardware.
-
Beyond Setup: Production Practices for Local LLM Deployment
A practical guide exploring what comes after initial local LLM setup, covering production considerations like monitoring, optimization, and operational best practices for sustained on-device inference.
-
Meet EverOS: An Open Source Markdown-First Agent Memory Runtime With Hybrid BM25 + Vector Retrieval
A new open-source framework provides markdown-based agent memory management with hybrid semantic and keyword search capabilities, enabling self-evolving AI agents that can run locally.
-
LLM-Free, Layout-Aware PDF Chunker in Pure Rust
A new PDF chunking utility written in Rust that preserves document structure without requiring LLM inference, improving RAG pipeline efficiency for local deployments.
-
Local Semantic Search Engine in Rust, No External DB
LocalMind brings a lightweight semantic search implementation written in Rust that operates without external database dependencies, ideal for self-contained local search applications.
-
You Can Now Run Max AI Models on Apple Silicon
Modular's Max platform now supports running AI models directly on Apple Silicon GPUs, expanding local deployment options for macOS users and M-series chip owners.
-
GEEKOM A9 Max Delivers 32GB RAM and Native LLM Support in Compact Form Factor
GEEKOM's A9 Max mini PC features 32GB RAM and is optimized for running language models locally. This hardware release targets the growing segment of practitioners seeking dedicated edge inference devices.
-
Qwable: New Free Local Model Brings Claude-like Capabilities to Edge Devices
Qwable is a new open-source local language model optimized for edge deployment, offering Claude-comparable reasoning and instruction-following without cloud dependencies. The model targets developers seeking private, self-hosted alternatives.
-
Mac Mini Positioned as Premier On-Device AI Computer for Local LLM Inference
Recent analysis highlights Mac Mini as an exceptional platform for running large language models locally, combining affordability with strong GPU performance and optimized software support for on-device AI workloads.
-
I Built a Bedside AI Assistant That Reads Me the News Without Touching the Cloud
A practical demonstration of building a completely local AI assistant that delivers personalized news without any cloud connectivity, showcasing real-world on-device LLM deployment techniques.
-
PageToMD – A CLI tool to turn web pages into clean Markdown for AI agents
A new command-line utility converts web pages into clean, structured Markdown format optimized for local LLM processing. This tool streamlines data preparation for local inference pipelines and agent workflows.
-
Architecting Modular Local AI Ecosystems to Escape Token Economics
New approaches to modular local AI architecture enable users to build custom ecosystems that avoid usage-based billing models entirely. This enables true cost predictability and ownership for long-term AI deployments.
-
Show HN: Veritrooper – find what your AI gets wrong about your own docs
A new tool for validating and benchmarking local LLM accuracy against proprietary documentation, helping teams identify hallucinations and verify RAG system quality before production deployment.
-
Ask HN: What is the AI setup for an experienced dev starting on a new project?
A community discussion on Hacker News where experienced developers share their practical AI tooling preferences and workflows, offering real-world insights for setting up local LLM development environments.
-
Show HN: LLM Memory Without Context Bleed – 100% Precision vs. <10% Vector Search
A new memory system for LLM applications achieves 100% precision in context retrieval compared to vector search's <10%, enabling more reliable and efficient local deployment of agentic systems.
-
Run Llama.cpp In-Process from Java with Project Panama FFM
A new project enables developers to run Llama.cpp directly from Java applications using Project Panama's Foreign Function & Memory API, eliminating subprocess overhead and expanding local LLM deployment options for JVM ecosystems.
-
LLM Memory Systems Benchmark: High Recall, Near-Zero Precision for Tested Systems
A new benchmark reveals critical weaknesses in LLM memory systems, showing high recall but near-zero precision across tested implementations. This finding is crucial for developers building stateful local LLM applications and agentic systems.
-
Local LLM Setup: How to Use RAG and an Embedding Model to Stop Wasting Context
A practical guide on optimizing local LLM deployments by combining retrieval-augmented generation with embedding models to maximize context efficiency and reduce token waste.
-
LLM Wiki App Chunker: Transform Documents Into Navigable Knowledge Trees
A new tool called Chunker enables document transformation into navigable knowledge tree structures for local LLM applications. This addresses a critical challenge in RAG and local knowledge management systems.
-
On-Device AI to Be in 80% of Wearables by 2032
Market research projects that on-device AI will become standard in 80% of wearables by 2032, driving demand for ultra-efficient models and hardware optimized for constrained environments. This trend indicates significant growth opportunities for local LLM deployment on edge devices.
-
Local LLM Integration Enables Replacement of Paid Subscription Services
A practitioner demonstrates replacing three subscription-based applications by deploying a local language model with access to personal files, showcasing cost savings and privacy benefits.
-
Discussion: Including New Mathematical Proofs in LLM Training Data for Rediscovery
A Hacker News discussion explores whether LLMs can rediscover novel mathematical proofs when included in training data, relevant to understanding model capabilities and knowledge synthesis.
-
Agentic AI Community Focus: Building Local Agents in 2026
The emerging agentic AI community shares resources and frameworks for building autonomous agents with local LLM backends. Focus areas include memory systems, tool integration, and edge deployment of multi-step reasoning tasks.
-
Show HN: Memex, Claude Memory via Local RAG with MCP and Offline Embeddings
Memex enables persistent memory for Claude through local retrieval-augmented generation using offline embeddings and Model Context Protocol, eliminating cloud dependency for context management.
-
SQL Server 2025 Adds Built-in Chunking and Vector Support
Microsoft SQL Server 2025 introduces native vector database capabilities and chunking utilities, streamlining local LLM deployment with RAG and semantic search workflows.
-
Xmemory: Benchmarking Structured AI Memory Against RAG and Hybrid RAG
A new benchmark comparing structured AI memory systems against retrieval-augmented generation (RAG) approaches, providing insights for optimizing local LLM deployments with better context management and memory efficiency.
-
N8n, Dify, and Ollama Might Be the Best Self-Hosted AI Automation Stack Right Now
A powerful combination of n8n, Dify, and Ollama creates a complete end-to-end self-hosted AI automation platform. This stack enables developers to build, deploy, and orchestrate local LLM workflows without cloud dependencies.
-
Mathesar 0.10.0
Mathesar releases version 0.10.0 with improvements that enhance data management capabilities for self-hosted deployments and local infrastructure projects.
-
16 Ways to Make a Small Language Model Think Bigger
Oracle has published a comprehensive guide on techniques to enhance the effective capability of small language models through prompting, retrieval, and architectural approaches—highly relevant for practitioners optimizing local deployments.
-
N8n, Dify, and Ollama Emerge as Leading Self-Hosted AI Automation Stack
The combination of Ollama for inference, Dify for LLM orchestration, and N8n for workflow automation is proving to be an exceptionally capable open-source stack for self-hosted AI applications.
-
Minisforum N5 MAX AI NAS Delivers 126 TOPS with 200TB Storage for Local LLM Workloads
Minisforum released the N5 MAX AI NAS, a specialized device combining 126 TOPS of AI compute with 200TB storage capacity, purpose-built for local LLM server deployment. This hardware bridges the gap between consumer devices and enterprise AI infrastructure.
-
Universal Knowledge Store and Grounding Layer for AI Reasoning Engines
New framework providing a knowledge store and grounding layer to improve reasoning capabilities and factual accuracy of local AI models.
-
Context Window Optimization: Extending Gemma 4 Context Length Through Efficient Projection Quantization
Community members discover that quantizing vision projections to Q8 format in Gemma 4 multimodal models eliminates quality degradation while enabling 30K additional context tokens without VRAM increase.
-
Does RAG Help AI Coding Tools?
Analysis examining whether Retrieval-Augmented Generation actually improves code generation quality in AI coding assistants and local deployment scenarios.
-
Ask HN: What do you use for local embeddings?
Community discussion on Hacker News exploring the best tools and approaches for running embedding models locally without external API dependencies.
-
RAG Deployment Lessons from Regulated Industries
Practical insights from deploying RAG-powered local AI assistants in highly regulated sectors including construction, aged care, and mining operations.
-
Lat.md: Agent Lattice – A Knowledge Graph for Your Codebase in Markdown
A new tool that builds structured knowledge graphs from codebases in Markdown format, enabling better context management and retrieval for AI agents operating on local codebases.
-
Velr: Embedded Property-Graph Database for Local LLM Applications
Velr introduces an embedded property-graph database built in Rust on top of SQLite, enabling local LLM systems to maintain structured knowledge graphs without external dependencies.
-
LM Studio Releases Reworked Plugins with Fully Local Web Research
LM Studio has published improved versions of its plugins including DuckDuckGo and website visiting capabilities, enabling fully local web research workflows for LLM applications. These tools eliminate the need for external API calls while maintaining practical web integration.
-
Powerful AI Search Engine Built on Single GeForce RTX 5090
An enthusiast successfully deployed a fully-featured AI search engine on a single GeForce RTX 5090 GPU, demonstrating the viability of complex local inference workloads on consumer hardware.
-
Llama 8B Matches 70B Performance on Multi-Hop QA Using Structured Prompting
Structured prompting techniques with Graph RAG enable smaller Llama 8B models to match 70B model performance on complex multi-hop question answering without fine-tuning. Research reveals reasoning, not retrieval, is the actual bottleneck.
-
LMCache Dramatically Accelerates LLM Inference on Oracle Data Science Platform
Oracle integrates LMCache, a cutting-edge prompt caching and KV cache optimization technique, into their cloud data science platform to accelerate LLM inference and reduce computational overhead.
-
MiniMax-M2.7: New Compact Model Announced for Local Deployment
MiniMax has announced the M2.7 model, generating interest in the community regarding its potential multimodal capabilities and suitability for local inference workloads.
-
Mamba 3: State Space Model Architecture Optimized for Inference
Mamba 3 introduces a state space model architecture specifically optimized for efficient inference performance, offering a potential alternative to traditional transformer-based architectures for local deployment.
-
Gloss: Open-Source, Local-First RAG Alternative to NotebookLM Built in Rust
A developer released Gloss, a privacy-focused research workspace featuring hybrid search, explicit RAG control, and local model support—a fully open alternative to Google's NotebookLM without proprietary API dependencies.
-
AI Agent Reliability Tracker
Princeton's reliability tracking tool provides benchmarking and monitoring capabilities for AI agents, offering metrics crucial for evaluating local deployment stability.
-
Framework Choice Critical: llama.cpp and vLLM Outperform Ollama for Qwen 3.5 Testing
Community PSA reveals significant performance and correctness differences between local inference frameworks when running Qwen 3.5 models, with llama.cpp, transformers, vLLM, and SGLang producing correct results while Ollama shows issues with reasoning and tool use.
-
RAG vs. Skill vs. MCP vs. RLM: Comparing LLM Enhancement Patterns
A comparative analysis of four major architectural patterns for augmenting LLMs with external knowledge and capabilities, helping developers choose the right approach for their local deployment needs.
-
RAG-Enterprise – 100% Local RAG System for Enterprise Documents
A new open-source RAG system designed for enterprise document processing that runs entirely locally, enabling organizations to implement retrieval-augmented generation without cloud dependencies or data exposure.
-
Building a Privacy-Preserving RAG System in the Browser
A guide for implementing retrieval-augmented generation entirely in the browser using local models, maintaining complete data privacy. Demonstrates advanced local LLM architectures running entirely client-side.
-
Researchers Develop Persistent Memory System for Local LLMs—No RAG Required
A novel approach enables local language models to retain facts learned during conversations by storing them directly in model weights through a sleep mechanism. The system runs on consumer hardware like MacBook Air and eliminates the need for traditional retrieval-augmented generation.
-
Elastic Introduces Best-in-Class Embedding Models for High Performance Semantic Search
Elastic announces optimized embedding models designed for efficient semantic search, enabling local deployment of vector search capabilities without cloud dependencies.
-
Elastic Introduces Best-in-Class Embedding Models for High Performance Semantic Search
Elastic releases optimized embedding models designed for local deployment and semantic search applications. These models enable efficient vector search on-device without external API dependencies.
-
Search and Analyze Documents from the DOJ Epstein Files Release with Local LLM
A practical demonstration of deploying local LLMs for large-scale document analysis, using the newly released DOJ files as a case study. This project showcases real-world applications of self-hosted language models for sensitive document processing.
-
NVIDIA Releases Dynamo v0.9.0: Infrastructure Overhaul With FlashIndexer and Multi-Modal Support
NVIDIA's Dynamo v0.9.0 update introduces significant infrastructure improvements including FlashIndexer and multi-modal support, advancing the capabilities of local inference frameworks on NVIDIA hardware.
-
Local-First RAG: Vector Search in SQLite with Hamming Distance
A practical guide to implementing retrieval-augmented generation entirely on-device using SQLite for vector search, eliminating the need for external databases.
-
InitRunner: YAML-Based AI Agent Framework with RAG and Memory
InitRunner is a new open-source framework that lets developers define AI agents using simple YAML configuration, including support for RAG, memory management, and API endpoints.
-
GPU-Accelerated DataFrame Library for Local Inference Workloads
A new DataFrame library that runs on GPUs, accelerators, and alternative hardware, enabling efficient data processing for local AI inference pipelines.
-
Microsoft MarkItDown: Document Preprocessing Tool for LLMs
Microsoft releases MarkItDown, a tool that converts various document formats (PDF, HTML, DOCX, PPTX, XLSX, EPUB) to markdown while also supporting audio transcription, YouTube links, and OCR for images.
-
Building a RAG Pipeline on 2M+ Pages: EpsteinFiles-RAG Project
A developer demonstrates building a large-scale RAG (Retrieval-Augmented Generation) pipeline processing over 2 million pages, showcasing advanced techniques for local document processing and retrieval optimization.