Tagged "gguf"
46 articles tagged gguf, 22 February 2026 to 15 October 2026. Newest first.
-
PS5LM: Running GGUF Models Natively on Jailbroken PlayStation 5 Consoles
PS5LM enables native execution of GGUF-formatted LLMs on jailbroken PlayStation 5 hardware, demonstrating practical local inference on consumer gaming hardware. The project achieves speeds up to 124 tokens/second with appropriately quantised models.
-
Liquid AI's LFM2.5-24B-A2B Enables Agent Tasks on Laptop Hardware
Liquid AI releases LFM2.5-24B-A2B, a 24B parameter model optimised for running agentic workflows directly on consumer laptops without cloud dependencies. The model outperforms larger alternatives like Gemma 4 26B and Qwen 3.6 35B while maintaining efficient local inference.
-
Saluki 27B: 2-bit Quantised Qwen Model Outperforms Original at Tool Calling
A new 2-bit quantised variant of Qwen3.8-27B demonstrates superior tool calling capabilities compared to the original model, showing that aggressive quantisation can preserve or improve specific task performance.
-
Janus: New Go Binary Runs GGUF Models via Vulkan on AMD, Intel, and NVIDIA
Janus is a newly released Go binary that enables GGUF model inference through Vulkan, providing cross-platform GPU acceleration for AMD, Intel, and NVIDIA hardware without vendor-specific dependencies.
-
Hugging Face Transformers Now Natively Supports Llama.cpp Quantizations
Hugging Face's transformers library has added native support for llama.cpp GGUF quantisations, eliminating friction when using quantised models in Python workflows. This integration significantly improves accessibility for local LLM deployment.
-
GGUF vs GPTQ vs AWQ vs EXL2: LLM Model Formats Explained
A comprehensive comparison of the major quantization formats used in local LLM deployment, covering GGUF, GPTQ, AWQ, and EXL2 formats and their tradeoffs for on-device inference.
-
Ollama 0.34.1 Stabilizes MLX Backend and GGUF Model Creation
Ollama v0.34.1 releases improved MLX memory handling for Apple Silicon, stabilizes GGUF creation workflows, and enhances repeat token detection for more reliable local inference.
-
Qwen3.8-Flash-Next achieves efficient inference on dual RTX 3090s via non-uniform quantization
A community-optimized GGUF quantization of Qwen3.8-Flash-Next demonstrates that large instruction-tuned models can now run efficiently on accessible consumer hardware through advanced quantization techniques.
-
Per-Tensor Layout Maps for GGUF Quantization
A new quantization optimization technique for GGUF models that enables per-tensor layout customization, improving inference performance and memory efficiency across diverse hardware targets.
-
Per-Tensor Layout Maps for GGUF Quantization
A new quantization approach enables fine-grained control over tensor layout in GGUF format, improving inference efficiency and memory utilization for locally deployed models.
-
Running Qwen3-Omni With Audio and Vision in llama.cpp
One mmproj carries both encoders, --image and --audio are the same flag, and speech output does not work at all. The verified commands, real file sizes and open bugs for the only open-weights omni model.
-
GGUF Quantization: Shrink LLMs 72% in 12 Steps
A practical guide to GGUF quantization techniques that can reduce LLM model sizes by up to 72%, enabling deployment on resource-constrained devices and improving inference speed.
-
Show HN: Single-File GGUF Inference
A browser-based GGUF inference solution enables running quantized models directly in WebAssembly, allowing local LLM inference without any backend server or installation required.
-
Choosing a Qwen3.8-27B Quantization and Backend: What Actually Fits
No Q4_K_M build of Qwen3.8-27B fits in 16GB from any repository, the quant that does fit has never been quality-tested, and llama.cpp silently stops generating at ~98K context. The measured file sizes and the open bugs behind each decision.
-
How to Run Qwen3.8-27B on a Single 16GB Card
Practical guide demonstrating techniques to fit the 27-billion parameter Qwen3.8 model within 16GB VRAM constraints using llama.cpp, quantization, and RTX 3080 optimizations.
-
Qwen3.8-Flash-Next Added to llama.cpp with GGUF Support
llama.cpp now supports Qwen3.8-Flash-Next with full GGUF architecture implementation, including low-rank hyper-connections and n-gram hash embeddings for optimized local inference.
-
Benchmarking Qwen3.8 27B Quantizations: 4-bit Shows Strong Performance, 1-bit Collapses
Detailed quantization benchmarks for Qwen3.8 27B reveal that 4-bit quantization maintains strong performance while 1-bit variants suffer significant degradation, providing practical guidance for local deployment scenarios.
-
Liquid AI Releases LFM2.5-DSpark Draft Models with 3.18x Faster Decoding
Liquid AI introduces speculative decoding models that achieve up to 3.18x faster inference without changing model outputs, significantly improving local LLM performance.
-
Liquid AI Releases LFM2.5 Q4_0 Checkpoints from Quantization-Aware Distillation
Liquid AI publishes LFM2.5 Q4_0 quantized checkpoints trained with quantization-aware distillation, enabling efficient local inference with maintained model quality. This approach combines distillation and quantization for optimal compression.
-
GGUF Quantization Deep Dive: Q4_K_M vs IQ4_XS vs IQ4_NL Performance
A comprehensive analysis compares different GGUF quantization formats, evaluating trade-offs between model quality, inference speed, and memory consumption for practical local LLM deployment decisions.
-
DeepSeek V4 Flash Shrunk to 57GB for Local macOS Inference with Compiler Generation
A community contributor has quantized DeepSeek V4 Flash to 57GB, enabling capable inference on Apple Silicon Macs with demonstrated ability to generate production-quality code. This showcases aggressive quantization techniques making frontier-grade models feasible on personal devices.
-
Qwen3.8-27B Surpasses 1 Million Downloads, Overseas Developers Race to Maximize Local Deployment
Alibaba's Qwen3.8-27B model has exceeded 1 million downloads within two weeks of its open-source release, with developers globally competing to optimize its performance for local deployment. This rapid adoption demonstrates strong community interest in accessible, high-quality models that can run on consumer hardware.
-
Unsloth Releases Qwen 3.8 27B GGUF Quantised Weights
Unsloth has published optimised GGUF format weights for Qwen 3.8 27B, enabling efficient local deployment with pre-quantised models that balance quality and memory footprint for consumer hardware.
-
Qwen 3.8 27B Successfully Runs on 16GB RAM Using LM Studio
Community testing confirms Qwen 3.8 27B operates efficiently on 16GB systems with LM Studio, making a capable 27-billion parameter model accessible to users with modest hardware. Quantised GGUF weights enable practical local deployment without expensive GPUs.
-
AMD Optimizes Qwen 3.8 27B for Ryzen AI Max and Radeon GPUs
AMD announces native support for running Qwen 3.8 27B on Ryzen AI Max processors and Radeon GPUs, enabling high-performance local inference on consumer AMD hardware.
-
Hugging Face State of Open Models: Summer 2026 Observations
Hugging Face publishes comprehensive analysis of the open model landscape in Summer 2026, documenting trends in model optimization, deployment patterns, and ecosystem maturation for local LLM inference.
-
Running DeepSeek's 284B LLM on a Laptop: Quantisation and GGUF Optimization
Practitioners demonstrated running DeepSeek's massive 284B parameter model locally on consumer laptops through aggressive quantisation and GGUF format optimization, showing feasibility of ultra-large model local inference.
-
llama.cpp Improves Muse Glimmer Tool Calling with Latest Update
The latest llama.cpp release (b10380) fixes critical tool calling behavior in Muse Glimmer models, ensuring proper handling of multiple tool invocations and preventing content swallowing issues. This update is essential for reliable agent-based local inference.
-
Show HN: GGUFun, Play Snake and a Simple Maze on Ollama Using Hand Crafted GGUFs
A creative demonstration of running game logic directly on Ollama using custom GGUF quantized models. This shows innovative approaches to local inference beyond traditional language understanding tasks.
-
What else is included in the 'GGUF' file format used by llama.cpp for AI language models, besides weights?
An in-depth technical analysis of the GGUF format ecosystem, exploring the metadata, configuration, and structural components beyond model weights. Understanding GGUF is essential for practitioners working with llama.cpp and quantized model deployment.
-
llama.cpp GGUF Parser Flaws: Critical Integer Overflow Enables Arbitrary Reads in Every Local AI Stack
A critical security vulnerability discovered in llama.cpp's GGUF parser threatens the integrity of local LLM deployments. The flaw allows attackers to read arbitrary memory through malicious model files.
-
Ollama Out-of-Bounds Read Vulnerability Allows Remote Process Memory Leak
A critical vulnerability in Ollama's GGUF parser enables remote attackers to read sensitive process memory, potentially exposing model weights and user data. This vulnerability affects all versions of Ollama and requires immediate patching for production deployments.
-
Malicious GGUF Models Could Trigger Remote Code Execution on SGLang Servers
Security researchers have identified a critical vulnerability where specially crafted GGUF model files can achieve remote code execution on SGLang inference servers, posing significant risks to organizations running local LLM deployments.
-
MiniMax M2.7 GGUF Investigation Reveals NaN Issues Affecting 21-38% of Hugging Face Conversions
Investigation into MiniMax-M2.7 GGUF quantizations found perplexity calculation errors affecting up to 38% of community GGUF uploads on Hugging Face, signaling broader quantization quality issues in the ecosystem.
-
Fine-Tuned Qwen3.5-0.8B for OCR Outperforms Previous 2B Release
A developer released an improved fine-tuned version of Qwen3.5-0.8B optimized for OCR tasks, surpassing the performance of their earlier 2B model with better training data and inference efficiency.
-
Unsloth Completes Comprehensive MiniMax M2.7 GGUF Quantization Suite
Unsloth has finished quantizing MiniMax M2.7 across the full range of GGUF quantization levels from 1-bit to BF16, providing practitioners with optimized variants for every hardware configuration from edge devices to high-end systems.
-
ByteShape Releases Qwen 3.5 9B Quantisations with Hardware-Matched Tuning Guide
ByteShape has released optimised GGUF quantisations of Qwen 3.5 9B with a comprehensive guide for selecting the best quantisation level for specific hardware. The resource includes comparative benchmarks against other popular quantisation approaches, enabling practitioners to make informed deployment decisions.
-
Coding Implementation to Run Qwen3.5 Reasoning Models Distilled With Claude-Style Thinking Using GGUF and 4-Bit Quantization
A new implementation enables running distilled Qwen3.5 reasoning models with 4-bit quantization and GGUF format, making advanced reasoning capabilities accessible on consumer hardware. This combines distillation, quantization, and standardized formats for practical local deployment.
-
Qwen 3.5 122B Uncensored (Aggressive) Released with New K_P Quantisations
The highly anticipated Qwen 3.5 122B uncensored variant has been released in GGUF format with new K_P quantisation options. This aggressive version removes all refusals while maintaining the original model's capabilities, making it immediately deployable on consumer hardware.
-
Qwen 3.5-35B Uncensored GGUF Models Now Available
Community releases optimized GGUF quantizations of Qwen 3.5-35B uncensored variants, enabling local deployment without refusal mechanisms. Multiple quantization levels tested on consumer GPUs.
-
Final Qwen3.5 Unsloth GGUF Update with Improved Size/Quality Tradeoffs
Unsloth releases final GGUF quantizations for Qwen3.5-122B-A10B and Qwen3.5-35B-A3B with optimized size/KL divergence tradeoffs at 99.9% quality retention. This represents a significant milestone in making large models efficiently deployable locally.
-
Qwen 3.5-27B Q4 Quantization Comparison and Analysis
Community-driven quantization sweep compares multiple GGUF quantization approaches for Qwen 3.5-27B, providing data-driven guidance for selecting optimal quantization formats.
-
Unsloth Dynamic 2.0 GGUFs
Unsloth releases Dynamic 2.0 GGUF format models, advancing quantized model optimization for local inference with improved efficiency and compatibility across edge devices.
-
Qwen3.5-35B Unsloth Dynamic GGUFs Achieve SOTA Across Nearly All Quantisation Levels
New state-of-the-art GGUF quantisations for Qwen3.5-35B released with 150+ KL Divergence benchmarks and 9TB of variants. Critical tool calling chat template bug fixed affecting all quantisation uploaders.
-
nanollama: Open-Source Framework for Training Llama 3 from Scratch with One-Command GGUF Export
nanollama enables full Llama 3 pretraining from scratch (not fine-tuning) with single-command execution and direct GGUF export compatible with llama.cpp, democratizing custom model development for local deployment.
-
Ouro 2.6B Thinking Model GGUFs Released with Q8_0 and Q4_K_M Quantization
Ouro 2.6B, a looped inference model, is now available as quantized GGUFs (Q8_0 at 2.7GB and Q4_K_M at 1.6GB) compatible with LM Studio, Ollama, and llama.cpp. This enables accessible local deployment of an innovative thinking model architecture.