Tagged "qwen"
151 articles tagged qwen, 12 February 2026 to 19 October 2026. Newest first.
-
Jailbroken PS5 Runs Qwen 3.8 27B Model at 20 tok/s via 'PS5LM Plus' llama.cpp Port
A custom llama.cpp implementation enables inference of large Qwen models on PlayStation 5 hardware, demonstrating creative hardware repurposing and achieving competitive inference speeds on consumer gaming consoles.
-
Liquid AI's LFM2.5-24B-A2B Enables Agent Tasks on Laptop Hardware
Liquid AI releases LFM2.5-24B-A2B, a 24B parameter model optimised for running agentic workflows directly on consumer laptops without cloud dependencies. The model outperforms larger alternatives like Gemma 4 26B and Qwen 3.6 35B while maintaining efficient local inference.
-
Saluki 27B: 2-bit Quantised Qwen Model Outperforms Original at Tool Calling
A new 2-bit quantised variant of Qwen3.8-27B demonstrates superior tool calling capabilities compared to the original model, showing that aggressive quantisation can preserve or improve specific task performance.
-
Ollama v0.40.0 Brings Multimodal Embeddings and MLX Default on Apple Silicon
Ollama's latest release makes MLX the default runtime for supported models on Apple Silicon and implements native multimodal embedding support through EmbeddingGemma2Model architecture. This brings efficient text, audio, and video embeddings to on-device deployment with significantly reduced memory footprint.
-
Fine-Tuned Qwen 1.5B Achieves GPT-4o Level Bash Generation Performance
A practitioner successfully fine-tuned the 1.5B Qwen model to match GPT-4o performance on bash command generation, demonstrating that smaller local models can be optimized for specific coding tasks. This shows the viability of task-specific fine-tuning for edge deployment.
-
llama.cpp Fixes K-Pool Graph Reallocation: Preventing Decode-Time Performance Regressions
llama.cpp release b11412 fixes an unexpected graph reallocation issue in k-pool models that was causing decode-time performance degradation, particularly affecting recent models like Qwen and GLM variants.
-
Ollama v0.40.0: MLX Runtime Now Default on Apple Silicon with Decision Model Support
Ollama's latest release automatically routes supported model architectures to the MLX runtime on Apple Silicon devices, improving performance. The release also introduces support for decision models, expanding the types of AI workloads suitable for local deployment.
-
Qwen 3.8 27B Runs at High Speed on 16GB VRAM with Quantization and Local Model Support
A successful test of the Hermes Agent with Qwen 3.8 27B demonstrates efficient local inference, achieving fast performance on modest hardware through effective quantization techniques.
-
Benchmarking Qwen3.8 27B Quantizations: 4-bit Holds Up, 1-bit Collapses
Detailed quantization benchmarks for Qwen3.8 27B reveal that 4-bit quantization maintains strong performance while extreme 1-bit quantization severely degrades output quality, providing practical guidance for practitioners choosing compression levels.
-
Benchmarking Qwen 3.8 27B Quantizations: 4-Bit Holds Up, 1-Bit Collapses
Detailed quantization benchmarks for Qwen 3.8 27B revealing how 4-bit quantization maintains model quality while 1-bit approaches fail significantly.
-
Benchmarking Qwen 3.8 27B on RTX 5090 and Beyond
Comprehensive performance benchmarking of Qwen 3.8 27B model on high-end consumer GPUs like the RTX 5090, providing practical insights for local deployment scenarios.
-
Alibaba Releases Qwen3.8 Flash Next for Local Deployment
Alibaba's Qwen3.8 Flash Next provides a lightweight, optimized model for on-device inference with previews of the more capable Qwen4 architecture.
-
MoE Expert Offload: What a 35B Model Actually Costs on a 12GB Card
92.9% of Qwen3.6-35B-A3B is routed expert weights, and only 3.1% of them are read per token — which is why a 19 GiB model runs on 12 GB at all. The roofline arithmetic for expert offload, and why the same sum that permits 50 tok/s at 8K refuses it at 128K.
-
Llama.cpp Fork Enables Qwen 3.8 27B with Large Contexts on 16GB VRAM GPUs
A specialized llama.cpp fork implements adaptive KV streaming to run Qwen 3.8 27B with large context windows on 16GB VRAM GPUs, significantly reducing hardware requirements for production-grade inference.
-
Choosing a Qwen3.8-27B Quantization and Backend: What Actually Fits
No Q4_K_M build of Qwen3.8-27B fits in 16GB from any repository, the quant that does fit has never been quality-tested, and llama.cpp silently stops generating at ~98K context. The measured file sizes and the open bugs behind each decision.
-
FreeToken: Edge-Native MoE Serving with CPU-GPU Co-Execution
FreeToken is an open-source engine for running 290B+ Mixture-of-Experts models locally on consumer hardware through bandwidth-adaptive CPU-GPU co-execution, with elastic memory management, expert caching, and support for DeepSeek, Qwen and GLM models across NVIDIA RTX 30/40/50 series.
-
Llama.cpp Fork Enables Qwen 3.8 27B with Large Contexts on 16GB VRAM
A specialized llama.cpp implementation adds adaptive KV-cache streaming to run Qwen 3.8 27B with large context windows on 16GB GPUs, demonstrating significant memory optimization advances.
-
What Actually Fits on Dual RTX 3090s: Qwen 27B and the KV Cache Math
Why a 27B model holds 262K context on 48GB — 48 of its 64 layers have no KV cache at all — and what decode speed you should honestly expect from two 3090s.
-
Ollama v0.33.1 Adds Qwen3.8-Flash-Next Support via MLX Backend
Ollama's latest release includes native Qwen3.8-Flash-Next support through its MLX backend, along with structured output capabilities and Metal GPU optimizations for macOS users.
-
IBM Releases Granite 4.2 Models Optimized for Local LLM Deployment
IBM's new Granite 4.2 model series addresses the growing market demand for locally-deployable open-source language models with improved efficiency and performance characteristics.
-
Qwen3.8 27B Quantization Benchmarks: 4-Bit Remains Optimal Trade-off
New quantization benchmarks for Qwen3.8 27B show that 4-bit quantization maintains excellent quality, while 1-bit approaches suffer significant quality collapse, providing crucial guidance for local deployment decisions.
-
Qwen3.8-Flash-Next Added to llama.cpp with GGUF Support
llama.cpp now supports Qwen3.8-Flash-Next with full GGUF architecture implementation, including low-rank hyper-connections and n-gram hash embeddings for optimized local inference.
-
Ollama v0.33.1 Adds Qwen3.8 Flash Next Support and Claude Desktop Integration
Ollama releases v0.33.1 with native support for Qwen3.8 Flash Next, enabling seamless integration with Claude Desktop as a third-party gateway provider. This update improves caching and resolves stability issues with long prefills.
-
Benchmarking Qwen3.8 27B Quantizations: 4-bit Shows Strong Performance, 1-bit Collapses
Detailed quantization benchmarks for Qwen3.8 27B reveal that 4-bit quantization maintains strong performance while 1-bit variants suffer significant degradation, providing practical guidance for local deployment scenarios.
-
Qwen 3.6 Now Easier to Run Locally on Mac with JetBrains Integration
JetBrains has released tooling that makes it significantly easier to run Qwen 3.6 models locally on macOS, reducing friction for developers wanting to deploy cutting-edge models on consumer hardware.
-
Strong Domain Adaptation Results with Qwen 3 4B Fine-Tuning
A practitioner achieved good results fine-tuning Qwen 3 4B to learn specialized domain knowledge, showing that small quantised models can be effectively adapted for specific use cases without requiring massive compute.
-
Teaching a Local LLM to Reason About a New Domain Through Continued Pretraining
A practical guide demonstrating how to adapt local LLMs like Qwen 3 4B to specialized domains using continued pretraining, with evidence of significant capability gains. This approach enables cost-effective domain customization without requiring cloud resources.
-
Qwen3.8-27B: Running a Frontier-class Open Model on Your Local GPU
A comprehensive guide to deploying Qwen3.8-27B, a frontier-class open model, on consumer GPUs with practical optimization techniques for local inference.
-
Qwen3.8-27B Matches Claude Opus 4.6 on Coding, Runs on Consumer GPUs
Alibaba's Qwen3.8-27B model achieves performance parity with Claude Opus 4.6 on coding benchmarks while remaining deployable on consumer-grade GPUs, representing a major milestone for affordable local LLM inference.
-
The Qwen MLX Challenge
A new challenge focused on optimizing Qwen models for Apple MLX framework. This initiative targets efficient inference on Apple Silicon hardware, bringing competitive incentives to local deployment optimization.
-
Google Pixel 11 Launches With Faster On-Device Gemini at $899 Starting Price
Google's Pixel 11 ships with improved on-device Gemini inference, indicating major investments by consumer electronics manufacturers in local LLM deployment. This signals mainstream acceptance of edge inference as a key feature.
-
Unsloth Releases Qwen 3.8 27B GGUF Quantised Weights
Unsloth has published optimised GGUF format weights for Qwen 3.8 27B, enabling efficient local deployment with pre-quantised models that balance quality and memory footprint for consumer hardware.
-
Qwen 3.8 27B Successfully Runs on 16GB RAM Using LM Studio
Community testing confirms Qwen 3.8 27B operates efficiently on 16GB systems with LM Studio, making a capable 27-billion parameter model accessible to users with modest hardware. Quantised GGUF weights enable practical local deployment without expensive GPUs.
-
Ollama Adds Qwen 3.8 27B with Optimised Apple Silicon Support
Ollama v0.32.12 now supports Qwen 3.8 27B, a 27-billion parameter model optimised for local deployment with special tuning for Apple Silicon devices. The model delivers substantial improvements in coding, professional work, and agentic tasks while running efficiently on consumer hardware.
-
Ollama Adds Qwen 3.8 27B with Apple Silicon Optimizations
Ollama v0.32.12 now supports Qwen 3.8 27B, a new open-source model with substantial improvements in coding, professional work, and agentic tasks. The release includes special optimizations for Apple Silicon devices to maximize performance and output quality.
-
AMD Optimizes Qwen 3.8 27B for Ryzen AI Max and Radeon GPUs
AMD announces native support for running Qwen 3.8 27B on Ryzen AI Max processors and Radeon GPUs, enabling high-performance local inference on consumer AMD hardware.
-
Minisforum N5 Max: Running Qwen 27B Locally with Open WebUI and Ollama
A practical guide to running large open-source models like Qwen 27B on compact edge hardware using Open WebUI and Ollama. This demonstrates viable deployment of substantial models on small form-factor devices.
-
Optimizing Qwen 3.6 for Local Development: A Developer's Guide
A practical developer guide for optimizing the Qwen 3.6 model specifically for local development environments, covering configuration and performance tuning.
-
Build Self-Scaling OCR Pipeline with Qwen 3.5 and Kubernetes
A production-ready course demonstrates deploying Qwen 3.5 for OCR workloads with Kubernetes auto-scaling, bridging the gap between local inference and distributed edge deployment.
-
Qwen 3.8 with 2.4T Parameters Going Open-Weight Soon
Alibaba announced Qwen 3.8, a massive 2.4 trillion parameter model that will be released as open-weight, significantly expanding options for self-hosted large-scale LLM deployment.
-
Qwen and Fable: Open-Weights 35B Mixture-of-Experts Agentic Coding Model
A new open-weights 35B Mixture-of-Experts model combining Qwen and Fable for agentic coding tasks, optimized for local deployment with improved efficiency through sparse computation patterns.
-
RTX 5080 and RTX 3090 Setup Achieves 80 Tok/s on Qwen 3.6 27B Q8
A practical benchmark demonstrating impressive inference throughput using dual NVIDIA GPUs running quantized Qwen 3.6 27B model. This setup showcases real-world performance metrics for local LLM deployment on consumer-grade hardware.
-
110 Tokens/Second on RTX 4070 Super with Qwen 3.6 35B
A significant performance benchmark demonstrates that consumer-grade GPUs can achieve excellent inference speeds with optimized models, enabling practical local deployment of 35B parameter models.
-
I Stopped Trying to Replace My Cloud LLMs, and Local Models Finally Made Sense
A practitioner shares insights on when and why local LLMs become practical replacements for cloud APIs, moving beyond the hype to focus on real-world use cases and total cost of ownership. The piece highlights recent improvements in inference speed and model quality that have shifted the economics.
-
llama.cpp Adds Multi-Token Prediction, Doubles Qwen 3.6B Throughput for Local Inference
llama.cpp, the popular C++ inference engine for local LLMs, has added multi-token prediction capabilities and achieved a 2x throughput improvement on Qwen 3.6B models. This breakthrough enables faster token generation for on-device deployments without sacrificing accuracy.
-
Continue.dev for Developers: Complete Local AI Coding Assistant Setup
A detailed guide to setting up Continue.dev, an open-source IDE extension framework for deploying local AI coding assistants. The guide covers configuration with self-hosted models and integration best practices.
-
Qwen3-Coder-Next Local Deployment: Complete Developer Guide for 2026
A comprehensive guide for deploying Qwen3-Coder-Next, a state-of-the-art coding model optimized for local environments. The guide covers setup, configuration, and practical deployment strategies for developers.
-
Using a Local LLM as a Zero-Shot Classifier
Detailed guide demonstrating how to leverage locally-running language models for zero-shot text classification tasks without fine-tuning, reducing infrastructure costs and inference latency.
-
DFlash Doubles Token Generation Speed of Qwen3.5 27B on Mac M5 Max
New DFlash support in oMLX 0.3.5 RC1 achieves 2x speedup for Qwen3.5 27B inference on Apple Silicon, reaching 22 T/S from 9 T/S using speculative decoding with draft models.
-
Copilot Rate-Limiting Issues Highlight Cloud AI Service Limitations
Users report severe rate-limiting issues with Copilot Pro+, with some facing wait times exceeding 181 hours. These incidents underscore the reliability challenges of cloud-dependent AI services and the value proposition of local alternatives.
-
Fine-Tuned Qwen3.5-0.8B for OCR Outperforms Previous 2B Release
A developer released an improved fine-tuned version of Qwen3.5-0.8B optimized for OCR tasks, surpassing the performance of their earlier 2B model with better training data and inference efficiency.
-
Qwen 3.5 Small – On-Device Multimodal Models Released
Alibaba's Qwen team has released Qwen 3.5 Small, a new multimodal model optimized for on-device inference. This lightweight model enables local deployment of vision and language capabilities without cloud dependencies.
-
Google Gemma 4 Delivers Exceptional Speed and Accuracy for Local Inference
Early adopters report that Google's Gemma 4 model runs with remarkable speed comparable to 4-9B parameter models while maintaining accuracy levels reminiscent of early Gemini releases, making it a compelling option for resource-constrained local deployments.
-
DFlash Speculative Decoding Achieves 3.3x Speedup on Apple Silicon
A native MLX implementation of DFlash speculative decoding reaches 85 tokens/second on Qwen 3.5-9B running on Apple M5 Max, delivering a 3.3x performance boost through parallel draft token generation and single-pass verification.
-
Gemma 4 31B vs Qwen 3.5 27B: Comprehensive Long Context Benchmark
Community benchmark comparing Gemma 4 31B and Qwen 3.5 27B for long context workloads on 24GB VRAM, establishing these as the top local models for mid-range GPU setups.
-
Intel Arc Pro B70 32GB Achieves 12 Tokens/Sec on Qwen 3.5-27B
Intel Arc Pro GPU hardware demonstrates strong performance running Qwen 3.5 27B quantized models with vLLM and llama.cpp, establishing alternative hardware viability for local deployment.
-
Qwen 3.5 122B Achieves 198 Tokens/sec on Dual RTX PRO 6000 Blackwell GPUs
A detailed optimization case study demonstrates running Qwen 3.5 122B at impressive inference speeds on a budget dual-GPU Blackwell setup. The community shares verified benchmarks with full methodology and reproducible results for large-scale local deployment.
-
Run Qwen3.5 on an Old Laptop: A Lightweight Local Agentic AI Setup Guide
KDnuggets publishes a practical guide demonstrating how to run Qwen3.5 with agentic AI capabilities on resource-constrained hardware, making advanced local inference accessible to resource-limited environments.
-
Gemini-CLI, Llama.cpp, and Qwen3.5 Running on NVIDIA Jetson TK1
Community members report successfully running multiple LLMs including Qwen3.5 and Gemini models via llama.cpp on NVIDIA Jetson TK1 edge devices, showcasing practical deployment on resource-constrained embedded hardware.
-
I Replaced My Local LLM With a Model Half Its Size and Got Better Results — and It Wasn't About the Parameters
A detailed account of how switching to a smaller, better-optimized model outperformed a larger predecessor on local hardware, challenging assumptions about model scaling and practical performance.
-
Quantization Strategy Comparison: Balancing Quality and Speed on Consumer Laptops
Detailed benchmarking of different GGUF quantization methods for Qwen 3.5 4B on Intel Lunar Lake iGPU reveals optimal compression strategies for small model deployment on resource-constrained hardware.
-
Qwen 3.6 Free Model Available via OpenRouter
Alibaba's Qwen 3.6 model is now available as a free inference option, providing accessible baseline for local LLM practitioners evaluating model quality and performance. This release expands the ecosystem of deployable models with strong performance-to-cost ratios.
-
Apple Research Shows Self-Distillation Significantly Improves Local Code Generation
A new Apple research paper demonstrates that embarrassingly simple self-distillation techniques can meaningfully improve code generation quality in smaller language models, with implications for on-device coding assistants.
-
Qwen 3.5 397B Reduced to 35% Parameters With Usable Quality on 96GB GPU
A community researcher successfully compressed Qwen 3.5 397B to 35% of its original size while maintaining practical quality, enabling the model to run on dual GPU setups. The REAP35 variant demonstrates advanced parameter reduction techniques for enterprise-scale model deployment.
-
Gemma 4 31B Achieves Third Place on FoodTruck Bench, Beating Larger Models
Google's Gemma 4 31B model has demonstrated exceptional performance on the FoodTruck Bench, ranking third and outperforming significantly larger models like GLM 5 and Qwen 3.5 397B. The result highlights major improvements in long-horizon task handling for locally deployable models.
-
Gemma 4 26B A4B Outperforms Qwen 3.5 35B on Apple Silicon
Testing on Mac Studio M5 Ultra shows Gemma 4 26B achieves comparable speed (1000 tokens/sec prompt, 60 tokens/sec generation) to larger Qwen 3.5 35B while demonstrating significantly better output quality and reasoning behavior.
-
Google Gemma 4 Released with GGUF Quantizations
Google has released Gemma 4 with multiple model sizes (26B, 31B variants) already quantized in GGUF format by Unsloth, enabling immediate local deployment on consumer hardware.
-
Qwen 3.6-Plus Released
Alibaba releases Qwen 3.6-Plus, a new model optimized for local deployment with improved performance characteristics for on-device inference.
-
TurboQuant Enables Qwen 3.5-27B on 16GB Consumer GPUs
Advanced quantization technique TurboQuant achieves near-Q4_0 quality at 10% smaller size, allowing high-performance models to fit on consumer-grade graphics cards.
-
Claude Code Source Leaked: Community Extracts Multi-Agent Orchestration Framework
Claude Code's source code was exposed via npm source maps, revealing 500K+ lines of TypeScript. Community developers have already extracted the multi-agent orchestration architecture and released it as an open-source framework compatible with any LLM, democratising advanced agentic capabilities for local deployment.
-
Qwen 3.5-27B Demonstrates Superior Performance vs Gemini 3.1 Pro and GPT-5.3
Community benchmarks show Qwen3.5-27B outperforming larger closed-source models in practical scenarios, particularly for code tasks. The open model's availability and performance characteristics make it an attractive option for local deployment when considering capability-per-resource tradeoffs.
-
ByteShape Releases Qwen 3.5 9B Quantisations with Hardware-Matched Tuning Guide
ByteShape has released optimised GGUF quantisations of Qwen 3.5 9B with a comprehensive guide for selecting the best quantisation level for specific hardware. The resource includes comparative benchmarks against other popular quantisation approaches, enabling practitioners to make informed deployment decisions.
-
GLM-5.1 Model Weights Launching Early April for Local Deployment
Zhipu AI has announced the upcoming release of GLM-5.1 model weights on April 6-7, bringing a new open-weight option to the local LLM community. This release adds another competitive choice alongside Qwen and other open models for on-device inference.
-
M5 Max Delivers 1.7x Faster Inference Than M3 Max on Qwen 3.5 Models
Comprehensive benchmarks comparing Apple's M5 Max and M3 Max chips show significant performance gains across Qwen 3.5 model variants (27B dense, 35B MoE, 122B MoE), with the newer chip delivering 1.4x to 1.7x faster token generation using the oMLX framework.
-
TurboQuant KV Cache Compression Achieves 22.8% Faster Decoding at 32K Context
Google's TurboQuant compression method has been successfully integrated into llama.cpp, enabling 4.6x KV cache compression and 22.8% decode speedup at 32K context length by skipping 90% of dequantization work. This breakthrough makes long-context inference practical on consumer hardware like MacBook Air M4.
-
Coding Implementation to Run Qwen3.5 Reasoning Models Distilled With Claude-Style Thinking Using GGUF and 4-Bit Quantization
A new implementation enables running distilled Qwen3.5 reasoning models with 4-bit quantization and GGUF format, making advanced reasoning capabilities accessible on consumer hardware. This combines distillation, quantization, and standardized formats for practical local deployment.
-
Homelab Consolidation: Replacing 3 Models with Single 122B MoE Model on AMD Ryzen AI MAX+
A homelabber consolidated their inference setup from three separate models down to a single 122B mixture-of-experts model on consumer hardware (Ryzen AI MAX+ 395 with 128GB RAM), providing detailed benchmarks and practical insights on model consolidation strategy.
-
Qwen 3.5 27B Achieves 1.1M Tokens/Second on B200 GPUs with Optimized vLLM Config
A developer optimized Qwen 3.5 27B to reach 1.1 million tokens per second on 96 B200 GPUs using vLLM, with detailed configurations and all settings published on GitHub. Key optimizations included distributed parallelism, reduced context windows, FP8 KV cache, and speculative decoding.
-
Intel Launches Arc Pro B70/B65 with 32GB VRAM for Local AI Inference
Intel has released the Arc Pro B70 and B65 GPUs with 32GB GDDR6 memory at competitive pricing, offering 608 GB/s bandwidth and 290W power consumption. The hardware is positioned as an affordable option for running quantized local LLMs like Qwen 3.5 27B.
-
Chinese LLM Ecosystem Landscape: ByteDance Doubao, Alibaba, and Open-Source Competition
Comprehensive analysis of the Chinese LLM scene reveals ByteDance's Doubao as the market leader with strong open-source alternatives from Alibaba, Deepseek, and others, highlighting the rapid innovation and diverse model ecosystem emerging from China's AI development.
-
Qwen3.5-27B Emerges as Sweet Spot for Single-GPU Local Deployment
Community enthusiasm peaks for Qwen3.5-27B as the optimal model size for single-GPU users with 24GB+ VRAM, with multiple appreciation posts and emerging fine-tunes showing strong performance on reasoning tasks at efficient token generation rates.
-
Alibaba Commits to Continuous Open-Sourcing of Qwen and Wan Models
Alibaba has publicly committed to ongoing open-source releases of new Qwen and Wan models, reinforcing their position as a major contributor to the local LLM ecosystem. This commitment ensures continued availability of high-quality open-weight models for on-device deployment.
-
Qwen 3.5 Models: Optimal Settings and Reduced Overthinking Configuration
Community exploration of Qwen 3.5 (35B and 27B) model settings and prompts reveals configurations that minimize overthinking behavior and excessive reasoning token usage. These practical optimizations help practitioners maximize output quality and inference speed.
-
Qwen 3.5 122B Uncensored (Aggressive) Released with New K_P Quantisations
The highly anticipated Qwen 3.5 122B uncensored variant has been released in GGUF format with new K_P quantisation options. This aggressive version removes all refusals while maintaining the original model's capabilities, making it immediately deployable on consumer hardware.
-
Nvidia Nemotron Cascade 2 30B Emerges as Powerful Alternative to Qwen Models
Nvidia's newest Nemotron Cascade 2 30B model offers a distinct non-Qwen architecture option for local deployment with competitive performance characteristics. Early community testing suggests this model deserves attention alongside the popular Qwen family.
-
ik_llama.cpp Fork Delivers 26x Faster Prompt Processing on Qwen 3.5 27B
A fork of llama.cpp called ik_llama.cpp is delivering dramatic 26x speed improvements for prompt processing on Qwen 3.5 27B models. Real-world benchmarks on Blackwell RTX PRO GPUs show tangible performance gains for production agentic workloads.
-
Qwen 3.5 397B emerges as top-performing local coding model
Users report that Qwen 3.5 397B significantly outperforms competing local models including GPT-OSS 120B and Nemotron 120B for code generation tasks, despite slower inference speeds.
-
Multi-Token Prediction support coming to MLX-LM for Qwen 3.5
Early support for Multi-Token Prediction (MTP) is being integrated into MLX-LM, enabling Qwen 3.5 to generate multiple tokens per forward pass with reported performance gains from 15.3 to 23.3 tokens per second.
-
Qwen 3.5 Emerges as Top Performer for Local Deployment with Extensive Quantization Options
Qwen 3.5 is establishing itself as a highly versatile model for local inference, with community members successfully creating dozens of custom quantizations and sharing best practices across different inference engines and hardware configurations.
-
Community Converges on Optimal KV Cache Quantization Strategies for Qwen 3.5 Models
The local LLM community is establishing practical guidelines for KV cache quantization with Qwen 3.5, balancing memory savings against accuracy loss to optimize inference on consumer hardware.
-
NVIDIA Nemotron Cascade 2 30B Delivers 120B-Class Performance in Compact Form Factor
NVIDIA's new Nemotron Cascade 2 30B achieves competitive performance with models 4x larger on math and code benchmarks, offering excellent efficiency for local deployment on resource-constrained hardware.
-
Local Qwen Models Master Browser Automation Through Iterative Replanning
Demonstration shows small local Qwen models (8B + 4B) dramatically improve browser automation accuracy by adopting a step-by-step replanning approach rather than generating full multi-step plans upfront.
-
Qwen 3.5 4B Outperforms Nvidia Nemotron 3 4B in Local Benchmarks
Community benchmarking reveals that Qwen 3.5 4B consistently outperforms Nvidia's newly released Nemotron 3 4B across demanding custom tests, challenging expectations for the Nemotron family.
-
Qwen 3.5 122B Demonstrates Exceptional Reasoning for Local Deployment
Qwen 3.5 122B is impressing local LLM enthusiasts with sophisticated reasoning capabilities and natural task decomposition, making it a strong candidate for on-device applications requiring complex problem-solving.
-
Practical Fix for Qwen 3.5 Overthinking in llama.cpp
Community members share techniques to mitigate Qwen 3.5's verbose internal reasoning loops, offering practical optimization strategies for controlling model behavior in local inference environments.
-
Fine-Tuned 14B Model Outperforms Claude Opus 4.6 on Ada Code Generation
A developer successfully fine-tuned QWEN 2.5-Coder-14B using compiler-verified Ada code, demonstrating that smaller specialized models can exceed state-of-the-art performance on domain-specific programming tasks.
-
Achieving 2000 Tokens Per Second with QWEN 3.5 27B on RTX-5090
A practitioner shares real-world performance benchmarks achieving 2000 TPS with QWEN 3.5 27B optimized for document classification workloads on consumer-grade RTX-5090 hardware.
-
Runpod Report: Qwen Has Overtaken Meta's Llama As The Most-Deployed Self-Hosted LLM
According to Runpod data, Qwen models have surpassed Llama as the most popular choice for self-hosted LLM deployments, signaling a major shift in the local AI ecosystem.
-
Intel Updates LLM-Scaler-vLLM With Support For More Qwen3/3.5 Models
Intel has expanded LLM-Scaler-vLLM compatibility to include additional Qwen3 and Qwen3.5 models, improving inference optimization for self-hosted deployments on Intel hardware.
-
Simple Layer Duplication Technique Achieves Top Open LLM Leaderboard Performance
Researchers demonstrate that duplicating middle layers in Qwen2-72B without modifying weights produces state-of-the-art benchmark results, challenging conventional understanding of model optimization.
-
Experiment: 0.8B Model Self-Improvement on MacBook Air Yields Surprising Results
Researcher demonstrates that ultra-small quantized language models can improve themselves through iterative problem-solving on consumer hardware like MacBook Air with minimal RAM requirements.
-
Qwen 3.5-35B Uncensored GGUF Models Now Available
Community releases optimized GGUF quantizations of Qwen 3.5-35B uncensored variants, enabling local deployment without refusal mechanisms. Multiple quantization levels tested on consumer GPUs.
-
Qwen 3.5 Ultra-Compact Models Enable On-Device AI from Watches to Gaming
The latest Qwen 3.5 lineup, including the 0.8B variant, demonstrates that state-of-the-art small language models can now run on severely constrained devices while maintaining impressive capabilities, from vision tasks to game-playing agents.
-
Fine-Tuned Qwen SLMs (0.6–8B) Demonstrate Competitive Performance Against Frontier LLMs on Specialized Tasks
A systematic benchmarking study shows that properly fine-tuned Qwen3 small language models can match or exceed the performance of frontier LLMs like GPT-5 and Claude on narrowly-scoped tasks, validating the viability of local model specialization strategies.
-
Qwen 3.5 Small Expands On-Device AI to Phones and IoT with Offline Support
Alibaba's Qwen 3.5 Small model brings efficient LLM inference to mobile devices and IoT hardware with full offline capabilities. This lightweight model expansion enables practical on-device deployment where connectivity and compute resources are severely constrained.
-
Strix Halo (Ryzen AI Max+ 395) Achieves Strong Local Inference Performance with ROCm 7.2
New benchmarks on AMD's Strix Halo platform with ROCm 7.2 backend show practical inference speeds for the Qwen 3.5 model family, with recent llama.cpp optimisations delivering measurable performance gains.
-
Qwen 3.5 Derestricted Model Available for Local Deployment
A derestricted variant of Qwen 3.5 27B has been released on Hugging Face, with community members requesting quantised GGUF versions for broader local deployment.
-
Qwen 3.5 Family Benchmark Comparison Shows Strong Performance Across Smaller Models
New benchmarks reveal that Qwen 3.5's 27B, 35B, and 122B variants retain most of the flagship model's performance, while smaller 2B and 0.8B models show steeper degradation on long-context and agent tasks.
-
Benchmark: Local Open-Source LLMs Competitive in Real-Time Trading Applications
A comprehensive benchmarking study comparing 10 LLMs including DeepSeek, Llama, and Qwen on real-time options trading reveals that local open-source models are surprisingly competitive with closed-source alternatives on practical decision-making tasks.
-
Qwen 3.5 27B Achieves Strong Local Inference Performance
Users report impressive performance metrics with Qwen 3.5 27B running locally, achieving 90 tokens/second on consumer hardware and demonstrating competitive results against proprietary models.
-
Llama.cpp Prompt Processing Optimization: Ubatch Size Configuration Guide
A community member shares practical troubleshooting advice for improving prompt processing performance on larger models like Qwen 27B by configuring ubatch size parameters in llama.cpp.
-
Qwen3-Coder-Next Achieves Top Ranking on SWE-bench at Pass@5
The Qwen3-Coder-Next model has reached the top position on SWE-bench leaderboards across both open-source and proprietary models, despite being an instruction-tuned model rather than a reasoning model. Its exceptional performance at error recovery and code fixing makes it a standout choice for local development workflows.
-
Open WebUI Adds Native Terminal Tool Calling with Qwen3.5 35B Support
Open WebUI has integrated native tool calling and open terminal functionality, enabling direct system command execution through Qwen3.5 35B. This breakthrough allows local LLM deployments to interact with system environments in real-time, significantly expanding their practical applications.
-
Alibaba Releases Qwen 3.5 AI Model with On-Device AI Support
Alibaba has released Qwen 3.5, a new AI model designed with on-device inference capabilities. This release expands the ecosystem of locally-deployable models optimized for edge devices and self-hosted environments.
-
Real-World Qwen 3.5 9B Agent Performance on M1 Pro Validates Edge Deployment
A developer successfully ran Qwen 3.5 9B as an autonomous agent on an M1 Pro MacBook with 16GB RAM, completing actual production tasks. Results demonstrate that capable local agents no longer require high-end hardware.
-
Final Qwen3.5 Unsloth GGUF Update with Improved Size/Quality Tradeoffs
Unsloth releases final GGUF quantizations for Qwen3.5-122B-A10B and Qwen3.5-35B-A3B with optimized size/KL divergence tradeoffs at 99.9% quality retention. This represents a significant milestone in making large models efficiently deployable locally.
-
Alibaba Releases Qwen 3.5 AI Model with On-Device AI Support
Alibaba has released Qwen 3.5, a new AI model offering optimised on-device AI capabilities for local deployment and edge inference scenarios.
-
Qwen 3.5-4B Generates Fully Functional OS in Single Prompt
A user demonstrates Qwen 3.5-4B generating a complete web-based operating system with games, text editor, audio player, and file browser in a single inference pass, showcasing impressive code generation capability.
-
Quantifying Cost Savings with Local LLMs for Development
A developer shares detailed analysis of cost savings achieved by using Qwen 3.5-35B locally instead of cloud-based coding assistants, demonstrating substantial financial benefits.
-
Apple M5 Pro and M5 Max: 4× Faster LLM Processing
Apple's new M5 chip generation delivers up to 4× faster LLM prompt processing than previous generations, dramatically improving on-device inference on MacBooks and iPads.
-
Qwen 3.5-35B-A3B Achieves 37.8% on SWE-bench Verified Hard
Qwen's 35B model hits near-Claude-Opus performance on the challenging SWE-bench Verified Hard benchmark, demonstrating significant capability for local code generation and software engineering tasks.
-
Qwen 3.5-27B Q4 Quantization Comparison and Analysis
Community-driven quantization sweep compares multiple GGUF quantization approaches for Qwen 3.5-27B, providing data-driven guidance for selecting optimal quantization formats.
-
Qwen 3.5 vs Qwen 3 Benchmark Analysis: Generational Performance Improvements Visualized
Comprehensive benchmark visualization comparing all Qwen 3.5 models against Qwen 3 predecessors, showing measurable improvements across reasoning, coding, and knowledge tasks at each size tier.
-
Qwen 3.5 Small Models Released: 0.8B to 9B Parameters Optimized for On-Device Inference
Alibaba's Qwen team released a new family of small multimodal models (0.8B, 2B, 4B, 9B) designed specifically for on-device and edge deployment, with demonstrated improvements across the generational progression from Qwen 2.5 to 3.5.
-
Qwen 3.5 0.8B Successfully Deployed on 7-Year-Old Samsung S10E Using llama.cpp
Successful demonstration of running Qwen 3.5's 0.8B model on aging smartphone hardware using llama.cpp and Termux, achieving 12 tokens per second on a 2019 device.
-
Alibaba's Qwen 3.5 Small Model Runs Directly on iPhone 17
Alibaba releases Qwen 3.5, a lightweight AI model optimized for on-device inference on Apple's iPhone 17. This breakthrough demonstrates practical edge deployment of capable language models on consumer mobile hardware.
-
Qwen 3.5 0.8B Running in Browser with WebGPU via Transformers.js
A practical demonstration of running Qwen 3.5's smallest 0.8B multimodal model directly in the browser using WebGPU and Transformers.js, eliminating backend requirements for inference.
-
Framework Choice Critical: llama.cpp and vLLM Outperform Ollama for Qwen 3.5 Testing
Community PSA reveals significant performance and correctness differences between local inference frameworks when running Qwen 3.5 models, with llama.cpp, transformers, vLLM, and SGLang producing correct results while Ollama shows issues with reasoning and tool use.
-
Critical: Qwen 3.5 Requires BF16 KV Cache, Not FP16 for Accurate Inference
Community member Daniel Han alerts users that Qwen 3.5 models require bfloat16 KV cache precision instead of the default float16, with perplexity measurements demonstrating the accuracy impact when using incorrect cache formats.
-
Qwen 3.5 27B on Dual RTX 3090s: 170K Context Holds, 100+ Tokens/s Claim Disputed
A widely shared r/LocalLLaMA video reported Qwen 3.5 27B running at 100+ tokens/second decode with a 170K context window on dual RTX 3090s. The context claim holds and is in fact understated — 262K fits. The decode figure is contradicted by independent benchmarks measuring 41.4 t/s on the same model and hardware, and the original video has never been independently verified.
-
Jan Releases Code-Tuned 4B Model for Efficient Local Code Generation and Development Tasks
The Jan team open-sources Jan-Code-4B, a specialized 4-billion parameter model fine-tuned for code generation, refactoring, debugging, and test writing while optimizing for local deployment and efficiency.
-
Switch Qwen 3.5 Thinking Mode On/Off Without Model Reload Using setParamsByID
Unsloth and Qwen community members have discovered how to toggle thinking vs. instruct mode on Qwen 3.5 without reloading the model, enabling dynamic workflow switching and reducing inference latency.
-
Qwen 3.5-35B-A3B Emerges as Efficient Daily Driver, Replacing 120B Models
Qwen 3.5-35B-A3B is delivering exceptional performance at one-third the size of previous daily drivers, offering significant efficiency gains for local deployment without sacrificing capability.
-
Qwen3.5-35B Successfully Runs on Raspberry Pi 5 at 3+ Tokens/Second
Demonstration of Qwen3.5-35B inference on Raspberry Pi 5 (16GB and 8GB variants) achieving over 3 tokens/second, proving high-capacity models viable on edge devices.
-
Qwen3.5-35B Unsloth Dynamic GGUFs Achieve SOTA Across Nearly All Quantisation Levels
New state-of-the-art GGUF quantisations for Qwen3.5-35B released with 150+ KL Divergence benchmarks and 9TB of variants. Critical tool calling chat template bug fixed affecting all quantisation uploaders.
-
Qwen3.5-35B RTX 5080 Experiments Confirm KV q8_0 as Free Lunch, Q4_K_M Remains Optimal
Follow-up benchmarking of Qwen3.5-35B-A3B on RTX 5080 16GB validates community-requested configurations, achieving 74.7 tokens/second and confirming KV cache quantisation strategies.
-
Qwen 3.5-27B Demonstrates Exceptional Performance with Thoughtful Prompt Engineering
Users report that Qwen 3.5-27B significantly exceeds expected performance for its size when paired with effective prompting strategies, suggesting prompt engineering can bridge the capability gap between model sizes.
-
Qwen 3.5 Underperforms on Hard Coding Tasks—APEX Benchmark Analysis
A comprehensive benchmark testing Qwen3.5 models against 70 real repositories reveals significant weaknesses in complex coding tasks compared to other models. The analysis challenges claims of Qwen3.5's general-purpose capability and highlights the importance of task-specific evaluation.
-
Qwen3.5 122B Achieves 25 tok/s on 72GB VRAM Setup
Users report exceptional performance running Qwen3.5 122B across three 3090s with 72GB total VRAM, reaching 25 tokens/second with full GPU loading. The model demonstrates strong inference speed and practical viability for enthusiasts with mid-range hardware stacks.
-
Qwen 3.5 MoE Delivers 100K Context Window at 40+ TPS on RTX 5060 Ti
Qwen3.5's mixture-of-experts variant achieves exceptional throughput with 100,000 token context window on a single mid-range GPU, reaching 41+ tokens per second using the Vulkan backend. This demonstrates practical feasibility of ultra-long context models on consumer hardware.
-
Qwen3.5 Thinking Mode Can Be Disabled for Production Inference Optimization
Users can now disable Qwen3.5's thinking capability via llama.cpp configuration, enabling optimized inference parameters for instruct mode deployments without the reasoning overhead.
-
Qwen3.5-27B Identified as Sweet Spot for Mid-Range Local Deployment
Users are reporting that Qwen3.5-27B offers the ideal balance of performance and resource efficiency for local inference, with verified setups running at 19.7 tokens/sec on consumer GPUs with reasonable memory footprints.
-
Qwen3.5 Series Releases Comprehensive Model Lineup Across All Tiers
Alibaba released the complete Qwen3.5 model family including 27B, 35B-A3B, and 122B-A10B variants, each optimized for different deployment scenarios and providing extensive benchmark comparisons.
-
Qwen3.5-35B-A3B Emerges as Game-Changer for Agentic Coding Tasks
The newly released Qwen3.5-35B-A3B model with MoE architecture is delivering exceptional performance for coding agents on consumer hardware, with users reporting impressive results running on a single RTX 3090.
-
Qwen3 Coder Next Remains Effective at Aggressive Quantization Levels
Testing reveals that Qwen3 Coder Next maintains usability even at Q2 quantization levels, suggesting Qwen models offer better quantization resilience than comparable 30B alternatives for code tasks.
-
SanityBoard Adds 27 New Model Evaluations Including Qwen 3.5 Plus, GLM 5, and Gemini 3.1 Pro
SanityBoard, a comprehensive LLM evaluation framework, has added 27 new benchmark results including evaluations of Qwen 3.5 Plus, GLM 5, Gemini 3.1 Pro, Sonnet 4.6, and three new open-source agents. The framework provides practical comparison metrics for practitioners selecting models for local deployment.
-
Local Vision-Language Models for Document OCR and PII Detection in Privacy-Critical Workflows
A developer has published an open-source application using local Qwen VLMs for document OCR with bounding box detection, enabling privacy-preserving PII detection and redaction without cloud services.
-
AMD Announces Day 0 Support for Qwen 3.5 LLM on Instinct GPUs
AMD has enabled immediate support for the Qwen 3.5 model on its Instinct GPU lineup, providing optimized inference performance for local deployments on AMD hardware accelerators.
-
Qwen3-Next 80B MoE Achieves 39 Tokens/Second on RTX 5070/5060 Ti Dual-GPU Setup
A community member has optimised Qwen3-Next 80B mixture-of-experts to run at 39 tokens/second on dual RTX 50-series GPUs with 32GB total VRAM, sharing previously undiscovered configuration solutions for consumer-grade hardware.
-
Qwen 3.5-397B-A17B Now Available for Local Inference with Aggressive Quantisation
Alibaba's Qwen 3.5-397B mixture-of-experts model is now available on HuggingFace with multiple quantisation options, including a 113GB IQ2_XS variant that fits on consumer hardware. Early benchmarks show performance competitive with Gemini 3 Pro and GPT-5.2 on spatial reasoning tasks.
-
Qwen Coder Next Shows Specialized Agent Performance
Community testing reveals Qwen Coder Next excels at agent work and research tasks rather than pure code generation, showing strong performance in planning, technical writing, and information gathering despite its coding-focused name.