Tagged "moe"
52 articles tagged moe, 12 February 2026 to 16 October 2026. Newest first.
-
Ollama Adds NVIDIA Nemotron 3.5 Super Vision-Language Model Support
Ollama v0.40.5 now supports NVIDIA's Nemotron 3.5 Super, a 120B-parameter hybrid Mamba-2/Transformer mixture-of-experts model optimized for local deployment. The model pairs efficient inference with advanced capabilities in coding, reasoning, and document understanding.
-
Liquid AI Releases LFM2.5-24B-A2B: A Decision Model Built for Fast On-Device Agents
Liquid AI has released LFM2.5-24B-A2B, a 24B parameter model optimised for on-device agent deployment with only 2B active parameters during inference. This model is designed to deliver both speed and capability for local LLM inference on consumer hardware.
-
JetBrains Releases Mellum 2.1: 12B MoE Model Optimised for Coding Agents
JetBrains released Mellum 2.1, a 12 billion parameter Mixture-of-Experts model designed specifically for coding tasks and agentic workflows, available as an open-source model.
-
Aleph Alpha Releases Kolibri: A 78.1B Open-Weight English-German MoE Model
Aleph Alpha has released Kolibri, a 78.1B Mixture-of-Experts model with only 3.46B active parameters, enabling efficient local deployment of high-capacity multilingual models with minimal compute requirements.
-
Allen Institute Releases Olmo-Core 3: Open Training Infrastructure for Large Mixture-of-Experts Models
Allen Institute has released Olmo-Core 3, an open-source training infrastructure designed for large-scale mixture-of-experts (MoE) models, enabling community-driven development of efficient models suitable for local deployment.
-
Llama.cpp Fork Achieves 2-4x MultiGPU Speedup for MoE Models Larger Than VRAM
A community fork of llama.cpp enables efficient distributed inference for Mixture-of-Experts models that exceed single GPU VRAM capacity, achieving 2-4x speedup improvements across multiple GPUs.
-
Llama.cpp Fork Delivers 2-4x Speedup for Multi-GPU MoE Model Inference
A specialized llama.cpp fork optimizes mixture-of-experts models for multi-GPU setups, achieving 2-4x performance improvements for models exceeding single-GPU VRAM limits. This enables practical local deployment of large MoE architectures.
-
llama.cpp Broadens MoE Optimization Heuristics for AMD RDNA3.5
llama.cpp release b10997 improves Mixture-of-Experts performance on AMD's latest architecture with refined tile heuristics and verified correctness on Ryzen AI MAX+.
-
Run 744B MoE Models on a Laptop With Disk Streaming, No GPU Needed
A breakthrough technique enables running massive 744B mixture-of-experts models on standard laptops through disk streaming without requiring dedicated GPU hardware. This dramatically expands the accessibility of large models for local deployment.
-
Llama.cpp B10758: Hexagon MUL_MAT Fusion and MoE Optimizations for Qualcomm Hardware
Latest llama.cpp release adds Qualcomm Hexagon MUL_MAT and MUL_MAT_ID fusion optimizations, enabling efficient inference on Qualcomm processors used in edge devices and Android hardware. This expands local inference support beyond traditional server/desktop GPUs.
-
FreeToken: Edge-Native MoE Serving with CPU-GPU Co-Execution
FreeToken is an open-source engine for running 290B+ Mixture-of-Experts models locally on consumer hardware through bandwidth-adaptive CPU-GPU co-execution, with elastic memory management, expert caching, and support for DeepSeek, Qwen and GLM models across NVIDIA RTX 30/40/50 series.
-
Gemma 4 MoE for Agentic Coding: Testing Open-Weight Models on AMD APU Hardware
Alex Ewerlof runs Gemma 4 26B MoE for coding on an AMD Ryzen 7 PRO 250 APU with 64GB of RAM, and reports that tooling closes much of the gap to proprietary models — at the cost of cold starts and slower inference.
-
How to Run Qwen3.8-27B on a Single 16GB Card
Practical guide demonstrating techniques to fit the 27-billion parameter Qwen3.8 model within 16GB VRAM constraints using llama.cpp, quantization, and RTX 3080 optimizations.
-
FreeToken: Edge-Native MoE Serving Engine for Consumer Hardware
FreeToken is a mixture-of-experts serving engine aimed at running frontier-scale open-weight models on consumer hardware, using CPU-GPU co-execution rather than a multi-GPU cluster.
-
llama.cpp b10524 Makes MoE Expert Scatter Deterministic in OpenCL
llama.cpp releases build b10524 with deterministic MoE expert scatter operations in OpenCL backend, improving reliability for Mixture of Experts models on GPU acceleration. This optimization is crucial for consistent inference behavior.
-
DeepSeek V4 Flash Shrunk to 57GB for Local macOS Inference with Compiler Generation
A community contributor has quantized DeepSeek V4 Flash to 57GB, enabling capable inference on Apple Silicon Macs with demonstrated ability to generate production-quality code. This showcases aggressive quantization techniques making frontier-grade models feasible on personal devices.
-
Show HN: I shrank DeepSeek V4 Flash to 57GB and it wrote a compiler on my Mac
A developer successfully compressed DeepSeek V4 Flash to 57GB and demonstrated its capability to write a compiler on a Mac. This showcases practical quantization and model optimization techniques for running state-of-the-art models on consumer hardware.
-
Ollama Releases NVIDIA Nemotron 3.5 Lightning for Agent Execution
Ollama v0.32.9 now includes NVIDIA's Nemotron 3.5 Lightning, a 30B MoE model with only 3B active parameters optimized for on-device agent execution. This lightweight model is designed for frameworks like OpenClaw and Hermes Agent, making powerful agentic AI accessible on local hardware.
-
Ollama Releases NVIDIA Nemotron 3.5 Lightning for Local Agent Deployment
NVIDIA's new 30B mixture-of-experts model with only 3B active parameters is now available in Ollama, optimized for building always-on agents with minimal resource requirements. The model is designed for agent frameworks like OpenClaw and Hermes.
-
Ollama Releases NVIDIA Nemotron 3.5 Lightning for Agent Execution
NVIDIA's new 30B mixture-of-experts model with 3B active parameters is now available in Ollama v0.32.9, optimized for agent workloads and on-device execution. The model is designed for frameworks like OpenClaw and Hermes, bringing efficient MoE inference to local deployments.
-
Thinking Machines Lab Releases Inkling-Small: A 276B Total, 12B Active Open Weights Multimodal MoE Model
Thinking Machines Lab has released Inkling-Small, an open-weights multimodal mixture-of-experts model with 276B total parameters but only 12B active during inference, enabling efficient local deployment on consumer hardware.
-
AMD ZenDNN 6.0 Boosts AI Inference on EPYC CPUs With FP16 and MoE Acceleration
AMD has released ZenDNN 6.0 with optimizations for FP16 inference and Mixture-of-Experts model acceleration on EPYC processors. This update enables efficient local LLM deployment on AMD server and workstation CPUs without requiring GPUs.
-
Tencent Open-Sources Hy3 295B MoE Model Built for STEM Reasoning
Tencent releases Hy3, a 295B mixture-of-experts model optimized for STEM reasoning tasks. This open-source release provides local LLM practitioners with a high-capacity model option for specialized reasoning workloads.
-
Qwen and Fable: Open-Weights 35B Mixture-of-Experts Agentic Coding Model
A new open-weights 35B Mixture-of-Experts model combining Qwen and Fable for agentic coding tasks, optimized for local deployment with improved efficiency through sparse computation patterns.
-
JetBrains Releases Mellum2: A 12B MoE Model for Fast, Specialized Tasks
JetBrains introduces Mellum2, a 12-billion parameter mixture-of-experts model designed for efficient local inference in multi-model AI pipelines. The model balances performance and resource consumption for on-device deployment scenarios.
-
The Brain vs. Deep Learning Part I: Computational Complexity Analysis
A detailed analysis comparing computational complexity between biological brains and deep learning systems provides theoretical foundations for understanding efficiency trade-offs in model design and local deployment. This research is foundational for optimizing inference on resource-constrained devices.
-
Dynamic Expert Cache in llama.cpp Achieves 27% Faster Inference on Large MoE Models
A new optimization technique for llama.cpp improves CPU+GPU token generation speed by 27% on Qwen3.5-122B through dynamic expert caching, raising practical inference rates from 15 to 23 tokens per second.
-
Gemma 4 26B MoE Emerges as Optimal All-Around Local Model for Consumer Hardware
Community testing reveals Gemma 4 26B MoE (Mixture of Experts) is well-suited for local deployment on consumer machines, with particular strength in coding tasks and memory efficiency. The model achieves impressive performance while remaining manageable on 16GB VRAM systems.
-
M5 Max Delivers 1.7x Faster Inference Than M3 Max on Qwen 3.5 Models
Comprehensive benchmarks comparing Apple's M5 Max and M3 Max chips show significant performance gains across Qwen 3.5 model variants (27B dense, 35B MoE, 122B MoE), with the newer chip delivering 1.4x to 1.7x faster token generation using the oMLX framework.
-
Homelab Consolidation: Replacing 3 Models with Single 122B MoE Model on AMD Ryzen AI MAX+
A homelabber consolidated their inference setup from three separate models down to a single 122B mixture-of-experts model on consumer hardware (Ryzen AI MAX+ 395 with 128GB RAM), providing detailed benchmarks and practical insights on model consolidation strategy.
-
Liquid AI's LFM2-24B Achieves 50 Tokens/Second in Web Browser via WebGPU
Liquid AI has demonstrated their LFM2-24B mixture-of-experts model running at 50 tokens/second in a web browser on M4 Max hardware using WebGPU. The 8B variant achieves over 100 tokens/second, showcasing practical edge inference in browser environments.
-
New Open-Weight Models Released: GigaChat-3.1-Ultra and Lightning Variants
Open-weight releases of GigaChat-3.1-Ultra (702B MoE) and GigaChat-3.1-Lightning (10B) models are now available under MIT license, targeting both high-resource and edge deployment scenarios.
-
Chinese LLM Ecosystem Landscape: ByteDance Doubao, Alibaba, and Open-Source Competition
Comprehensive analysis of the Chinese LLM scene reveals ByteDance's Doubao as the market leader with strong open-source alternatives from Alibaba, Deepseek, and others, highlighting the rapid innovation and diverse model ecosystem emerging from China's AI development.
-
FlashAttention-4 Delivers 2.7x Faster Inference with 1613 TFLOPs/s on Blackwell GPUs
FlashAttention-4, written in Python, achieves near-matmul-speed attention kernels with 71% GPU utilization on NVIDIA B200, delivering 2.1-2.7x faster inference than Triton. This breakthrough optimizes the attention bottleneck for local LLM deployment.
-
FOMOE: Running 397B Parameter Qwen3.5 MoE at 5-9 tok/s on $2,100 Desktop Hardware
Fast Opportunistic Mixture of Experts (FOMOE) enables inference of massive 397-billion parameter models using Q4_K_M quantization on dual $500 consumer GPUs with 32GB RAM, solving the memory bottleneck of MoE models through intelligent flash-backed weight streaming.
-
Researcher Discovers Universal "Danger Zone" in Transformer Model Architecture at 50% Depth
Experimental layer surgery across six different model architectures reveals a critical vulnerability at approximately 50-56% model depth where layer duplication consistently degrades performance, offering new insights into transformer architecture optimisation.
-
llmfit Checks Your Hardware and Ranks Models Before You Download
llmfit scans RAM, CPU, GPU and VRAM in one command, then ranks models on quality, speed, fit and context, picking a quantization and labelling each result ideal, okay or borderline. It accounts for active parameters on MoE models rather than total.
-
Qwen3.5-397B Achieves 282 tok/s on 4x RTX PRO 6000 Blackwell Through Custom CUTLASS Kernel
A developer achieved a 5x performance improvement on the massive Qwen3.5-397B model by building a custom CUTLASS kernel to fix SM120's broken MoE GEMM tiles, reaching 282 tokens/second on Blackwell GPUs. This breakthrough demonstrates significant optimization potential for running large models locally with multi-GPU setups.
-
Nvidia Releases Nemotron 3 Super: 120B MoE Model for Local Deployment
Nvidia has released Nemotron 3 Super, a 120B mixture-of-experts model with only 12B active parameters, designed as an open-source alternative for agentic reasoning tasks. The hybrid Mamba-Transformer architecture offers competitive performance with reduced computational requirements.
-
Comprehensive MoE Backend Benchmarks for Qwen3.5-397B: Real Numbers vs Hype
A detailed benchmark of every major MoE backend for Qwen3.5-397B NVFP4 on workstation GPUs reveals actual sustained performance of 50.5 tok/s, significantly lower than commonly cited claims. The analysis uncovers kernel issues in Nvidia's own CUTLASS implementation.
-
Krasis: Hybrid CPU/GPU MoE Runtime Achieves 3,324 Tokens/Second Prefill on RTX 5080
New open-source runtime optimises mixture-of-experts models by splitting prefill to GPU and decode to CPU, enabling larger MoE models to run on single consumer GPUs with dramatic throughput improvements.
-
Qwen 3.5 MoE Delivers 100K Context Window at 40+ TPS on RTX 5060 Ti
Qwen3.5's mixture-of-experts variant achieves exceptional throughput with 100,000 token context window on a single mid-range GPU, reaching 41+ tokens per second using the Vulkan backend. This demonstrates practical feasibility of ultra-long context models on consumer hardware.
-
Qwen3.5-35B-A3B Emerges as Game-Changer for Agentic Coding Tasks
The newly released Qwen3.5-35B-A3B model with MoE architecture is delivering exceptional performance for coding agents on consumer hardware, with users reporting impressive results running on a single RTX 3090.
-
[Release] Ouro-2.6B-Thinking: ByteDance's Recurrent Model Now Runnable Locally
ByteDance's novel recurrent Universal Transformer architecture (Ouro-2.6B-Thinking) is now functional for local inference after fixes for transformers 4.55, enabling access to a unique thinking-focused model on consumer hardware.
-
Alibaba's Qwen3.5-397B Achieves #3 Position in Open Weights Model Rankings
Alibaba's newly released Qwen3.5-397B mixture-of-experts model ranks #3 in the Artificial Analysis Intelligence Index among open-weight models, offering a powerful option for large-scale local deployment.
-
Qwen3-Next 80B MoE Achieves 39 Tokens/Second on RTX 5070/5060 Ti Dual-GPU Setup
A community member has optimised Qwen3-Next 80B mixture-of-experts to run at 39 tokens/second on dual RTX 50-series GPUs with 32GB total VRAM, sharing previously undiscovered configuration solutions for consumer-grade hardware.
-
Qwen 3.5-397B-A17B Now Available for Local Inference with Aggressive Quantisation
Alibaba's Qwen 3.5-397B mixture-of-experts model is now available on HuggingFace with multiple quantisation options, including a 113GB IQ2_XS variant that fits on consumer hardware. Early benchmarks show performance competitive with Gemini 3 Pro and GPT-5.2 on spatial reasoning tasks.
-
MiniMax-M2.5 230B MoE Model Released with GGUF Support for Local Deployment
MiniMax-M2.5, a 230B parameter mixture-of-experts model, is now available in GGUF format for local deployment with impressive performance benchmarks on consumer hardware.
-
GPT-OSS 120B Uncensored Model Released in Native MXFP4 Precision
An uncensored version of GPT-OSS 120B has been released featuring native MXFP4 precision training, offering 117B parameters with MoE architecture for efficient local deployment.
-
MiniMax M2.5: 230B Parameter MoE Model Coming to HuggingFace
MiniMax officially confirms open-source release of M2.5, a 230B parameter MoE model with only 10B active parameters, showing impressive SWE-Bench performance at 80.2%.
-
Ming-flash-omni-2.0: 100B MoE Omni-Modal Model Released
Ant Group releases Ming-flash-omni-2.0, a 100B MoE model with 6B active parameters supporting unified speech, SFX, music generation alongside image, text, and video processing.
-
GLM-5 Released: 744B Parameter MoE Model Targeting Complex Tasks
Zhipu AI releases GLM-5, a massive 744B parameter MoE model with 32B active parameters, designed for complex systems engineering and long-horizon agentic tasks with significant performance improvements over GLM-4.5.