Breaking the VRAM Ceiling: Strategic MoE Offloading Boosts Qwen3.6-35B Prefill by 2.36x
Core Summary
By strategically offloading 8 MoE expert layers of Qwen3.6-35B-A3B to the CPU, developers managed to free up critical VRAM on an RTX 3090 (24GB), enabling a jump in prompt processing (PP) speed from 564 tok/s to 1330 tok/s via increased batch sizes.
- ▶ Asymmetric MoE Advantage: Due to the sparse activation of MoE models, offloading a subset of experts has a negligible impact on decoding speed while reclaiming VRAM for KV cache and batching overhead.
- ▶ Throughput over Raw Latency: In 64K long-context scenarios, VRAM bottlenecks are driven by batch capacity rather than compute. Doubling the batch size (-b) from 512 to 1024 was the primary catalyst for the 136% performance gain.
Bagua Insight
At Bagua Intelligence, we view this as a definitive shift in local LLM optimization: Intelligent Tiered Memory Management is superseding the “All-in-VRAM” dogma. The Qwen3.6-35B A3B (Active 3B) architecture provides a unique leverage point—since only a fraction of parameters are active per token, the penalty for CPU-side experts is masked by the massive throughput gains of larger micro-batches. Standard “auto-fit” logic in tools like llama.cpp is often too conservative. Manual tuning of expert distribution effectively uses high-capacity system RAM to “unshackle” the GPU’s high-bandwidth compute. For RAG-heavy workflows where prefill latency is the primary UX killer, this trade-off is not just optimal—it is essential.
Actionable Advice
- For RAG Developers: When deploying on 24GB hardware, prioritize VRAM for batching parameters (-b and -ub) by offloading non-critical MoE experts. This maximizes preprocessing throughput for long documents.
- Quantization Strategy: Prefer higher-bit quantizations (e.g., Q6) with strategic offloading over aggressive low-bit quants (e.g., Q4) just to fit in VRAM. The former preserves reasoning integrity while the offloading strategy recovers the lost performance.