Performance Breakthrough: Qwen3.5 35B Hits 60 tok/s on RTX 5060 Ti via Custom Gated Delta Kernels
A developer recently unveiled the “Garlic” project on Reddit, demonstrating a massive performance leap for Qwen3.5 35B (A3B) on consumer-grade hardware. By implementing a custom Gated Delta Network kernel, the project achieved inference speeds of 55-61 tok/s on an RTX 5060 Ti, significantly outclassing industry-standard backends like llama.cpp.
- ▶ Unlocking MoE Efficiency: Qwen3.5 35B’s Mixture-of-Experts architecture, which activates only 3B parameters per token, combined with FP8 quantization, allows mid-range silicon to deliver enterprise-level throughput.
- ▶ The Power of Specialized Kernels: The Garlic implementation proves that architecture-specific CUDA kernels provide a “performance alpha” over general-purpose frameworks, maximizing hardware utilization for specific Gated Delta structures.
- ▶ Redefining Local UX: Sustaining 60 tok/s on an entry-level GPU transforms the local LLM experience, enabling near-instantaneous reasoning and seamless real-time Agentic workflows.
Bagua Insight
The Qwen3.5 35B (A3B) model is the “sweet spot” for the current generation of local AI, but the Garlic project highlights a critical gap: general-purpose inference engines are leaving significant performance on the table. Achieving 60 tok/s on an RTX 5060 Ti—a card often dismissed for serious AI work due to memory constraints—is a paradigm shift. It suggests that the frontier of Edge AI isn’t just about shrinking models (distillation), but about hyper-optimizing the software stack to match the specific sparsity patterns of MoE architectures. We are moving from a “brute force” era of compute to a “surgical” era of kernel optimization.
Actionable Advice
Developers should pivot from generic backends to specialized MoE inference engines when deploying locally. For enterprise AI architects, Qwen3.5 with FP8 quantization should be a top-tier candidate for edge deployment, offering the best balance of reasoning depth and low latency. Furthermore, keep a close watch on the FP8 throughput of the RTX 50-series; these cards, when paired with custom kernels like Garlic, will likely become the gold standard for high-performance, cost-effective local AI workstations.