FP8 Quantization

A developer recently unveiled the "Garlic" project on Reddit, demonstrating a massive performance leap for Qwen3.5 35B (A3B) on consumer-grade hardware. By implementing a custom Gated Delta Network kernel, the project achieved inference speeds of 55-61 tok/s on an RTX 5060 Ti, significantly outclassing industry-standard backends like llama.cpp. ▶ Unlocking MoE Efficiency: Qwen3.5 35B’s Mixture-of-Experts architecture, which activates only 3B parameters per token, combined with FP8 quantization, allows mid-range silicon to deliver enterprise-level throughput. ▶ The Power of Specialized Kernels: The Garlic implementation proves that architecture-specific CUDA kernels provide a "performance alpha" over general-purpose frameworks, maximizing hardware utilization for specific Gated Delta structures. ▶ Redefining Local UX: Sustaining 60 tok/s on an entry-level GPU transforms the local LLM experience, enabling near-instantaneous reasoning and seamless real-time Agentic workflows. Bagua Insight The Qwen3.5 35B (A3B) model is the "sweet spot" for the current generation of local AI, but the Garlic project highlights a critical gap: general-purpose inference engines are leaving significant performance on the table. Achieving 60 tok/s on an RTX 5060 Ti—a card often dismissed for serious AI work due to memory constraints—is a paradigm shift. It suggests that the frontier of Edge AI isn't just about shrinking models (distillation), but about hyper-optimizing the software stack to match the specific sparsity patterns of MoE architectures. We are moving from a "brute force" era of compute to a "surgical" era of kernel optimization. Actionable Advice Developers should pivot from generic backends to specialized MoE inference engines when deploying locally. For enterprise AI architects, Qwen3.5 with FP8 quantization should be a top-tier candidate for edge deployment, offering the best balance of reasoning depth and low latency. Furthermore, keep a close watch on the FP8 throughput of the RTX 50-series; these cards, when paired with custom kernels like Garlic, will likely become the gold standard for high-performance, cost-effective local AI workstations.