543 tok/s on a Single GPU: NInfer Engine Shatters Qwen3.6 Performance Records on RTX 5090
A developer has open-sourced NInfer, a ground-up C++/CUDA inference engine that achieves a blistering 543 tokens per second (tok/s) for the Qwen3.6-35B-A3B model on a single RTX 5090. This performance remains consistent even across a massive 65,536-token decode window.
- ▶ Bare-Metal Engineering: By bypassing Python-heavy frameworks and utilizing raw C++/CUDA, NInfer eliminates software overhead, allowing the RTX 5090 to operate at its theoretical hardware limit.
- ▶ MoE Sweet Spot: The architecture of Qwen3.6-35B-A3B (35B total, 3B active parameters) perfectly aligns with the RTX 5090’s memory bandwidth, proving that sparse MoE models are the gold standard for high-speed consumer-grade inference.
Bagua Insight
The emergence of NInfer signals a pivotal shift in the Local LLM ecosystem: we are moving from “hobbyist experimentation” to “high-performance systems engineering.” Achieving 543 tok/s on consumer hardware effectively beats most commercial cloud APIs in raw throughput and latency. This development challenges the necessity of enterprise-grade silicon (like the H100) for specific low-latency MoE workloads. When consumer GPUs are paired with hyper-optimized C++ kernels, the ROI for local workstations and edge deployments skyrockets. This also highlights a growing trend where software optimization, rather than just raw TFLOPS, becomes the primary differentiator in the GenAI race.
Actionable Advice
- For Developers: Study NInfer’s memory management and kernel fusion techniques. Vertical optimization for specific architectures (MoE) and specific hardware (Blackwell consumer cards) is the most effective path to sub-millisecond latency.
- For Enterprises: Re-evaluate the cost-to-performance ratio of consumer GPU clusters. For high-throughput private deployments, an array of optimized RTX 5090s may offer superior value compared to traditional data center rentals for MoE-based tasks.
- Model Strategy: Prioritize MoE models with low active parameter counts (like the A3B variant) to maximize the throughput potential of local hardware deployments.