Performance Breakthrough: QEngine 1.3.1 Hits 504 tok/s on Dual RTX 5090s with Qwen 3.8B
Core Event
A recent benchmark from the LocalLLaMA community reveals that QEngine 1.3.1, running on a dual NVIDIA RTX 5090 setup, has achieved a staggering 504 tokens/second (tok/s) during single-stream decoding with the Qwen3.8-Flash-Next model. This performance represents a 1.5x to 1.9x improvement over the next-best inference engines, setting a new gold standard for local LLM throughput on consumer-grade hardware.
- ▶ Unrivaled Single-Stream Speed: QEngine 1.3.1 leverages specialized kernels to outperform mainstream engines like Strata and vLLM by a significant margin in low-concurrency scenarios.
- ▶ KV-in-RAM Advantage: The engine features a unique mode that keeps KV caches in system RAM, drastically reducing the cost of context switching for asynchronous or agentic workflows.
- ▶ Hardware Synergy: The results highlight the massive potential of the RTX 5090’s memory bandwidth when paired with software optimized specifically for the Blackwell architecture.
Bagua Insight
At Bagua Intelligence, we view the 500+ tok/s milestone as a transition from “human-speed AI” to “machine-speed AI.” This level of throughput is the prerequisite for sophisticated agentic loops that require multiple internal reasoning steps before providing an output. QEngine’s dominance stems from its aggressive hardware-specific optimization—eschewing broad compatibility in favor of squeezing every cycle out of the 3090/5090 architectures. This signals a future where local AI performance is driven by bespoke, “bare-metal” inference stacks rather than generic, one-size-fits-all solutions.
Actionable Advice
Developers focusing on high-frequency RAG or real-time autonomous agents should prioritize inference backends that offer advanced KV cache management and hardware-native kernels. For enterprises, the dual RTX 5090 performance data suggests that local clusters are now more cost-effective and responsive than mid-tier cloud APIs for latency-sensitive tasks like real-time coding assistance. We recommend early adoption of Blackwell-optimized deployment stacks to maintain a competitive edge in local GenAI workflows.