Performance Beast: Qwen3.8-Flash-Next Hits 120 t/s on 4xR9700 Local Cluster
Event Core
Leveraging the MXFP4-FP8 quantization scheme developed by tcclaviger and a custom vLLM image optimized for AMD R9700 GPUs, Qwen3.8-Flash-Next has achieved a breakthrough in local inference performance. The setup delivers 80-120 tokens/s in Token Generation (TG) and a massive 12k tokens/s in Prompt Processing (PP) for single requests.
- ▶ Quantization Frontier: The implementation of MXFP4 (Microscaling Formats) demonstrates that ultra-low bit-width quantization is the primary lever for maximizing throughput on prosumer-grade hardware.
- ▶ Software-Hardware Synergy: This isn’t just raw horsepower; the use of a custom vLLM image tailored for the R9700 architecture allows for kernel-level optimizations that saturate theoretical memory bandwidth.
- ▶ The Latency-Context Trade-off: The current configuration limits the total shared context length to 7, signaling a hyper-focus on raw speed for real-time agentic workflows rather than long-form document analysis.
Bagua Insight
At Bagua Intelligence, we view this as a pivotal moment for the “Prosumer Local Cluster.” Achieving 12k tokens/s prefill speed effectively eliminates the “thinking pause” in standard RAG pipelines. This performance level suggests that the bottleneck for local LLMs is shifting from compute-bound to memory-bandwidth-bound faster than anticipated. Furthermore, the success of this AMD-based optimization highlights a growing trend: the open-source community is successfully breaking the CUDA monopoly by building highly specialized software stacks for alternative silicon.
Actionable Advice
- For Developers: Prioritize the adoption of MXFP4 and FP8 hybrid kernels within vLLM to squeeze enterprise-grade performance out of local setups, especially for latency-sensitive applications.
- For Infrastructure Architects: Re-evaluate the ROI of AMD-based clusters for internal inference. When paired with custom optimized images, these setups can rival dedicated cloud instances in specific throughput-per-dollar metrics.
- Technical Caveat: Monitor context window constraints. The current speed gains come at the expense of KV Cache capacity; ensure your application logic can handle the restricted context or plan for dynamic scaling of memory resources.