FP4 Quantization

Event CoreRecent vLLM logs surfaced from the LocalLLaMA community have unveiled the raw power of NVIDIA’s Blackwell architecture utilizing FP4 (nvfp4) precision. In a batch image captioning stress test with 30 concurrent streams, the Blackwell setup achieved a staggering average prompt throughput of 1301.0 tokens/s and a generation throughput of 1924.0 tokens/s. This benchmark underscores Blackwell's dominance in handling compute-intensive multimodal workloads at scale.▶ FP4 as the New Efficiency Standard: The transition to nvfp4 quantization is the primary driver behind the 2000 TPS milestone, offering a massive leap in throughput and memory efficiency without compromising model integrity.▶ Concurrency as a Catalyst: The use of 30 concurrent streams demonstrates that Blackwell requires high-density workloads to fully saturate its compute engines, highlighting its suitability for high-traffic inference clusters.▶ Caching Synergy: The performance delta between initial prompts and subsequent requests validates the critical role of vLLM’s caching mechanisms in maximizing output for iterative multimodal tasks.Bagua InsightAt 「Bagua Intelligence」, we view these results as a paradigm shift in the economics of GenAI. The native hardware support for FP4 in Blackwell effectively solves the historical trade-off between quantization speed and model accuracy. Achieving nearly 2000 tps for multimodal generation suggests that the operational cost for sophisticated AI agents—such as real-time video analytics and massive-scale visual indexing—is about to plummet by an order of magnitude. For enterprises, Blackwell is no longer just a faster chip; it is the foundational infrastructure required to make high-throughput multimodal AI commercially viable.Actionable Advice1. Prioritize Blackwell Migration: Developers of high-frequency multimodal applications should immediately benchmark their pipelines against Blackwell’s FP4 capabilities to assess ROI. 2. Redesign for High Concurrency: Legacy inference architectures optimized for lower concurrency will leave Blackwell’s performance on the table; engineers must shift toward massive parallel stream management. 3. Double Down on KV Cache Optimization: For repetitive prompt patterns like batch image processing, refining KV cache strategies is essential to hitting the theoretical throughput ceiling of the Blackwell architecture.

This report provides a rigorous performance evaluation of leading inference engines—vLLM, SGLang, and llama.cpp—operating on a 7-GPU heterogeneous cluster. The setup mixes Blackwell (RTX 5090) and Ada (RTX 6000 Ada, 4090) architectures to test Pipeline Parallelism (PP) efficiency during long-context prefilling workloads. ▶ The FP4 Paradigm Shift: The transition to NVFP4 (vLLM/SGLang) and MXFP4 (llama.cpp) for 4-bit weights signifies that low-precision inference is no longer experimental. It is now a production requirement for maximizing throughput on Blackwell-era hardware. ▶ Heterogeneous Bottlenecks: In clusters mixing high-end workstation cards and consumer flagships, the efficiency of Pipeline Parallelism is dictated by the engine's ability to balance compute-heavy prefilling across disparate memory bandwidths and interconnects. Bagua Insight This benchmark reveals a critical inflection point in the AI infrastructure stack. The hardware-level FP4 acceleration introduced by the Blackwell architecture isn't just a spec bump; it’s a catalyst for a complete rewrite of inference kernels. While vLLM remains the industry standard for stability, SGLang is currently winning the "speed war" in long-context RAG scenarios due to its aggressive memory management and superior handling of heterogeneous pipelines. Interestingly, llama.cpp continues to punch above its weight, offering a highly flexible alternative for "Frankenstein clusters" where mixed-architecture compatibility is more critical than raw enterprise-grade concurrency. The industry is moving from "compute-bound" to "orchestration-bound" in these fragmented hardware environments. Actionable Advice For Blackwell Adopters: If you are running RTX 50-series or B200s, prioritize engines with native FP4 Tensor Core support. SGLang currently shows a slight edge in raw throughput for prefilling-heavy tasks. For Mixed-Gen Deployments: When combining Ada and Blackwell cards, utilize Pipeline Parallelism (PP) rather than Tensor Parallelism (TP) to mitigate interconnect bottlenecks. Monitor memory fragmentation closely, as the disparity in VRAM speeds can cause significant pipeline bubbles. Standardize Quantization: Evaluate the trade-offs between NVFP4 and MXFP4. For production RAG pipelines, perform rigorous Perplexity (PPL) testing to ensure that the jump to 4-bit weights doesn't degrade the model's reasoning capabilities in long-context windows.