Basalt Unlocks Blackwell’s Potential: Native NVFP4 Support for Qwen 3.8B Flash-Next
Core Event
Basalt, a specialized local LLM runner, has introduced native support for the NVFP4 format on NVIDIA’s Blackwell architecture (RTX 5090). This update focuses on optimizing Qwen 3.8B Flash-Next, achieving record-breaking inference speeds of 304 tok/s for structured data on a single 5090, while maintaining near-lossless precision.
- ▶ Hardware-Native Efficiency: By implementing native NVFP4 support, Basalt taps into Blackwell’s specialized Tensor Cores, signaling a shift from legacy INT4 quantization to hardware-optimized 4-bit floating-point formats.
- ▶ Hybrid Quantization Architecture: The “Basalt Pack” utilizes a sophisticated hybrid approach—keeping stock NVIDIA experts while storing dense layers in Q6_K—balancing the raw speed of FP4 with the accuracy of higher-bit formats.
- ▶ Asymmetric Scaling: The introduction of “Peer GPU” support (e.g., pairing a 5090 with a 5060 Ti) boosts throughput to 479 tok/s, proving that Blackwell can effectively orchestrate workloads across disparate GPU tiers.
Bagua Insight
This development confirms that NVFP4 is the definitive standard for the Blackwell era. The marginal increase in Perplexity (3.308 vs. 3.286 for Q8_0) demonstrates that FP4 has reached production-grade maturity for small-to-medium language models. For the local LLM community, this is a paradigm shift: we are moving away from software-emulated quantization toward hardware-accelerated formats that offer a 2x-3x performance uplift without the typical “quantization tax” on intelligence. Basalt’s success with asymmetric GPU setups also suggests that the future of local AI won’t require identical multi-GPU clusters, but rather smart pooling of heterogeneous silicon.
Actionable Advice
1. Prioritize FP4 Toolchains: Organizations deploying on Blackwell hardware should immediately pivot their inference pipelines to support NVFP4 to maximize ROI on new silicon.
2. Adopt Hybrid Quantization: For models under 10B parameters, avoid monolithic quantization; instead, use hybrid schemes that protect critical dense layers while offloading the bulk of weights to FP4.
3. Leverage Legacy Hardware: Don’t decommission mid-range cards (like the 5060 Ti or 40-series); use frameworks like Basalt that support Peer GPU offloading to augment primary Blackwell compute nodes.