DeepSeek V4 Full Paper Unveiled: How FP4 QAT Redefines the Efficiency Frontier of LLMs
Core Event Summary
DeepSeek released the full technical report for V4 this week, detailing a sophisticated transition to FP4 Quantization-Aware Training (QAT) during the late stages of pre-training, achieving a massive leap in inference throughput and memory efficiency.
- ▶ VRAM Bottleneck Breakthrough: By quantizing MoE expert weights—the primary memory hog—into FP4, DeepSeek has effectively lowered the hardware barrier for deploying trillion-parameter models without sacrificing performance.
- ▶ Hardware-Native Acceleration: Implementing FP4 activations in the Compressed Sparse Attention (CSA) indexer’s QK path resulted in a 2x speedup for the QK selector while maintaining a near-perfect 99.7% recall rate.
- ▶ Stability Engineering: The paper reveals critical “stability tricks” for low-precision training, providing a blueprint for maintaining gradient health during ultra-low-bit optimization.
Bagua Insight
The DeepSeek V4 paper signals a strategic pivot in the LLM arms race: the focus is shifting from raw scaling to “Inference-Optimized Training.” DeepSeek’s brilliance lies in treating quantization as a first-class citizen within the training loop rather than an afterthought. By integrating FP4 QAT, they are essentially co-designing the model with the underlying silicon. This level of hardware-aware algorithmic design is what allows DeepSeek to punch far above its weight class, proving that numerical precision management is the new frontier for competitive advantage in the GenAI era.
Actionable Advice
Enterprises aiming for sustainable AI scaling must look beyond standard FP16/BF16 training regimes. Architects should investigate the feasibility of late-stage QAT to optimize models for next-gen hardware. Furthermore, the optimizations applied to the CSA indexer should be studied by any team building high-performance RAG or long-context applications. The industry takeaway is clear: if your model architecture isn’t optimized for FP4/INT4 at the training level, your inference TCO will be dead on arrival in the coming year.