Taming the 2.8T Beast: Hosting Kimi K3 on 8x B300s at $190 per Million Tokens

Event Core

A developer successfully deployed the 2.8 trillion parameter Kimi K3 model on an 8x NVIDIA B300 GPU cluster via the Modal platform. Utilizing native MXFP4 quantization and vLLM, the setup achieved a stable decoding speed of 92 tok/s, marking a milestone for localized hosting of frontier-class models on Blackwell architecture.

• ▶ Blackwell’s Performance Moat: Leveraging the 192GB HBM3e VRAM of the B300, an 8-card node can now ingest 1.56 TB of model weights, with MXFP4 quantization serving as the catalyst for high-throughput inference.
• ▶ The Economics of Scale: While the $190/M token cost remains prohibitive compared to public APIs, a TTFT of ~0.92s proves that Blackwell can handle ultra-large MoE models with production-grade responsiveness.

Bagua Insight

This benchmark underscores a shift in AI infrastructure: the “barrier to entry” for 2T+ parameter models is being aggressively lowered by Blackwell. What previously required multiple H100 nodes is now consolidated into a single 8-GPU chassis. The transition from FP8 to MXFP4 is the real story here—it is rapidly becoming the gold standard for ultra-large model inference, offering a superior balance of compression and numerical stability. However, the 27-minute cold start for a 1.56 TB payload highlights that storage I/O and interconnect bandwidth are now the primary bottlenecks for on-demand scaling of massive LLMs.

Actionable Advice

Enterprises prioritizing performance should pivot to Blackwell-based clusters with native MXFP4 support to maximize throughput-per-watt. For cost-conscious R&D, monitor Unsloth’s dynamic GGUF implementations; their 1-bit quantization can shrink a 2.8T model’s footprint to under 600GB, enabling experimentation on significantly more accessible hardware tiers.