GB10

Core Summary Moonshot AI's Kimi K3 full model has been successfully deployed on a 16x GB10 cluster using the dspark framework, delivering a stable throughput of 20+ tps, marking a significant milestone for domestic large-scale model optimization and distributed inference. Bagua Insight ▶ Paradigm Shift in Compute Efficiency: Achieving 20+ tps on a GB10 cluster via dspark demonstrates that distributed inference architectures have effectively mitigated the memory bandwidth and latency bottlenecks inherent in ultra-long-context LLMs. ▶ Democratizing High-Performance AI: The successful deployment of the full K3 model signals that high-performance private infrastructure is becoming increasingly accessible, accelerating the adoption of enterprise-grade GenAI for complex, long-context tasks. Actionable Advice ▶ Technical Monitoring: Track the upcoming release of the vllm image and assess its integration with existing inference stacks, specifically focusing on memory optimization for long-context workloads. ▶ Strategic Planning: Organizations in data-sensitive sectors (legal, R&D, finance) should evaluate the TCO of deploying the full K3 model on-premise to bypass data residency concerns associated with public API reliance.

Event Core Zhipu’s GLM-5.2 (Int4/Int8) demonstrates exceptional inference efficiency on an 8× GB10 GPU cluster, achieving a prefill speed of ~1,200 t/s and a sustained decode throughput of 33–54 t/s, while maintaining sufficient VRAM headroom to concurrently run the Mimo 2.5 multimodal model. Bagua Insight ▶ Shift in Compute Efficiency: The GB10 architecture, when paired with TP8 (Tensor Parallelism), proves that high-throughput inference no longer requires dedicated hardware silos. The ability to stack models suggests a shift toward more dense, multi-model deployment strategies in enterprise production. ▶ Engineering Multimodal Synergy: Running GLM-5.2 and Mimo 2.5 simultaneously on the same cluster validates the feasibility of unified compute orchestration for complex AI Agents, effectively reducing the TCO (Total Cost of Ownership) for multimodal pipelines. Actionable Advice Optimize Deployment Density: Organizations should audit their current inference workloads. With high-end hardware like the GB10, focus on maximizing VRAM utilization by co-locating complementary models rather than scaling individual instances. Prioritize Quantization: The 33-54 t/s decode performance confirms that Int4/Int8 quantization is now production-ready for latency-sensitive applications. Shift focus from raw precision to throughput-optimized serving architectures.

A breakthrough deployment on a 4× DGX Spark (GB10) cluster has successfully enabled GLM-5.2 with Multi-Token Prediction (MTP) speculative decoding. By reconstructing missing build recipes and pinning specific vLLM forks, developers achieved a stable 9.4 tok/s throughput, overcoming critical AWQ weight loading issues.▶ The Missing Link in Public Recipes: Existing open-source documentation for GLM-5.2 often lacks the Docker image construction layer. This successful run utilized Claude-assisted kernel reconstruction to bridge the gap between raw code and a functional production environment.▶ Dependency Fragility: The deployment highlights a strict dependency on specific vLLM versions; mismatched environments lead to immediate system crashes during AWQ weight initialization, emphasizing the need for precise environment parity.▶ Hardware-Software Synergy: By leveraging ported Sparse MLA (Multi-Head Latent Attention) Triton kernels and TP=4 configurations, the implementation maximizes the throughput capabilities of NVIDIA’s latest GB10 silicon.Bagua InsightThis case underscores the "Engineering Friction" inherent in deploying state-of-the-art models like GLM-5.2. The reliance on MTP and custom Triton kernels signals a shift in the LLM landscape: raw FLOPs are no longer enough; inference efficiency is now won in the trenches of operator optimization. The fact that developers are using LLMs (Claude) to fix the build scripts of other LLMs creates a fascinating recursive loop in AI engineering. For the industry, this proves that GLM-5.2’s architecture is viable for high-end clusters, provided the inference stack is sufficiently customized.Actionable AdviceInfrastructure teams should prioritize "Golden Image" management for GLM-series deployments, ensuring that pre-compiled Triton kernels and specific vLLM forks are baked into the CI/CD pipeline. Avoid generic inference servers; instead, invest in tuning Tensor Parallelism (TP) settings specifically for the GB10 interconnect. For those seeking maximum performance, MTP should be treated as a mandatory optimization rather than an optional feature, requiring deep integration with the underlying sparse attention mechanisms.