Heterogeneous Computing

This report provides a rigorous performance evaluation of leading inference engines—vLLM, SGLang, and llama.cpp—operating on a 7-GPU heterogeneous cluster. The setup mixes Blackwell (RTX 5090) and Ada (RTX 6000 Ada, 4090) architectures to test Pipeline Parallelism (PP) efficiency during long-context prefilling workloads. ▶ The FP4 Paradigm Shift: The transition to NVFP4 (vLLM/SGLang) and MXFP4 (llama.cpp) for 4-bit weights signifies that low-precision inference is no longer experimental. It is now a production requirement for maximizing throughput on Blackwell-era hardware. ▶ Heterogeneous Bottlenecks: In clusters mixing high-end workstation cards and consumer flagships, the efficiency of Pipeline Parallelism is dictated by the engine's ability to balance compute-heavy prefilling across disparate memory bandwidths and interconnects. Bagua Insight This benchmark reveals a critical inflection point in the AI infrastructure stack. The hardware-level FP4 acceleration introduced by the Blackwell architecture isn't just a spec bump; it’s a catalyst for a complete rewrite of inference kernels. While vLLM remains the industry standard for stability, SGLang is currently winning the "speed war" in long-context RAG scenarios due to its aggressive memory management and superior handling of heterogeneous pipelines. Interestingly, llama.cpp continues to punch above its weight, offering a highly flexible alternative for "Frankenstein clusters" where mixed-architecture compatibility is more critical than raw enterprise-grade concurrency. The industry is moving from "compute-bound" to "orchestration-bound" in these fragmented hardware environments. Actionable Advice For Blackwell Adopters: If you are running RTX 50-series or B200s, prioritize engines with native FP4 Tensor Core support. SGLang currently shows a slight edge in raw throughput for prefilling-heavy tasks. For Mixed-Gen Deployments: When combining Ada and Blackwell cards, utilize Pipeline Parallelism (PP) rather than Tensor Parallelism (TP) to mitigate interconnect bottlenecks. Monitor memory fragmentation closely, as the disparity in VRAM speeds can cause significant pipeline bubbles. Standardize Quantization: Evaluate the trade-offs between NVFP4 and MXFP4. For production RAG pipelines, perform rigorous Perplexity (PPL) testing to ensure that the jump to 4-bit weights doesn't degrade the model's reasoning capabilities in long-context windows.

Event CoreA developer on the r/LocalLLaMA Reddit community has sparked a firestorm in the AI hardware space by demonstrating significant progress in making NVIDIA’s Blackwell GPUs plug-and-play on macOS. While the successful recognition of Blackwell cards and driver loading is a milestone, the real "Information Gain" lies in the discovery of hidden RDMA (Remote Direct Memory Access) symbols within the macOS kernel. This suggests that Apple’s Metal framework may already possess the underlying plumbing to support zero-copy GPU memory sharing across network interfaces, a feature Apple has never publicly documented for its consumer or pro-sumer lines.In-depth DetailsTechnically, the project is currently navigating the complexities of GSP (GPU System Processor) firmware initialization over Thunderbolt 5 (TB5). While the PCIe passthrough is functional, the GSP firmware—essential for modern NVIDIA architectures—fails to boot over the TB5 link, a known hurdle currently being tackled in collaboration with the tinygrad team. However, the discovery of RDMA symbols specifically targeting Metal GPU buffers changes the narrative. RDMA allows for high-throughput, low-latency data transfer directly into memory without involving the CPU. By embedding these symbols, Apple has effectively built a foundation for a "Metal-native" version of NVIDIA's GPUDirect RDMA. This capability is the holy grail for distributed LLM training and inference, as it allows multiple nodes to share massive parameter sets with near-zero latency overhead.Bagua InsightAt 「Bagua Intelligence」, we view this as a clear signal that Apple is preparing for a future beyond the standalone workstation. The presence of RDMA symbols suggests that Apple is architecting macOS for data-center-scale deployments or high-performance compute (HPC) clusters. This discovery shatters the binary view of "Apple vs. NVIDIA." If macOS can natively handle zero-copy transfers between Metal buffers and external network controllers, it opens the door for the Mac to act as a sophisticated orchestrator for heterogeneous AI clusters. Apple isn't just building a walled garden; they are building a high-speed transit system that could eventually bridge the gap between their Unified Memory Architecture (UMA) and external accelerators. This is a strategic "sleeper cell" in the macOS kernel that could be activated to challenge the dominance of Linux-based AI infrastructure.Strategic RecommendationsFor AI infrastructure engineers, the move is clear: stop treating macOS as a mere client-side OS. The emergence of RDMA support indicates that Apple Silicon clusters (like Mac Studio arrays) may soon support high-speed interconnects comparable to InfiniBand or NVLink. For developers, we recommend tracking the tinygrad repository's progress on GSP firmware patches; a breakthrough here would instantly turn the Mac into the premier platform for heterogeneous GenAI development. For enterprises, keep a close watch on Apple’s upcoming WWDC or hardware refreshes—any mention of "Enhanced Interconnects" or "Metal Distributed Compute" will likely be the public-facing activation of these hidden RDMA capabilities. The era of the "Mac AI Server" is closer than the market realizes.