Local Compute

Event Core A landmark achievement in the Local LLM space has been reached: developers have successfully deployed 300B-parameter Mixture-of-Experts (MoE) models on a single AMD Strix Halo (Ryzen AI Max+ 395) mini PC equipped with 128GB of unified memory. Utilizing the Kyojin engine (built on ExLlamaV3), the setup achieved impressive performance: GLM-5.3-Flash clocked 30 tok/s in decoding, while MiMo-V2.6-Flash reached a staggering 44 tok/s. This effectively shrinks the compute requirements of top-tier models from multi-GPU clusters down to a palm-sized desktop form factor. In-depth Details The technical synergy driving this breakthrough involves several critical layers: Hardware Architecture: The AMD Strix Halo platform leverages a high-bandwidth unified memory architecture. The Ryzen AI Max+ 395 bridges the gap between consumer hardware and professional workstations, offering memory throughput that rivals Apple’s M-series Ultra chips, which is essential for LLM inference. Software Optimization: The use of EXL3 weights and the Kyojin engine (optimized for ROCm) is pivotal. The engine delivers exceptional prefill speeds—580 tok/s for GLM-5.3-Flash and 650 tok/s for MiMo-V2.6-Flash—making it highly viable for long-context RAG (Retrieval-Augmented Generation) applications. MoE Efficiency: Despite the 300B total parameter count, the sparse activation nature of MoE models allows them to run efficiently within the 128GB VRAM envelope when properly quantized, avoiding the latency penalties of system memory swapping. Bagua Insight At Bagua Intelligence, we view this as a pivotal moment for "Sovereign AI." For years, Apple’s Mac Studio was the only viable path for running massive models locally, albeit within a walled garden. AMD’s Strix Halo, paired with the maturing ROCm ecosystem, represents a formidable open-source alternative that challenges Apple's dominance in the prosumer AI market. The economic implications are profound. When a sub-$2,000 device can generate text at 40+ tok/s—faster than any human can read—the value proposition of expensive, privacy-compromising cloud APIs begins to erode. This is not just a benchmark win; it is the democratization of high-parameter intelligence, shifting the power dynamic from centralized hyperscalers back to the edge. Strategic Recommendations For Hardware OEMs: Shift focus toward high-bandwidth, large-capacity unified memory configurations. The future of the AI PC is defined by memory bus width and VRAM capacity, not just NPU TOPS. For Enterprise Architects: Re-evaluate the TCO of private AI deployments. Localized 300B models on AMD-based workstations now offer a competitive, secure, and cost-effective alternative to cloud-hosted frontier models for sensitive RAG tasks. For Developers: Double down on the ROCm and EXL3 ecosystem. The ability to run and potentially fine-tune 300B+ models locally opens up a new frontier for specialized agentic workflows that were previously cost-prohibitive.