TritonMoE: Breaking the CUDA MoE Monopoly with Cross-Platform Fused Kernels

A new research preprint introduces TritonMoE, an inference kernel written entirely in OpenAI Triton that achieves high-performance MoE dispatch across NVIDIA and AMD hardware by fusing gate and up GEMM operations to bypass memory bottlenecks.

• ▶ Fused GEMM as a Performance Multiplier: By fusing SwiGLU projections into a single tile load, TritonMoE eliminates 35% of global memory traffic, outperforming Megablocks on A100 for standard inference batch sizes (up to 512 tokens).
• ▶ The End of Vendor Lock-in: The kernel demonstrates true portability, running on AMD MI300X with zero code changes, proving that high-level DSLs are now competitive with vendor-specific assembly-level optimizations.

Bagua Insight

TritonMoE represents a strategic shift in the GenAI infrastructure stack. Traditionally, MoE kernels were the “black box” of LLM serving, requiring deep CUDA expertise and vendor-specific tuning. By leveraging Triton to implement a fused gate+up GEMM, this project effectively democratizes high-performance MoE kernels. The fact that it outperforms Megablocks—the gold standard for MoE—in typical inference scenarios suggests that the industry is moving past the “CUDA-at-all-costs” era. For AMD, this is a massive win; it validates the MI300X as a plug-and-play alternative for MoE workloads provided the software stack is Triton-native.

Actionable Advice

• For Infrastructure Architects: Prioritize the adoption of Triton-based kernels for MoE deployments to ensure future-proof compatibility with diverse GPU clusters (NVIDIA/AMD/Intel).
• For Performance Engineers: Focus on memory traffic reduction via operator fusion rather than raw TFLOPS optimization, as MoE inference remains primarily memory-bandwidth bound.
• For AI Startups: Utilize hardware-agnostic kernels like TritonMoE to gain leverage in cloud compute negotiations, reducing dependency on specific NVIDIA instances.