Monokernel

Event Core Developers have engineered a "monokernel" for LLM inference on the AMD MI300X, executing the entire decoding sequence as a single, persistent GPU-resident program. By mapping memory access to the chip's physical topology and grouping Compute Units (CUs) by Input/Output Die (IOD), the implementation hits the hardware's theoretical performance ceiling. The result is a staggering 3,300 output tokens/s per request at Batch Size 1, achieved without the use of speculative decoding. ▶ GPU Residency: Eliminates CPU-side kernel launch overhead by keeping the entire inference loop within the GPU's execution context. ▶ Topology-Aware Engineering: Leverages the MI300X's chiplet architecture to optimize data movement across the physical silicon layout. ▶ Raw Throughput Milestone: Sets a new industry benchmark for single-request latency, proving AMD's CDNA 3 architecture can outperform H100 in specific high-speed inference scenarios. Bagua Insight This breakthrough represents a strategic pivot from generic software abstractions to hardware-native optimization. While NVIDIA relies on its massive CUDA ecosystem to maintain dominance, the "monokernel" approach demonstrates that AMD’s hardware can be a beast if you bypass the standard ROCm overhead. This is a classic "bare-metal" play—by treating the GPU as a specialized processor rather than a general-purpose accelerator, developers are unlocking performance that generic frameworks like PyTorch often mask. It signals that the next phase of the AI chip war won't just be about TFLOPS, but about who can write the most efficient, topology-aware kernels. Actionable Advice Enterprises focused on low-latency, high-throughput GenAI services should look beyond standard benchmarks and investigate custom kernel optimizations for AMD silicon. If your workload involves high-frequency, single-user interactions (e.g., real-time agents), the MI300X with a monokernel stack offers a significantly higher performance-per-dollar ratio than the current NVIDIA-centric status quo. It is time to diversify the hardware strategy by investing in specialized engineering talent capable of low-level GPU programming.