Event Core
A developer in the LocalLLaMA community has successfully reverse-engineered the proprietary engine format of the Axera AX8850 NPU, enabling direct GGUF model execution via llama.cpp. By bypassing the vendor's closed-source toolchain and runtime, the implementation achieved a staggering 21-22 tokens per second (t/s) on a Qwen3-0.6B model—outperforming the official vendor runtime (13.5-14.5 t/s) by approximately 50%. The feat was demonstrated on the M5Stack LLM-8850, a Raspberry Pi 5-powered edge device.
In-depth Details
The technical breakthrough centers on deciphering how the AX8850 handles memory layout for INT8 weights. The hardware utilizes a "two nibble planes" format, splitting 8-bit weights into two 4-bit segments stored across different memory planes to optimize NPU throughput.
The Hack: Instead of relying on the vendor’s opaque conversion tools to generate .axmodel files, the developer wrote a custom loader that reshuffles GGUF weight tensors into the required nibble-plane structure in real-time.
Architecture Integration: By integrating this as a llama.cpp backend, the developer leveraged the framework's robust feature set (e.g., KV cache management, advanced sampling) while utilizing the raw power of the NPU.
Optimization Paradox: The 1.5x speedup suggests that the vendor's proprietary runtime is bogged down by unnecessary overhead or suboptimal kernels, highlighting a common gap between hardware potential and software execution in the NPU industry.
Bagua Insight
At 「Bagua Intelligence」, we view this as a pivotal moment for the Edge AI landscape. It signals the end of the "Software Lock-in" era for hardware vendors.
1. The GGUF Hegemony: GGUF is effectively becoming the "PDF of LLMs." Developers are no longer willing to jump through the hoops of proprietary SDKs. If a hardware vendor doesn't provide a llama.cpp driver, the community will build one—often outperforming the vendor’s own engineers in the process.
2. Hardware is a Commodity, Ecosystem is the Moat: The AX8850 is a capable piece of silicon, but its value was capped by its software barriers. This reverse-engineering effort essentially "liberated" the hardware, making it viable for the broader open-source AI community. Vendors who resist this trend risk becoming irrelevant in the face of "Open-First" silicon.
3. Democratizing Edge Intelligence: Achieving 20+ t/s on a sub-$100 edge setup (Raspberry Pi + NPU) proves that local LLM deployment is moving past the hobbyist phase into serious industrial and consumer applications without the "NVIDIA Tax."
Strategic Recommendations
For Silicon Vendors: Pivot your software strategy. Stop trying to win the "Runtime War." Instead, focus on being the best-supported backend for llama.cpp and ONNX Runtime. Open-sourcing your memory layout specifications is no longer a risk—it’s a prerequisite for adoption.
For Enterprise Buyers: When sourcing edge AI hardware, prioritize "Time to Hello World" over theoretical TOPS. A chip that requires a proprietary, buggy toolchain is a long-term liability.
For the Open Source Community: This success story provides a blueprint for unlocking other proprietary NPUs (like those from Rockchip or MediaTek). The focus should remain on building unified abstractions that treat various NPUs as pluggable backends.
SOURCE: REDDIT LOCALLLAMA // UPLINK_STABLE