Ternary Neural Networks

PR #24448 introduces Q2_0 quantization support to the llama.cpp ecosystem, specifically targeting the Ternary Bonsai 1.58-bit model family for high-efficiency CPU-based inference. ▶ Completing the Spectrum: The addition of Q2_0 fills the critical gap in the Q1_0-Q8_0 quantization suite, optimized for the unique {-1, 0, 1} weight structure of ternary architectures. ▶ Edge AI Catalyst: With initial support for ARM NEON, this move positions high-parameter models (up to 8B) for efficient execution on mobile and embedded hardware with minimal power envelopes. Bagua Insight The shift toward 1.58-bit (ternary) models represents the most significant paradigm shift in LLM deployment since the advent of 4-bit quantization. By constraining weights to {-1, 0, 1}, we are effectively moving away from the "Matrix Multiplication Tax." This PR in llama.cpp is the bridge from academic research (Bonsai) to production-ready edge AI. While the current implementation focuses on CPU scalar and NEON backends, the roadmap for CUDA and Metal support suggests a future where memory bandwidth—not compute—is the only bottleneck. We are witnessing the birth of the "Addition-only" inference era, which will redefine the performance-per-watt metrics for local LLMs. Actionable Advice AI Engineers should prioritize benchmarking the Bonsai 8B model for local RAG and agentic workflows, as the memory footprint reduction allows for significantly larger context windows on consumer hardware. Hardware architects should view this as a signal to optimize silicon for ternary logic, moving beyond traditional FP16/INT8 pipelines to capture the next wave of on-device GenAI efficiency.