Ornith-1.5-35B: Redefining Local LLM Latency with 250 tok/s on RTX 5090
Core Event
The AI community is buzzing over the Ornith-1.5-35B-A3B model, which has set a new benchmark for local inference performance. Running on an RTX 5090 via the NInfer framework for Windows, the model achieves a blistering 250 tokens per second (tok/s) with a prefill speed of 5k-8k tokens, making it the premier choice for interactive and agentic local workflows.
- ▶ Unprecedented Throughput: Achieving 250 tok/s on consumer hardware effectively eliminates the latency bottleneck, providing a “zero-wait” user experience.
- ▶ Optimized for Agents: The model demonstrates superior reasoning capabilities for agentic tasks, where rapid feedback loops are critical for success.
- ▶ Framework Synergy: The performance gains highlight the importance of NInfer’s Windows-specific optimizations for the 35B parameter class.
Bagua Insight
The era of “smart but sluggish” local models is coming to an end. Ornith-1.5-35B proves that Mixture of Experts (MoE) architectures—implied by the A3B active parameter designation—are hitting the sweet spot for high-end consumer GPUs. By activating only a fraction of its 35B parameters per token, the model leverages the massive memory bandwidth of the RTX 50-series to deliver server-grade speed locally.
At 250 tok/s, the bottleneck shifts from the silicon to human cognition. This performance overhead is crucial for autonomous agents that require multiple internal reasoning steps (Chain-of-Thought) before delivering a final answer. When an AI can “think” and “act” within milliseconds, the friction of local AI disappears, paving the way for truly seamless human-AI collaboration.
Actionable Advice
- For Developers: Prioritize testing Ornith-1.5 within NInfer for low-latency RAG or agentic loops where real-time response is the primary KPI.
- For Hardware Enthusiasts: Use this model as the gold standard for benchmarking the inference throughput of the RTX 5090 in a Windows environment.
- For Enterprise: Re-evaluate the feasibility of local deployment for privacy-centric, high-frequency interactive tasks that previously required cloud-based APIs for speed.