JSON Repair

A developer conducted an empirical study across 288 LLM calls—spanning Llama 3, Mistral, DeepSeek, and Qwen via OpenRouter—to catalog the specific ways models break JSON output. The findings, which led to the creation of a dedicated repair library, suggest that the gap between open-source and proprietary models in terms of formatting reliability is virtually non-existent. ▶ Structural Fragility is Model-Agnostic: Whether it is a frontier model or a local lightweight variant, LLMs consistently fail in predictable ways: unescaped characters, trailing commas, and the persistent habit of wrapping output in Markdown code blocks. ▶ Post-Processing Over Prompt Engineering: The data suggests that "prompting for perfection" is a losing battle. Implementing a robust "Repair Layer" to sanitize and fix malformed JSON is significantly more cost-effective and reliable for production-grade RAG and Agentic workflows. Bagua Insight The industry has long operated under the assumption that proprietary models hold a monopoly on reliable structured output. This report shatters that narrative. The fact that Llama 3 and GPT-4 exhibit nearly identical failure modes in JSON generation indicates that formatting logic is a fundamental challenge of the tokenization/sampling paradigm, not a measure of raw reasoning capability. For AI architects, this means the competitive advantage is shifting from "which model you use" to "how you handle the output." As constrained decoding and post-repair libraries mature, the premium for closed-source APIs for structured data tasks is becoming increasingly difficult to justify. The real moat is now the orchestration layer, not the completion engine. Actionable Advice First, move away from bloated system prompts that beg the model for valid JSON; instead, allocate those tokens to task-specific logic. Second, integrate a regex-based or grammar-constrained repair layer into your pipeline to handle common artifacts like trailing commas and Markdown syntax. Finally, for high-throughput structured data extraction, consider migrating to fine-tuned local models (e.g., Llama 3 8B or 70B) paired with a robust post-processor. This setup can match the reliability of proprietary models while slashing inference costs by an order of magnitude.