LLM-based config verification at scale

Challenge

Large, deeply nested JSON configs lived across multiple GitLab repos and branches. Changes in one file often required updates elsewhere, but manual verification was slow, brittle, and missed cross-file dependencies.

Approach

• Built a web tool that indexes JSON key paths and maps them to reference docs.
• Added LLM-guided verification with structured outputs (PASS/FAIL/UNKNOWN).
• Implemented cross-file consistency checks and optional diff coverage.
• Enabled MR-only patch proposals with human review.

Results

• Verification became repeatable and accessible for non-engineering teams.
• Cross-file inconsistencies surfaced before release checks.
• Manual review time dropped from hours to minutes.

Key metrics