Siddhant Bidarkar Operator behind p { 055 } ible
Start a conversation
Operational dossier

LLM-based config verification at scale

Built an automated config verifier across interdependent repositories, reducing manual validation effort and improving release confidence before launch.

Report metadata

AI Tooling Automation Infrastructure Release Engineering

2024-10

Solutions Architect

Manual verification reduced from hours to minutes
Cross-file inconsistencies surfaced before release
Repeatable release checks usable by non-engineering teams
GitLab Python Vector Search LLM
LLM config verification flow diagram

Challenge

Large, deeply nested JSON configs lived across multiple GitLab repos and branches. Changes in one file often required updates elsewhere, but manual verification was slow, brittle, and missed cross-file dependencies.

Approach

  • Built a web tool that indexes JSON key paths and maps them to reference docs.
  • Added LLM-guided verification with structured outputs (PASS/FAIL/UNKNOWN).
  • Implemented cross-file consistency checks and optional diff coverage.
  • Enabled MR-only patch proposals with human review.

Results

  • Verification became repeatable and accessible for non-engineering teams.
  • Cross-file inconsistencies surfaced before release checks.
  • Manual review time dropped from hours to minutes, improving release confidence.

Key metrics

Measured impact
MetricBeforeAfter
Verification timeHoursMinutes
Cross-file checksManual spot checksAutomated consistency rules
AuditabilityAd hoc notesStructured PASS/FAIL reports
Implementation notes
  1. Mappings provided the best balance between automation and precision.
  2. Deterministic outputs improved trust and adoption.
  3. Auto-proposals helped, but merge decisions stayed with humans.