Agent Regression Diff

Compare agent behavior before and after harness changes without burning money on LLM judges

Problem

Teams building vertical AI agents find offline evals can't cover real user questions, while LLM judges on all traffic are too expensive and can only sample a tiny slice, letting regressions slip into production. After a harness change nobody knows if the agent got better or worse except by gut feel.

Solution

A lightweight release health checker: snapshot every change to prompts, tools and model config, sample production/shadow traffic to compare old vs new outputs and key metrics, use deterministic rules (structural diffs, field hits, error codes) instead of an LLM judge for first-pass triage, and only surface suspicious samples for human review.

Users

1-10 person vertical AI agent teams and solo AI engineers

MVP

Register a change config and take version snapshots; ingest or shadow-sample real request logs; rule-based diff of old vs new outputs with highlighting; a per-change regression report; one-command or webhook recording

Revenue

Subscription by project/request volume; free tier with N comparisons per month, paid tiers unlock sampling volume and history retention

Why now

Agents are hitting production and teams constantly complain about breaking things with every change, while LLM judge cost is a recurring pain