FlakyAgent state-replay test bench

Repeatable CI regression tests for third-party AI assistants with memory and account state

Problem

When teams build MCP or API integrations against proprietary assistants that carry memory and account state, tests are nearly impossible to reproduce: the same prompt yields different answers because history differs, so CI can't assert anything. Ad-hoc debug scripts pile up and regressions are caught by hand.

Solution

A testing tool aimed at the consumer side of agents: record and snapshot session state and memory, reset the environment to a deterministic state before each run, and get repeatable input/output comparisons. Assertions run as CI jobs, and failures show a state diff.

Users

3-20 person engineering teams integrating third-party AI assistants or their own agents who need behavioral regression coverage

MVP

Record an interaction and save a state snapshot; one-click replay from a pinned state; structured diff and assertions on output; one-line CLI plus GitHub Action integration; failure reports with state diff

Revenue

Subscription by project/concurrency; open-source CLI with paid cloud snapshot storage and team dashboard

Why now

Agents are moving from demo to production, memory and state break traditional unit tests, and big vendors focus on building agents rather than testing them — a gap suited to a solo builder