AI Prompt Drift Detector

Automatically detect when the same prompt behaves differently over time or across models

Problem

Users complain that the same paid account and model can be brilliant one day and unhinged the next: following a recipe then dumping gutter oil into it, or abandoning instructions entirely. Developers suspect silent routing to older models but have no independent way to quantify whether the model got dumber today. They only have gut feelings and community rants.

Solution

A lightweight prompt regression testing tool. Users save critical prompts and expected output samples. The tool periodically or manually calls the target model with the user's API key, compares outputs for semantic drift, instruction-following loss, or format breaks, and shows a visual trend plus alerts. Multi-model side-by-side comparison helps confirm routing changes.

Users

Indie developers building on LLM APIs, prompt engineers, AI product teams, power users

MVP

Save prompt sets and expected outputs; manual or scheduled calls to target models with result logging; lightweight scoring (instruction following, format, similarity) to flag drift; trend charts and email/webhook alerts; multi-model side-by-side comparison

Revenue

Free tier limits prompt count and check frequency; subscription unlocks more prompt sets, scheduled checks, history retention, and multi-channel alerts; team tier adds shared prompt libraries and audit logs

Why now

Model versions update frequently and routing is opaque, so 'it got dumber' complaints are rising. An independent, self-hostable drift detector builds trust and needs no model training, only API calls and lightweight evaluation.