Model Nerf Tracker

Automatically detect and alert on silent AI model degradations for heavy users

Problem

Developers keep noticing that models get silently nerfed: the same prompt that produced great output before now returns lazy or worse answers. Some devs have started building nerf trackers, showing this is a real, recurring anxiety, but most detection is still manual vibes. Users need an objective, automated, auditable degradation log.

Solution

Users pick which models to watch and which task suites matter. The product runs a fixed benchmark prompt set daily, covering code, translation, reasoning, long context, and refusal rate, then charts capability curves and alerts by email/SMS when a threshold is crossed. Raw outputs are kept for side-by-side comparison and screenshots.

Users

Developers who rely heavily on Claude Code / Codex / API models, AI app founders, and teams that must explain model behavior swings to clients

MVP

30-50 high-quality benchmark prompts covering common tasks; support 3-5 mainstream model APIs; daily scheduled runs with charts; email alert on threshold breach; viewable and exportable output history.

Revenue

Subscription by number of models and frequency ($9-$29/mo) with a higher tier that includes API and history export; a public read-only community dashboard as a lead magnet.

Why now

Silent model updates are frequent and user-perceived quality drops lack objective evidence; big labs won't build transparency tools, leaving room for indie devs. Technically it's just scheduled jobs and simple stats, shippable in a few weeks.