RelayCheck: Model quality test bench for AI API resellers

A reproducible quality comparison tool for developers using third-party AI API relays

Problem

Many developers use third-party relay services to access coding models for cheaper prices or when official quotas are unavailable, but relays may silently downgrade models or inject cheaper versions. One user handed the same requirements doc to a relayed GPT and to DeepSeek: the first burned money for three hours and produced nothing usable, the second delivered working features in half an hour — but there was no way to know beforehand which one was genuine.

Solution

A local test bench with a fixed set of coding tasks (small feature implementation, bug fixing, refactoring). Users plug in a relay endpoint and run one-click benchmarks that output reproducible quality, cost, and latency comparison reports. Test cases and scoring criteria are public, replacing gut feeling with data so developers can immediately see which relay is trustworthy.

Users

Indie developers using third-party AI relays, tech leads at small teams, heavy users of AI coding tools

MVP

Around 10 standardized coding tasks with automated scoring scripts; one-click OpenAI-compatible endpoint integration; quality/cost/latency comparison report; export results as Markdown for sharing;

Revenue

Free tier runs the basic task set; paid tier adds more tasks, historical trends, and team dashboards; affiliate deals with relay providers

Why now

AI coding is now a mainstream workflow, relay demand is strong but quality varies wildly, and users lack a neutral verification tool; the engineering scope is small enough for one developer to prototype in a week.