AgentBench Private PR Eval

Benchmark coding agents on your own PR history—no more gaming public leaderboards.

Problem

Public benchmarks get benchmaxxed fast (Goodhart's law): labs are incentivized to maximize scores, but those scores don't predict whether an agent actually works in your codebase. Developers today rely on scattered trial and error with no trustworthy private evaluation.

Solution

Automatically turn your GitHub repo's historical PRs into real task sets, let coding agents replay them in isolated environments, and score pass rate, time, and divergence in a continuously updating private leaderboard.

Users

Tech leads and platform engineers who need to select or continuously monitor AI coding agents.

MVP

Connect GitHub repos and pull historical PRs; auto-generate task descriptions and verification scripts; run chosen agents in local containers and collect results; simple pass-rate and time comparison dashboard.

Revenue

Open-source CLI is free; hosted version at $29/user/month with historical trends, team comparisons, and CI integration.

Why now

Coding agents are proliferating, selection is hard, and public benchmarks are losing credibility—private evals can start from a single personal repo and are a natural indie build.