Agent Skill Eval Bench

Let developers systematically evaluate AI agent skills like running unit tests

Problem

More people are building AI agents with custom skills, but whether a skill works well still relies on reading SKILL.md and trying it a few times by hand; there is no repeatable evaluation method, and no one knows if behavior regresses when a skill is updated or the model is switched.

Solution

A lightweight eval framework: define input samples and expected behaviors for each skill, run batch evaluations with one click, and generate pass-rate and diff reports, with version and model comparison.

Users

Indie developers, small AI teams, engineers putting agents into production

MVP

Define test cases in YAML; integrate mainstream APIs like OpenAI/Claude; batch run and output pass rates; record version and model per run; simple web result dashboard

Revenue

Open-source core + hosted subscription (per run or per seat); enterprise private deployment

Why now

The agent skill ecosystem is just emerging and evaluation is an obvious gap; it doesn't need massive data, and one person can cover the core scenarios