SandboxBench – Neutral sandbox provider benchmark

Help AI agent developers compare code sandbox startup latency and cost at a glance

Problem

A new sandbox provider seems to launch every week, yet developers have no objective way to compare startup latency, cold start time, and cost per execution. One HN user directly asks whether there is an objective third-party benchmark that specifies architecture, startup latency, cold start, and cost. Another asks the same about the growing number of Jev alternatives and cannot find a benchmark list.

Solution

Build an open-source benchmark service that continuously measures mainstream sandboxes and runtimes on standardized tasks—cold start, startup latency, throughput, and unit cost—and presents a filterable comparison dashboard with a public API. All test scripts are open source, and providers can submit adapters to join the benchmark.

Users

AI agent developers, DevOps engineers, and teams choosing a code execution environment.

MVP

Pick 5–8 mainstream sandboxes (e.g., E2B, Modal, Daytona); design a set of standard measurement tasks logging latency and cost; provide a sortable public comparison page; support a simple PR adapter submission flow for providers; expose results via a JSON API.

Revenue

Free basic comparisons; paid historical trends, custom private benchmarks, and CI integrations; or API usage-based billing.

Why now

The sandbox space is exploding, there is no neutral third-party evaluation, and developers face high decision costs. An indie developer can build credibility through continuous automated measurement.