PelicanBench – longitudinal model benchmark with reproducible submissions

Save source, not screenshots, when comparing models across reasoning tiers and tools

Problem

People test model degradation with prompts like 'pelican riding a bicycle', but usually post a screenshot and move on. The same model at a different reasoning tier or tool chain produces very different output, and there is no way to retrieve the original for later comparison or turn the discussion into reusable data.

Solution

Store each run's prompt, model, reasoning tier, tool chain and original SVG/HTML source in a searchable benchmark library. Animated entries run in an isolated sandbox subdomain, and you can generate shareable side-by-side comparison pages.

Users

Heavy AI users, model-evaluation hobbyists, engineering teams doing model selection, content creators

MVP

Upload form capturing prompt, source, model and tier; filter and side-by-side compare views; safe SVG rendering and sandboxed animation subdomain; shareable comparison links per submission; public leaderboard and latest-submissions feed.

Revenue

Free public browsing; subscription for private benchmark sets, automated re-runs across models and exportable reports; sponsored slots from model vendors or tool makers

Why now

Model-degradation debates keep heating up but the community has no structured archive; screenshots are the status quo, so whoever turns this into a reproducible benchmark library first accumulates the data, and a small team can ship it.