GhostCrawler – AI crawler reality check for site owners

Shows site owners how much AI crawlers read, how little traffic they send, and which pages to fix

Problem

An operator analyzed 27 days of logs across four small B2B sites and found 2,817 verified fetches from AI assistant crawlers, but only 5 visits with an AI referrer. Everyone is told to get cited by ChatGPT and Perplexity, yet no server-side tool shows that gap, which pages get crawled most, or which fetches are wasted.

Solution

Ingest web server or CDN logs, automatically identify major AI crawlers (GPTBot, PerplexityBot, ClaudeBot, etc.), and report fetch frequency, page distribution, and robots blocking. Then compare against real AI-referred visits and output a prioritized list of pages worth rewriting for summaries and structured data.

Users

Indie site owners, small B2B operators, content-driven SaaS founders

MVP

Upload Nginx/CDN logs; auto-detect and classify major AI crawlers; count fetches and last-crawl time per page; generate a crawl-vs-referral gap report; suggest 10 actionable page rewrites

Revenue

Per-site subscription (one site free, multi-site $9–19/mo); report export and competitor comparison paid

Why now

AI traffic anxiety is peaking, but the real evidence lives in server logs that GA and Plausible ignore. A log-only analyzer is cheap for one developer to build and directly answers the hot question: should I optimize for AI at all?