DocTokenLens — Token Cost Analyzer for Docs

Estimate and cut token spend before reviewing or cross-linking large document sets

Problem

As project docs pile up, token usage for planning, test cases, and code implementation is bearable, but reviewing and cross-analyzing large document sets blows up token consumption. Developers lack tools to predict which document combinations are most expensive and how to trim context without losing useful signal.

Solution

A local-first CLI/desktop tool that analyzes token volume and duplication in a document set before feeding it to an LLM. It simulates costs for different context combinations and suggests summarization, chunked references, and deduplication. It integrates with common API pricing and outputs estimated costs.

Users

Developers, PMs, and technical writers who frequently use LLMs to review docs, audit plans, and link knowledge

MVP

Import Markdown/TXT from paste or folder; count tokens with a tiktoken-like library; show document-level and paragraph-level token heatmap; simulate costs for different context window combos; suggest deduplication and summarization; exportable report

Revenue

Free tier with monthly analysis limits; Pro subscription for unlimited analysis and multi-model pricing ($8-12/month); self-hosted enterprise option

Why now

Even as context windows grow, token cost for document review remains a real pain, especially after teams stuff knowledge bases into agent workflows; indie developers can build a precise tool on open-source tokenizers quickly