Spanlens

Most LLM cost overruns are invisible until the invoice lands. Spanlens intercepts every OpenAI, Anthropic, and Gemini call at the proxy layer — capturing model, tokens, cost, latency, and full body — then organizes that data into per-request logs, daily rollups, budget alerts, and per-user attribution. The SDK (TypeScript and Python) and a CLI that rewrites your baseURL are the two entry points; the vendor describes the CLI path as requiring one line to instrument existing code.

The differentiating feature is the combination of agent trace trees with an experiment layer that reads from the same span store. A multi-step agent run surfaces as a waterfall with critical path highlighted and per-LLM, per-tool cost attributed at each node — so debugging an 18-second support agent means clicking to the exact summarize step rather than reading logs. That same span data feeds the experiment runner: capture real traffic into a dataset, score responses with an LLM-as-judge (0–1 per version), and replay the dataset across prompt versions and models side by side before deploying. The vendor’s own example shows a prompt-plus-model swap from GPT-4o v7 to GPT-4o-mini v8 yielding a 0.11 quality improvement and a 94% cost reduction on the same 320-case dataset.

Spanlens fits teams that are already running agents or multi-step workflows and need evidence — not intuition — to justify model swaps, catch budget spikes, or sign off on a new prompt version. It is self-hostable via Docker Compose under an MIT license, which the vendor states directly, making it viable for organizations with data residency requirements. The PII scanning is regex-based at log time with API keys auto-masked before storage — adequate for flagging and review workflows, not a substitute for a dedicated data-loss-prevention pipeline if your compliance posture requires guaranteed redaction before data touches any storage layer.

The free hosted tier handles up to 50,000 requests per month; above that, paid cloud tiers apply. The SDK version at time of listing adds Ollama and LangGraph tracing, covering local LLM runs alongside hosted providers.