Open Source LLM Observability
As of August 2026, AIDiveForge tracks 15 open source llm observability. The top three by verified-data score are burnban, Causari, and J-lens Qwen. Curated open source llm observability tracked by AIDiveForge. Each project has a verified public source repository. Listings are verified against each tool's live website and re-checked regularly.
Last updated July 17, 2026 · 15 tools
Ranked by AIDiveForge's verified-data score: data completeness, verification recency, community rating, and real visitor engagement. How we rank · No tool can pay for placement.
1. burnban
Burnban reads supported agent log files already sitting on disk, prices the recorded usage against public API list rates, and lets you set daily, weekly, monthly, or per-agent spend caps enforced in the request path — all from a local dashboard at localhost:4141. The ledger is SQLite on your machine. No keys leave to a Burnban server, no prompts hit a control plane, no account is required. The sharp edge is the word 'supported': log format and provider coverage are scoped, and anything outside that scope remains invisible to the meter. Teams tracking unsupported agents or providers find Burnban shows them a partial picture.
FreeOpen SourceSelf-hostedVerified Jul 17, 2026
2. Causari
Causari is a content-addressable ledger that sits between your agents and both the LLM API and the filesystem, joining wire-captured prompts to disk-captured edits by matching the lines an agent wrote against the completions it received moments before. The causal fingerprint requires no cooperation from the agent itself — a passive proxy and a filesystem watcher do the work. Distilled sessions become signed, portable skills that agents can query before acting, turning past verified work into ranked recall. The architecture holds well for single-repo, single-team debugging and provenance. Teams running distributed, multi-repo setups or wanting hosted verification will hit the boundary of what the self-hosted binary covers — that functionality sits behind a paid-only Trust Plane.
PaidOpen SourceAPISelf-hostedVerified Jul 8, 2026
3. J-lens Qwen
jlens-qwen36 fits a Jacobian lens to a local Qwen3.6-27B (4-bit) model running on Apple Silicon via MLX, then renders a layer-by-layer visualization of which tokens the model is pushing toward at each position during generation. The canonical demo is a blackmail email prompt: the model outputs a calm, compliant reply, but the workspace band exposes what the intermediate layers were actually predicting — a gap that standard logit inspection never surfaces. This is a single-model, single-hardware tool. It runs on macOS with MLX; there is no cloud backend, no REST API, and no adapter for any other model family. Researchers without Apple Silicon and Qwen3.6-27B (4-bit) specifically cannot run it at all.
FreeOpen SourceSelf-hostedVerified Jul 14, 2026
4. Subtext
Subtext is an open-source, self-hosted tool that surfaces a local language model's internal representations — specifically the J-space global workspace identified in Anthropic research — as the model reads and generates, before output tokens appear. You get a browser-based live view and the ability to record, export, and replay sessions for later analysis. The stack is Python-based and runs against local models, so there is no cloud dependency and no data leaves the machine. The ceiling appears quickly: Subtext has no API, no integration hooks, and no support for models it cannot instrument directly — which means anything running behind a remote endpoint stays opaque.
FreeOpen SourceSelf-hostedVerified Jul 14, 2026
5. role-model
role-model is an open protocol and reference router runtime that routes LLM requests across endpoints using declared capability profiles, routing policy, and measured performance — then emits a structured decision artifact you can inspect after the fact. The router narrows candidates by role and task metadata, rejects endpoints that fail capability, locality, or budget checks, and scores what's left against latency, cost, and reliability evidence. The decision is deterministic enough to audit, not a hidden runtime guess. The baseline role set covers chat, code editing, code review, tool calling, and embeddings — so it fits mixed-workload deployments out of the box. Teams needing autonomous multi-step planning or tool loops will find this deliberately out of scope.
FreeOpen SourceAPISelf-hostedVerified Jun 30, 2026
6. AgentMeter
AgentMeter runs locally — no cloud sync, no account creation, no vendor dashboard to log into — and parses the tool calls, token counts, and caching splits that CLI agents like Claude Code, Gemini CLI, Codex CLI, and Copilot CLI generate. It surfaces the three-tier cost structure that prompt caching creates (input, cached-input, and output tokens each priced differently), which the raw API bill flattens into noise. The value-multiplier calculation compares API spend against estimated developer time saved, giving you a number to put in front of a manager. The wall appears when you need alerting, real-time budget enforcement, or integration with a team billing system — none of that is here.
FreeOpen SourceSelf-hostedVerified Jun 12, 2026
7. Beacon
Beacon is an open-source endpoint telemetry layer that runs locally alongside AI agents, capturing prompts, tool calls, file modifications, and approval workflows before any of that activity disappears into the void. It normalizes that telemetry and forwards it to SIEM platforms like Wazuh, Elastic, or Splunk, so security teams can apply the same detection logic they already run against the rest of the fleet. The architecture is self-hosted by design — no data leaves the endpoint unless you route it there yourself. The project is early-stage; the plugin ecosystem covers the major local agent harnesses but gaps exist for less common runtimes. Teams with agents not yet on the supported list write custom collector plugins — which means more surface area to maintain.
FreeOpen SourceSelf-hostedVerified Jun 1, 2026
8. Flightdeck
Every LLM call, MCP event, and tool invocation your agents make streams to a live dashboard — per-agent timelines and a fleet-wide feed, not batched logs you dig through after the incident. The vendor describes token budgets and MCP allow/block rules you set before problems hit, plus the ability to issue live directives to running agents without restarting them. The self-hosted, Apache-2.0 model means no telemetry leaves your infrastructure — critical for teams in regulated environments or those burned by SaaS observability vendors billing by event volume. The project is early-stage by star count, and the operational surface you take on by self-hosting is real.
FreeOpen SourceSelf-hostedVerified Jun 12, 2026
9. Foglamp
Foglamp is an observability layer built for production AI agents: two lines of SDK integration wrap every `generateText` and `streamText` call and surface cost, latency, distributed traces, per-agent spend, and output quality in one place. The instrumentation is designed specifically around the Vercel AI SDK, so teams already on that stack see immediate coverage without rethinking their pipeline. Evals and alerts let you catch output regressions before users file support tickets. The ceiling appears when your stack moves outside Vercel AI SDK conventions — the docs describe no native integrations for other frameworks, and teams on LangChain or custom agent loops will need to assess how much of the trace fidelity carries over.
PaidOpen Source$49/monthAPIVerified Jun 19, 2026
10. Latitude LLM
Latitude is an open-source AI agent monitoring platform that captures full conversation traces, clusters similar failures into triage-ready issue groups, and turns confirmed failure modes into automated evaluations that run against every new trace. The vendor states it ingests via OpenTelemetry, so teams already using OTEL pipelines point their existing setup at Latitude without reformatting data. Semantic search runs across 100% of traces — no sampling — which means finding 'frustrated users on a specific model version after a specific release' takes filters, not queries. The ceiling appears when your team needs the monitoring layer to also drive prompts or chain agents: that is not what this tool does.
PaidOpen SourceFree Trial · 30 days$99/monthAPISelf-hostedVerified Jun 24, 2026
11. Orchid
Orchid sits between your agent and any API it talks to, capturing traffic into a local SQLite file — no cloud account, no SDK changes, no telemetry leaving your machine. The built-in web UI lets you step through a completed run, inspect every prompt, response, token count, and cost. The proxy also runs a built-in MCP server, so your IDE assistant in Cursor, VS Code, or Claude Code can query recorded traffic directly. Replay is deterministic and costs nothing in API fees. The ceiling appears when your team needs cross-service aggregation or production alerting — this tool is a local debugger, not an observability platform.
FreeOpen SourceSelf-hostedVerified Jun 25, 2026
12. PromptShark
PromptShark is a local MITM proxy written in Go and C++ that sits between your agent and any OpenAI-compatible API endpoint. Every request and response pair is captured, logged, and surfaced in a real-time dashboard — no changes to your agent code, just a single base_url swap. The loop detector flags infinite tool-calling cycles automatically. The time-travel replay feature lets you re-run or edit any historical API step without firing a live request, which means no extra token spend during debugging. The self-hosted, MIT-licensed architecture means your prompts and API keys never leave your own infrastructure.
FreeOpen SourceAPISelf-hostedVerified Jun 18, 2026
13. Rootsign
RootSign is an open-source Python library that attaches tamper-evident provenance logging to AI agent actions — tool calls, API hits, database writes — capturing a verifiable record of what happened, in what order, and under whose authorization. The vendor describes it as the agent capture layer of a broader Agent Accountability Platform. It installs via pip and ships a Docker Compose quickstart for self-hosting, so the audit trail stays inside your infrastructure. The library integrates with LangGraph and CrewAI by wrapping agent actions at the point of execution. At low log volume the architecture holds; teams with high-throughput agents running thousands of tool calls per hour will hit questions the current documentation does not answer about storage scaling and query performance.
FreeOpen SourceSelf-hostedVerified Jun 20, 2026
14. Selvedge
Selvedge is a local MCP server that AI coding agents (Claude Code, Cursor, Copilot) call as they work, logging the reasoning behind every change into a SQLite file that lives next to your code under .selvedge/. Queries are entity-scoped — you ask about users.email or deps/stripe, not line numbers — so the answer surfaces in the same terms you search in. The vendor describes zero telemetry, no accounts, and no external servers; everything stays on disk. The wall appears when your team needs cross-repo provenance or wants to pipe this data into an existing observability stack — Selvedge emits records but does not integrate with those systems out of the box.
FreeOpen SourceSelf-hostedVerified Jun 1, 2026
15. Spanlens
Spanlens sits in front of your LLM provider via a single baseURL change, recording every call's cost, latency, tokens, and full request-response body with no SDK rewrite required. Agent runs surface as waterfall span trees so you can identify the one step consuming 80% of wall-clock time. The model recommender flags GPT-4o calls that look like classification tasks and shows the cost delta if you swap — with numbers from your own traffic, not benchmarks. The eval and experiment layer lets you replay a fixed dataset across prompt versions before you ship, so quality regressions don't surprise you in production. PII scanning and anomaly detection run at log time, which matters when sensitive data crosses the wire at 3 a.m. with nobody watching.
PaidOpen Source$29/moAPISelf-hostedVerified Jun 9, 2026
Listings on this page are sourced and verified by the AIDiveForge data pipeline. AIDiveForge is editorially independent — no money changes hands for inclusion.