Self-Hosted Inference Engines & Infra
As of September 2026, AIDiveForge tracks 105 self-hosted inference engines & infra. The top three by verified-data score are Plugsky.com, OmniRoute, and Patronus Scanner. Curated self-hosted inference engines & infra tracked by AIDiveForge. Listings are verified against each tool's live website and re-checked regularly.
Last updated September 20, 2026 · 105 tools
Ranked by AIDiveForge's verified-data score: data completeness, verification recency, community rating, and real visitor engagement. How we rank · No tool can pay for placement.
1. Plugsky.com
Plugsky routes calls across 31+ models through a single OpenAI-compatible endpoint on fixed monthly pricing, so a traffic spike doesn't trigger a surprise invoice or a throttle. Agent Cloud adds function-calling, memory, and task chaining, with an OpenAI Assistants-compatible interface that the docs describe as a one-line migration path. Private RAG, white-labeling, and sovereign deployment options cover regulated and SaaS use cases without requiring separate vendors. The self-hosted path is concrete — the page shows working curl commands for CLI, Desktop, and Web installs. Where the architecture gets less clear is at the edges: community-reported depth on complex multi-step branching logic inside Agent Cloud, and independent validation of the stated throughput figures, remains thin.
PaidFree Trial · 7 daysfrom $5APISelf-hostedVerified Jul 13, 2026
2. OmniRoute
The vendor describes OmniRoute as a self-hosted gateway that exposes a single OpenAI-compatible endpoint at localhost:20128/v1 and routes requests across 268 providers, with automatic fallback — the docs state a sub-10ms switch when quota runs out on any one provider. Sixteen-plus coding agents, including Claude Code, Cursor, and Copilot, point at that one endpoint without reconfiguration. Token compression via stacked RTK and Caveman algorithms cuts 15–95% of tokens on tool-heavy sessions, which keeps free-tier quotas lasting longer. The circuit breaker operates per provider, so one bad key does not take down the whole pool.
FreeOpen SourceAPISelf-hostedVerified Jul 26, 2026
3. Patronus Scanner
Patronus AI positions itself as evaluation and monitoring infrastructure for production LLM applications, with its Lynx model — the vendor states it beats GPT-4 on hallucination detection tasks — as the core detection engine. The platform covers automated evaluation via API, multimodal image-text alignment checks, and experiment tracking for prompt and model optimization. Self-hosted deployment is available for teams with data residency requirements, which means regulated industries can run evaluations without shipping sensitive outputs to a third-party endpoint. The free tier gets you started, but the on-premises and enterprise-grade monitoring features are paid-only. Teams operating at high evaluation volume will hit throughput questions that the docs do not answer publicly.
Paid$10 / 1k small evaluator API calls; $20 / 1k large; Enterprise customAPISelf-hostedVerified Sep 20, 2026
4. Contextual
The engine indexes your repository — symbols, commits, PR threads, docs — into a local graph stored in ~/.contextual, then keeps it live with a filesystem watcher that reindexes on save and forks the timeline on branch. Semantic search returns not just the matching code, but the commit that changed it, the PR discussion that explains why, and a staleness signal telling you if that context is stale. Zero bytes leave the device; embeddings run on-device with CodeRankEmbed. The wall appears when you need multi-repository memory, team-shared context, or any workflow that requires more than one developer's machine to be in sync — none of that is described in the docs, and the tool is in beta.
PaidFree Trial · 14 days$10 /moSelf-hostedVerified Sep 8, 2026
5. Eclusia
Eclusia runs locally on your machine and replaces names, IBANs, phone numbers, API keys, and social security numbers with neutral placeholder tokens before you send anything to an external AI. The anonymized version goes to ChatGPT, Claude, or whichever tool you choose; the real values never leave your machine. After the AI responds, Eclusia swaps the tokens back in one step. The detection engine has a dedicated French-language name model — a real edge in a category where most tools are English-first. The ceiling appears at scale: the free tier covers three documents per thirty days, paid tiers cap out as well, and teams processing high document volumes at speed will find no API to wire Eclusia into a pipeline.
Paid6.99 €/month or 19.99 €/monthSelf-hostedVerified Sep 9, 2026
6. Fabraix Red Team Playground
Fabraix Red Team Playground is an open-source, MIT-licensed testing environment structured around CTF-style challenges that let you probe published agent prompts for prompt injection vulnerabilities and guardrail failures. The workflow is concrete: challenges present a live agent prompt, you craft adversarial inputs, and the platform scores whether your attack broke containment. Self-hosting via npm means your proprietary prompts never leave your environment, which matters when the thing you are testing is a production system prompt you cannot share publicly. The challenge library is community-contributed and public, so coverage depends entirely on what the research community has submitted — niche agent architectures with specialized guardrail logic will find thin coverage fast.
FreeOpen SourceAPISelf-hostedVerified Aug 14, 2026
7. LM-Kit One
The vendor describes LM-Kit One as a self-hosted AI application server that exposes OpenAI, Anthropic, and Ollama-compatible endpoints, so existing clients reconnect by swapping a base URL. Document extraction, vector search with source citations, and function-calling agents with allow/deny rules all run on your hardware, with no per-token billing regardless of query volume. The single-engine architecture means updates ship as one versioned release rather than coordinated upgrades across mismatched libraries. The ceiling appears when teams need deep Python ecosystem integrations — the .NET-native engine fits .NET shops naturally and creates friction elsewhere. Teams that need GPU cluster scheduling or fine-tuning pipelines will hit gaps the server does not address.
PaidAPISelf-hostedVerified Sep 16, 2026
8. AutoYou
AutoYou runs a local AI server on your Windows, macOS, or Linux machine, then lets your phone reach it over an encrypted peer-to-peer route — no third-party cloud in the middle. Chat, live voice calls, note-taking, remote desktop control, and custom agents all run on hardware you own, using models you choose. The QR pairing flow connects a mobile client in under two minutes, and the x402 micropayment protocol means you pay per request rather than committing to a subscription. The ceiling appears fast for teams: there is no API surface, so integrating AutoYou into an existing backend or CI pipeline requires workarounds the vendor does not document. Solo users and experimenters hit a productive ceiling before teams do.
PaidSelf-hostedVerified Sep 20, 2026
9. Semelbase
The vendor positions Semelbase as a ledger and routing layer that sits between your agents and your AI providers, deduplicating inference requests, surfacing usage across teams, and directing traffic to whichever provider fits your cost and budget constraints at that moment. The audit capability gives engineering leads a cross-team view of what is being called, how often, and what it costs — a gap that becomes expensive when five teams are running overlapping agents against the same endpoints. The self-hosted path matters for organizations that cannot route production inference traffic through a third-party cloud. Where this architecture shows strain is in setups with highly dynamic, non-repeatable prompts, where deduplication recovers little and routing optimization becomes the only lever.
PaidAPISelf-hostedVerified Aug 16, 2026
10. Agentmetry
Agentmetry is an open-source, local-only session recorder for AI coding agents — Cursor, Claude Code, Codex, Antigravity — that hooks at the tool boundary, tags every call with MITRE ATT&CK, and fires correlated alerts when a chain of events matches a known attack pattern. The trail is a hash-chained JSONL file that never leaves your machine; Merkle proofs let you hand a single-event proof to an auditor without forwarding the full log. It ships with 15+ sequence detection rules and three native SIEM sinks — Elastic ECS, Splunk HEC, and webhook. The ceiling is the ruleset: detection coverage is what the project's 15+ rules plus your custom additions cover, and anything outside that scope passes through silently.
FreeOpen SourceSelf-hostedVerified Aug 14, 2026
11. Panguard.AI
Panguard installs in one command, runs entirely offline with zero telemetry, and auto-detects agents across a wide surface — Claude Code, Cursor, VS Code Copilot, Gemini CLI, and more. The vendor states 768 ATR (Agent Threat Rules) execute locally as deterministic checks before any skill loads, then continue guarding each action at runtime against prompt injection and poisoned MCP tools. Rules contributed anywhere benefit every adopter — the vendor describes this as 'threat crystallization.' The ceiling appears when a threat is genuinely novel: deterministic rules only catch what someone has already seen and codified, so the AI analysis fallback carries the weight for zero-day patterns. Teams with regulated environments get signed, audit-ready output without routing data to a third party.
FreeOpen SourceSelf-hostedVerified Jul 20, 2026
12. TokenTelemetry
The tool installs in one curl command, requires no SDK integration, and auto-detects up to 16 agents — Claude Code, Codex, Cursor, Gemini CLI, and a dozen others — by scanning log files on disk. Nothing leaves the machine. The dashboard shows token counts, cost totals, session traces, and reasoning chains per agent. Hermes Agent operators get a dedicated surface covering gateway health, subagent delegation chains, skills, memory state, and cost anomaly detection for silent reasoning-token waste. The ceiling appears fast for teams whose agents do not write logs in a format the parser understands, or who need cross-machine aggregation.
FreeOpen SourceSelf-hostedVerified Aug 14, 2026
13. CMEM
The open-source claude-mem engine hooks into Claude Code, Cursor, Windsurf, and CLI agents, writing decisions and dead ends into a local SQLite observations database as your agent works. CMEM Cloud mirrors that database behind a private MCP endpoint any agent or IDE can read, so the context one agent built in one session is available to the next one without manual handoff. Vector search over the observations store means retrieval is semantic, not keyword-based — you query by meaning, not by remembering what you typed three sprints ago. The ceiling appears at the team coordination layer: role-based read/write scoping and per-project isolation are paid-only features, so solo developers get the full engine but teams hit a paywall before they get the shared-brain behavior the product is built around.
PaidAPISelf-hostedVerified Jul 26, 2026
14. Strix
Strix runs multi-agent Think-Plan-Act loops that scan infrastructure, attempt exploitation, and return findings backed by working PoCs — so your team reviews confirmed vulnerabilities rather than triaging noise. The vendor page describes CI/CD integration and PR-level fix suggestions, which means security gates can live inside the same pipeline where code ships. Self-hosted and air-gapped deployment options are confirmed, making it viable for teams with data residency requirements. The agentic model works well when scope is defined and targets are enumerable — cloud misconfigs, known CVE classes, API surface. Where it strains is against novel logic flaws and business-layer vulnerabilities that require context no automated agent carries.
PaidOpen SourceSelf-hostedVerified Jul 14, 2026
15. Cortega AI Governance Platform
Cortega deploys as a gateway layer on infrastructure you control, intercepting every LLM call — from browsers, agents, servers, and cloud runners — and running identity verification, data inspection, and policy enforcement before the request egresses. The control plane manages policy, identity, routing, and budget caps across every gateway; the analytics plane turns that governed traffic into an org-wide picture of who is calling what model, at what cost, with what data categories. Where it earns its place is regulated environments: audit trails record identity, data category, policy decision, approval, model, provider, and outcome — the evidence reviewers actually ask for, not a raw log dump. The ceiling appears when your governance requirements demand deep application-layer logic that lives outside the gateway path, or when your teams are still in early AI experimentation and the architecture review overhead exceeds the compliance exposure you are trying to close.
PaidAPISelf-hostedVerified Aug 16, 2026
16. NeuralTrust TrustGate
NeuralTrust TrustGate sits between your agents and the models, tools, and services they reach, enforcing policy at the interaction level rather than bolting controls on after the fact. The vendor states the gateway handles real-time enforcement at sub-100ms latency and claims behavioral, contextual, and multilingual detection across 22 million-plus AI interactions analyzed. Self-hosted deployment via on-prem or VPC keeps data inside your perimeter — a hard requirement in regulated industries where data leaving the environment ends the conversation. The open-source core is Apache-2.0 licensed, which means your security team can audit what is actually running. Enterprise features, SIEM integration, and dedicated support are paid-only.
PaidOpen SourceAPISelf-hostedVerified Aug 14, 2026
17. burnban
Burnban reads supported agent log files already sitting on disk, prices the recorded usage against public API list rates, and lets you set daily, weekly, monthly, or per-agent spend caps enforced in the request path — all from a local dashboard at localhost:4141. The ledger is SQLite on your machine. No keys leave to a Burnban server, no prompts hit a control plane, no account is required. The sharp edge is the word 'supported': log format and provider coverage are scoped, and anything outside that scope remains invisible to the meter. Teams tracking unsupported agents or providers find Burnban shows them a partial picture.
FreeOpen SourceSelf-hostedVerified Jul 17, 2026
18. Engramma Memory
The library combines exact kNN search, Hopfield energy networks, and multi-head attention in a single local install, so agents can retrieve, pattern-complete, and generalize across stored knowledge without stitching together separate systems. The dependency surface is intentionally minimal — NumPy and nothing else — which means local prototyping adds no infrastructure overhead. The ceiling arrives when you move beyond a laptop: local mode has no persistence layer built for concurrent production writes, and the path to production runs through Engramma Cloud, a paid-only hosted backend. Teams scaling beyond local experiments will be evaluating that cloud offering rather than a self-managed stack.
PaidOpen SourceSelf-hostedVerified Jul 9, 2026
19. Free AI Prompt Gen
The tool walks you through established prompt engineering frameworks — RTF, CO-STAR, TAG, APE — with labeled fields for role, context, task, output format, tone, constraints, and few-shot examples. A live quality score and suggestions update as you fill in fields, so you see the structural gap before you copy anything. It runs entirely in the browser with no account, no server call for the core wizard, and no data leaving your machine. The optional AI refinement step requires you to supply your own API key in settings — the tool does not proxy model calls on your behalf. The ceiling arrives fast: there is no programmatic output, no saved workspace with version history, and no way to chain or batch prompts for repeated workflows.
FreeOpen SourceSelf-hostedVerified Aug 14, 2026
20. OpenTrust
OpenTrust runs in the browser, gathers signals about the environment and interaction patterns, and returns a trust score your server can act on. It fits teams that want a first-pass human-presence check layered on top of existing auth — not a replacement for it. The SDK is open-source and self-hostable, so the signal pipeline stays off third-party servers. The ceiling appears when you need decisioning logic, model retraining on your own fraud data, or server-side verification depth — none of that ships in the box. Teams with those needs wire OpenTrust as one input into a broader risk engine they build and maintain separately.
FreeOpen SourceAPISelf-hostedVerified Jul 24, 2026
21. ASL V6
ASL V6 combines AST-based static analysis with Docker-isolated runtime verification to find and confirm exploitable vulnerabilities in AI agent frameworks before they ship. The dual-layer approach means a finding isn't just flagged — it's verified in a sandboxed execution environment, which cuts the false-positive rate that burns security team time. It runs entirely offline with no external API calls, so sensitive proprietary code never leaves your machine. The ceiling appears quickly on non-Python codebases and on teams that need ticketing integrations or cloud-native CI pipelines baked in rather than assembled by hand.
FreeOpen SourceSelf-hostedVerified Jul 28, 2026
22. bitsandbytes
The platform centralizes model hosting, dataset management, application deployment, and multi-provider inference under one roof, with over two million public models available and a unified API covering 45,000+ models from external providers at no added service fee. Public collaboration is free and uncapped; the organizational controls that enterprise teams actually need — SSO, audit logs, private dataset viewers, regional data residency — are paid-only features. GPU compute bills by the hour, which keeps costs predictable for sporadic workloads but adds up fast for teams running persistent endpoints. Self-hosting the Hub is an option, so data never has to leave your infrastructure.
PaidOpen SourceStarting at $20/user/month; $0.60/hour GPUAPISelf-hostedVerified Jul 13, 2026
23. TokenDam
The tool runs entirely in the browser — nothing you paste leaves your machine — which means no SDK swap, no gateway, no routing prod traffic through a third party. Drop a trace file (OpenAI, Anthropic, LangSmith, Langfuse, Vercel AI SDK, or raw logs), and the report identifies duplicate context, low-value prompt content, and caching misses in roughly the time it takes to read this sentence. The CI integration is the sharper edge: `tokendam --ci` fails the build when token cost regresses, turning a vague cost concern into a hard pipeline gate. The tool does not monitor live traffic — it audits what you've already captured. Teams that need runtime dashboards or routing failover will reach for a gateway alongside it.
PaidSelf-hostedVerified Aug 16, 2026
24. Value System Kernel
The repo describes a blueprint for accelerator-native guardrail logic: input vectors are geometrically verified against pre-loaded danger references using IEEE 754 bit-masking and native FMA intrinsics, with the explicit goal of eliminating branch misprediction and warp divergence that stall GPU pipelines. V1 and V2 architectures are both present, with V2 repositioning the scan space as a multi-dimensional physical memory address structure rather than a semantic parsing layer. The vendor states this is a concept-proof blueprint, not a production-ready drop-in — teams expecting a plug-and-play safety layer will need to adapt the kernel to their inference stack. Community activity is minimal: zero forks, one star at the time of listing.
FreeOpen SourceSelf-hostedVerified Jul 13, 2026
25. PassControl
PassControl sits between your agents and your LLM provider, giving each agent its own scoped credential while keeping the actual provider key server-side only. Before a call reaches OpenAI or any other provider, PassControl checks the agent's identity, its kill state, the model scope it's allowed, and whether its budget is still available — then returns a signed receipt so you have a verifiable record. Suspend one agent and the rest of the fleet keeps running. The product is in private beta for the hosted cloud path, which means production teams with strict uptime requirements should treat it as early-stage. Self-hosting via npm is available, but the vendor is explicit that the system has not yet been independently audited.
PaidAPISelf-hostedVerified Aug 16, 2026
26. ButterClaw
ButterClaw runs as a self-hosted security monitor for AI agent deployments, watching post-authorization behavior for prompt injection attempts, credential exfiltration, and policy violations in real time. When it detects a breach condition, it can automatically revoke OAuth tokens and shred local credentials without waiting for a human to intervene. All of this runs locally — no cloud telemetry, no outbound data — which matters when the agents themselves are handling sensitive credentials. The Apache 2.0 license and Docker deployment model mean you own the stack entirely. The trade-off: without a vendor-hosted control plane, your team carries the operational weight of tuning and maintaining policies.
FreeOpen SourceAPISelf-hostedVerified Aug 14, 2026
27. J-lens Qwen
jlens-qwen36 fits a Jacobian lens to a local Qwen3.6-27B (4-bit) model running on Apple Silicon via MLX, then renders a layer-by-layer visualization of which tokens the model is pushing toward at each position during generation. The canonical demo is a blackmail email prompt: the model outputs a calm, compliant reply, but the workspace band exposes what the intermediate layers were actually predicting — a gap that standard logit inspection never surfaces. This is a single-model, single-hardware tool. It runs on macOS with MLX; there is no cloud backend, no REST API, and no adapter for any other model family. Researchers without Apple Silicon and Qwen3.6-27B (4-bit) specifically cannot run it at all.
FreeOpen SourceSelf-hostedVerified Jul 14, 2026
28. ModelFuzz
The library ships two halves: a red-team scanner that fires deceptive prompt-injection payloads at any OpenAI-compatible endpoint so you can see which attacks actually trigger a tool call, and a decorator that wraps individual tools and checks every argument against your policies before the function executes. The decorator approach means enforcement lives in your code, not in a separate proxy or prompt. The policy engine works on argument content — keyword matching and pattern rules the docs describe — which catches known-bad patterns well but leaves gaps for novel exfiltration routes that do not match existing rules. A hosted dashboard with centralized policies and audit logs is on a waitlist and not yet available, so teams running multiple agents coordinate policy changes manually across codebases.
FreeOpen SourceSelf-hostedVerified Jul 28, 2026
29. Mwe-MCP
The store lives on your own server as human-readable Markdown files, which means you can open a file, spot a hallucinated fact, and correct it without touching any agent code. Per-fragment access controls let you scope which agents or users can read or write each memory entry — so a household assistant and a work agent can coexist without leaking context across boundaries. The docs describe an overnight self-organizing pass that restructures the wiki without agent intervention. Where this breaks: teams expecting a managed cloud endpoint will find none — the vendor states AGPL self-hosted only. Standing up and maintaining the server is your problem.
FreeOpen SourceAPISelf-hostedVerified Jul 25, 2026
30. QuantmLayer
QuantmLayer installs as a single static binary and builds a least-privilege cell from Linux kernel primitives — six enforced walls covering the filesystem, network egress, exec allowlisting, and audit. Credentials don't appear inside the cell as 'permission denied' — they don't exist at all. Egress is default-deny; nothing phones home unless a profile explicitly allows it. Every run commits its governing policy to a hash chain, so you can prove after the fact what the agent was and wasn't allowed to do. The MCP gateway adds a JSON-RPC-layer check so tool-server calls are validated against the server's own schema before they ever execute.
FreeOpen SourceSelf-hostedVerified Jul 22, 2026
31. Rampart
Rampart runs a two-layer pipeline entirely in the browser: a 14.7 MB ONNX token-classification model from Hugging Face combined with a deterministic recognizer layer that catches what the model misses. Nothing leaves the client unredacted — the architecture makes server-side PII exposure structurally impossible, not just policy-dependent. The npm package ships as a complete, reproducible artifact, so your redaction behavior is auditable and consistent across builds. The ceiling arrives when your entity types fall outside what the bundled model was trained to recognize — at that point you are retraining or replacing the model, not tweaking a config. Teams needing real-time redaction across high-volume server-side pipelines will hit the browser-only constraint immediately.
FreeOpen SourceSelf-hostedVerified Jul 11, 2026
32. Stonefold
Stonefold intercepts each action an agent tries to take, checks it against rules your team wrote and approved, routes sensitive ones to a named person for sign-off, and writes every attempt to an audit record — regardless of outcome. The vendor's own test is the clearest argument for it: a cheap model fell for a planted fraud email twelve out of twelve times, while an expensive model caught it every time, but with Stonefold's bank-detail rule in place, neither model let money reach the attacker. The checkpoint lives outside the model, so swapping models doesn't change your exposure. The project is at specification v0.4 and explicitly described as a working prototype — not a hardened production system.
FreeOpen SourceAPISelf-hostedVerified Aug 14, 2026
33. Subtext
Subtext is an open-source, self-hosted tool that surfaces a local language model's internal representations — specifically the J-space global workspace identified in Anthropic research — as the model reads and generates, before output tokens appear. You get a browser-based live view and the ability to record, export, and replay sessions for later analysis. The stack is Python-based and runs against local models, so there is no cloud dependency and no data leaves the machine. The ceiling appears quickly: Subtext has no API, no integration hooks, and no support for models it cannot instrument directly — which means anything running behind a remote endpoint stays opaque.
FreeOpen SourceSelf-hostedVerified Jul 14, 2026
34. Vultrino
Vultrino runs as a self-hosted credential proxy: agents reference an alias like 'github', and Vultrino injects the real token server-side, scrubs it from egress, and enforces a policy layer that decides whether the request is allowed at all. The agent never sees the key. AES-256-GCM encryption with Argon2 key derivation protects credentials at rest, so a stolen storage file doesn't yield anything readable without the master password. WASM plugins extend the model to PGP signing, SSH authentication, and crypto wallet transaction signing — credential types most proxy tools don't touch. The policy engine lets you scope each agent to specific URL patterns and HTTP methods, so a compromised agent role can't pivot beyond what it was granted.
FreeOpen SourceAPISelf-hostedVerified Aug 14, 2026
35. WeAura AI Agent
Aura pulls your Prometheus metrics, Kubernetes state, runbooks, and Git history into a single retrieval layer, then answers incident questions with citations pointing to the exact file and line that informed the response. When a Grafana alert fires, it correlates infrastructure state, classifies severity, deduplicates events, and can open a Jira ticket with a root-cause hypothesis attached — all before you have finished reading the alert body. Every suggested write operation stops for your approval before anything touches production. The self-hosted path runs via Helm with on-prem embeddings, which matters for teams whose data cannot leave their network. Teams with sparse runbook coverage or thin observability instrumentation will get proportionally thin answers.
PaidAPISelf-hostedVerified Jul 11, 2026
36. Argot
Argot runs statistical checks against your repository's own commit history to flag code that looks foreign to the codebase — wrong libraries, reversed import direction, misplaced logic, or tests quietly skipped to cover up a failing change. It checks a diff in roughly 200ms with no network call and no second LLM in the loop. The vendor reports 97.3% of foreign patterns caught against a 0.25% false-positive rate on real edits. The ceiling is detection scope: it surfaces what statistically doesn't fit, but it cannot enforce rules that have never appeared in your history. Repos with sparse or inconsistent histories get weaker signals.
FreeOpen SourceSelf-hostedVerified Aug 14, 2026
37. Legibility Field Kit
The kit is a zero-dependency Python CLI that runs three checks against your AI decision records: does every change name a specific human owner, is time-to-reverse recorded for reversible actions, and does each entry carry a complete OTW receipt — Owner, Time, Witness. Run `audit` against a directory of governance files and it surfaces every defect in seconds. The scoring command turns those findings into a maturity grade across the three tests, giving compliance teams a number to track sprint over sprint. The wall appears fast: the kit reads files, flags gaps, and stops — it does not integrate with your ticketing system, your CI pipeline, or your approval workflow.
PaidOpen SourceSelf-hostedVerified Jul 11, 2026
38. Axtary
Axtary sits beside the agent and evaluates each tool call against deterministic policy before the underlying provider is called. Routine actions — ones that clear the policy rules — pass automatically. Higher-risk actions pause for a human to review the normalized payload, and the approval is cryptographically tied to that specific hash. If anything changes after sign-off, the adapter catches the mismatch and blocks execution. Every attempt, pass, and rejection lands in a verifiable ledger. The self-hosted path keeps provider credentials local, which matters for teams that cannot route credentials through a third-party service.
PaidFree Trial · 30 daysSelf-hostedVerified Jul 26, 2026
39. AGEF
The specification defines a content-addressed, Merkle-linked event structure so every decision in an agent session can be hashed, bundled, and checked offline — no live service required. The reference implementation is Akmon (v2.0.0 and later), which handles bundle export, import, and journaling via akmon-journal. AGEF is a format standard, not a deployed platform: there is no SaaS, no API, and no hosted verification service. Teams adopting it are taking on the work of building or integrating bundle-producing substrates into their existing agent infrastructure. At v0.1.1, the spec is pre-stable — conformance profiles and bundle structure are defined, but tooling outside the Akmon reference implementation is essentially absent.
FreeOpen SourceSelf-hostedVerified Jun 1, 2026
40. Agent Governance Toolkit
Policy enforcement, zero-trust identity, execution sandboxing, and reliability engineering for autonomous AI agents.
FreeOpen SourceAPISelf-hostedVerified May 1, 2026
41. Agent Passport
Agent Passport is an open-source authorization layer that sits between your agents and the actions they take. You issue a scoped passport — specific permissions, a spend cap, a time window — then every action checks against it before executing. Denied calls are logged alongside allowed ones, so you have a record either way. Delegation chains shrink permissions as they pass down to sub-agents; a child passport cannot exceed what the parent granted. Revocation cascades instantly across the whole chain, so you're not hunting down API keys across six services.
FreeOpen SourceAPISelf-hostedVerified Jul 1, 2026
42. AgentMeter
AgentMeter runs locally — no cloud sync, no account creation, no vendor dashboard to log into — and parses the tool calls, token counts, and caching splits that CLI agents like Claude Code, Gemini CLI, Codex CLI, and Copilot CLI generate. It surfaces the three-tier cost structure that prompt caching creates (input, cached-input, and output tokens each priced differently), which the raw API bill flattens into noise. The value-multiplier calculation compares API spend against estimated developer time saved, giving you a number to put in front of a manager. The wall appears when you need alerting, real-time budget enforcement, or integration with a team billing system — none of that is here.
FreeOpen SourceSelf-hostedVerified Jun 12, 2026
43. AgentRecall
AgentRecall is a memory layer that gives AI agents persistent context across sessions — so a support agent recalls a customer's past issue, a sales agent remembers where a deal stalled, and a coding assistant doesn't ask you to re-explain your architecture for the third time. The vendor describes a retrieval-and-storage infrastructure that indexes memories and surfaces relevant ones at query time, rather than stuffing the full conversation history into every prompt. The cloud tier caps at 1,000 stored memories, which is adequate for prototyping but a ceiling teams hit in production. Self-hosting under the MIT license removes that ceiling and keeps data inside your own infrastructure — the tradeoff is that you own the ops. API access covers JavaScript and Python environments.
Paid$9/month for Pro (cloud); self-hosted is freeAPISelf-hostedVerified Jun 1, 2026
44. ArXiv Scholar
ArXiv Scholar is an open-source RAG infrastructure that indexes roughly 5,600 curated AI engineering papers from arXiv and exposes them through a streaming API, so agents and developers can query verified literature instead of relying on a model's training memory. The retrieval pipeline runs a 1ms ML-based router that classifies each query as Direct, Decompose, or HyDE before spinning up hybrid dense-plus-sparse search and a cross-encoder re-ranker. Every answer ships with real arXiv paper IDs attached. The hard ceiling is the corpus: 5,600 papers covering RAG, LLMs, agents, training, and inference — nothing outside that domain, and nothing beyond what was ingested through the pipeline as of June 2026. The public endpoint is rate-limited to 5 requests per minute per IP, which breaks any agent loop that needs to fire queries in bursts.
FreeOpen SourceAPISelf-hostedVerified Jun 18, 2026
45. Atlas Inference Engine
The vendor page benchmarks Atlas at 3.1x the decode throughput of vLLM on Nvidia DGX Spark hardware — 111 tok/s average versus 37 tok/s on Qwen3.5-35B, with a cold start measured in two minutes instead of ten. That gap exists because Atlas ships no Python, no PyTorch, and no JIT warm-up: every path from HTTP request to kernel dispatch is compiled. The tradeoff is hardware specificity — hand-tuned CUDA kernels target Blackwell SM120/121, so teams not running DGX Spark get none of the headline numbers. The model matrix covers Qwen, Gemma, Nemotron, Mistral, and MiniMax, but every recipe is written for that hardware profile. Teams running other GPU generations are not the audience.
FreeOpen SourceAPISelf-hostedVerified Jun 9, 2026
46. Beacon
Beacon is an open-source endpoint telemetry layer that runs locally alongside AI agents, capturing prompts, tool calls, file modifications, and approval workflows before any of that activity disappears into the void. It normalizes that telemetry and forwards it to SIEM platforms like Wazuh, Elastic, or Splunk, so security teams can apply the same detection logic they already run against the rest of the fleet. The architecture is self-hosted by design — no data leaves the endpoint unless you route it there yourself. The project is early-stage; the plugin ecosystem covers the major local agent harnesses but gaps exist for less common runtimes. Teams with agents not yet on the supported list write custom collector plugins — which means more surface area to maintain.
FreeOpen SourceSelf-hostedVerified Jun 1, 2026
47. Bitloops
Bitloops runs as a local CLI that builds a semantic model of your codebase and captures AI interactions — prompts, reasoning, decisions — then links them to the Git commits they produced. The vendor describes it as an intelligence layer sitting between your repository and your agents, so Claude Code, Cursor, Codex, or Copilot pull structured context instead of crawling raw source. Everything stays local: no cloud proxy, no data leaving your environment. The constraint enforcement pillar is listed as coming soon, which means teams that need automated rule enforcement on generated code are buying a roadmap item, not a shipping feature. Early-stage tooling with real architectural intent, but the feature set reflects a pre-seed trajectory.
FreeOpen SourceSelf-hostedVerified Jun 1, 2026
48. Cactus
Open-source inference engine for deploying AI models locally on mobile and edge devices with automatic cloud fallback.
PaidFree tier; paid hybrid inference and NPU acceleration featuresAPISelf-hostedVerified Apr 29, 2026
49. Causari
Causari is a content-addressable ledger that sits between your agents and both the LLM API and the filesystem, joining wire-captured prompts to disk-captured edits by matching the lines an agent wrote against the completions it received moments before. The causal fingerprint requires no cooperation from the agent itself — a passive proxy and a filesystem watcher do the work. Distilled sessions become signed, portable skills that agents can query before acting, turning past verified work into ranked recall. The architecture holds well for single-repo, single-team debugging and provenance. Teams running distributed, multi-repo setups or wanting hosted verification will hit the boundary of what the self-hosted binary covers — that functionality sits behind a paid-only Trust Plane.
PaidOpen SourceAPISelf-hostedVerified Jul 8, 2026
50. Cognita
An open-source RAG framework for building and deploying scalable retrieval-augmented generation applications.
FreeAPISelf-hostedVerified Apr 29, 2026
51. ComplyEdge
ComplyEdge is an open-source compliance engine that runs on every production request your AI agent processes, enforcing EU AI Act Article 5 prohibitions and emitting structured audit trails instead of opaque scores. The decorator-based Python SDK wraps agent entry points with a single annotation, so enforcement is tied to the code path rather than bolted on downstream. TrustLint, the companion CLI tool, moves the same rule set into CI/CD so violations surface before deployment. The ceiling appears when you need jurisdictions beyond EU or rule sets beyond Article 5 — the repo shows EU coverage, and teams with broader regulatory scope will find themselves extending the rule library themselves. With three GitHub stars and zero open issues at time of writing, production battle-testing is still accumulating.
PaidOpen SourceAPISelf-hostedVerified Jun 18, 2026
52. Context Mode Insight
Context Mode is built to answer that question honestly. It sits between your AI coding tools and your engineering metrics, correlating actual usage patterns with sprint velocity, incident rates, and individual blockers surfaced through manager 1:1 data. The Remote MCP endpoint lets AI agents call live functions — engagement health checks, blocker detection — so a manager can ask a question in Claude and get a sourced answer instead of a stale report. The platform also generates compliance audit logs formatted for CISO reviews, which keeps security teams out of your sprint. The wall appears when your org is under 50 developers: the signal-to-noise ratio on correlations drops, and the per-seat cost structure stops making sense before the insights do.
Paid$20/seat/monthAPISelf-hostedVerified Jun 8, 2026
53. Core AI Models
The repository ships three concrete layers: Python export recipes for popular Hugging Face models, reusable PyTorch primitives for authoring custom models in Core AI format, and a Swift package that slots those exported models into macOS and iOS apps. The CLI tooling lets you run models directly on a Mac before touching Xcode. Where the workflow breaks is at the edges of what the export recipes cover — models outside the supported Hugging Face roster require you to author your own export logic using the Python primitives, which assumes familiarity with both PyTorch internals and Core AI's model format. The skills directory adds coding-agent plugins, but the core offering is an export-and-runtime pipeline, not an autonomous agent loop.
FreeOpen SourceSelf-hostedVerified Jun 18, 2026
54. debate.tellodb
The core mechanism is fact supersession: when a user moves from NYC to SF, TelloDB marks the old location as stale and filters it from active agent context — so the LLM never hallucinates a two-year-old truth. A hybrid HNSW vector plus BM25 search index handles recall, while a separate Metric Vault layer resolves numeric queries deterministically before they ever reach the LLM. The vendor reports p99 retrieval at 4.2ms and benchmarks recall precision above 95% on LongMemEval-S against 68% for standard RAG. The engine ships as a single Rust binary, self-hostable or deployable on the vendor's platform. At v0.1.0, the surface area is narrow — this is a memory layer, not a full agent runtime.
PaidAPISelf-hostedVerified Jun 14, 2026
55. Deep Memory
The library pairs a GraphRAG implementation with a Vocabulary system: a shared, schema-enforced dictionary of node types, relationship labels, and property constraints that every agent queries before writing. The result is consistent graph data across sessions without prompting every agent with walls of example documents — the schema replaces the examples, trimming token overhead. Backends include Neo4j, SQL Server, Azure Cosmos DB, and an in-memory option, all wired up via Docker Compose quickstarts the docs describe. Where the ceiling appears: there is no hosted service, no GUI, and no API surface — this is a library you embed and operate, which means your team owns the infra from day one.
FreeOpen SourceSelf-hostedVerified Jun 11, 2026
56. Dify
Open-source LLM app development platform combining AI workflow, RAG pipeline, agent capabilities, model management, observability features and more.
Paid$59/moAPISelf-hostedVerified May 1, 2026
57. Dream Server
The installer handles the assembly: LLM inference via Ollama, a chat interface, voice input/output, RAG over private documents, local image generation, and n8n-backed workflow automation land as one unit rather than five separate setup guides. For a homelab or an air-gapped environment where data cannot leave the machine, that single-step setup removes the friction that kills most local AI experiments before they start. The ceiling appears when your workflow logic grows — n8n handles the automation layer, but that means a separate tool you now own and maintain alongside DreamServer itself. Teams building anything production-grade with complex branching or multi-system integrations will find themselves extending past what a local server wrapper can reasonably absorb.
FreeOpen SourceAPISelf-hostedVerified Jun 18, 2026
58. Elysia
An open-source framework that spins up an end-to-end agentic RAG application with just two terminal commands.
FreeAPISelf-hostedVerified May 1, 2026
59. Engram
Engram sits between your IDE and its file reads, maintaining a local SQLite summary of your codebase so agents pull compressed context instead of raw files. The vendor states an 89% measured token reduction. It installs via npm, runs locally with zero cloud dependency, and connects to Claude Code, Cursor, Cline, Continue, Aider, Codex, Windsurf, and Zed through a combination of OpenVSX extensions, an Anthropic plugin, and adapter scripts. The bug-prevention layer surfaces past mistakes from revert history before the agent touches that code path again. This is a passive interceptor, not an agent — it does not plan tasks or run autonomously.
FreeOpen SourceAPISelf-hostedVerified Jun 1, 2026
60. Flightdeck
Every LLM call, MCP event, and tool invocation your agents make streams to a live dashboard — per-agent timelines and a fleet-wide feed, not batched logs you dig through after the incident. The vendor describes token budgets and MCP allow/block rules you set before problems hit, plus the ability to issue live directives to running agents without restarting them. The self-hosted, Apache-2.0 model means no telemetry leaves your infrastructure — critical for teams in regulated environments or those burned by SaaS observability vendors billing by event volume. The project is early-stage by star count, and the operational surface you take on by self-hosting is real.
FreeOpen SourceSelf-hostedVerified Jun 12, 2026
61. GalaxDB
The core bet is that keeping structured rows, dense embeddings, JSON, blobs, and training snapshots in one storage engine eliminates the synchronization failures that happen when each lives somewhere else. You declare an EMBEDDING MODEL in your DDL and every INSERT triggers a local sidecar that computes and indexes the vector — no Airflow, no Lambda, no external API call. Time-travel lets you tag a snapshot before a training run and replay the exact data the model saw months later, which means reproducibility stops being a manual discipline. The ceiling appears at scale: v1.0-beta.1 benchmarks are real but the project is pre-GA, and teams running serious production traffic will be betting on a single vendor with no public track record at that load. If your stack already runs on managed Postgres and a mature vector service, the migration cost has to pencil out against the consolidation savings.
FreeSelf-hostedVerified Jun 18, 2026
62. HarvestGuard
The system fuses live satellite vegetation indices, rainfall anomaly data, and WFP food security indicators, then routes that combined signal through Claude to produce country-level crop failure risk assessments. Docker handles deployment; an Anthropic API key handles the inference. For an NGO standing up a proof-of-concept or a research institution prototyping AI plus Earth observation, the architecture is legible and the cost surface is clear — you pay for API calls, not a platform license. The wall appears when you need operational guarantees: this is a single-maintainer GitHub project with one star, no issue history, and no documented accuracy benchmarks against historical famine events. Teams that need auditable model provenance or SLA-backed uptime will hit that ceiling fast.
FreeOpen SourceAPISelf-hostedVerified Jun 9, 2026
63. Honcho
Every message written to Honcho triggers automatic reasoning via the vendor's Neuromancer model, which learns user psychology and behavioral patterns rather than just indexing text. The `context()` call returns a curated summary plus conversation history shaped to a token budget you set — the vendor claims 60–90% token reduction versus naive retrieval. Multi-participant sessions model each peer separately, so a group conversation doesn't collapse everyone's state into one blob. The ceiling appears when you need reasoning beyond user memory — Honcho does not run tasks, make decisions, or coordinate agents; it only informs them. Teams building full autonomous pipelines still wire Honcho into a separate orchestration layer.
PaidOpen SourceAPISelf-hostedVerified Jun 9, 2026
64. Kontext
Kontext captures the full conversation from ChatGPT or Claude via the page's internal API, runs an on-device summary using Gemini Nano or a user-supplied API key, and places a formatted handoff prompt into the target AI's input field — without sending anything to an external server. The workflow is one-click once installed. The tool is MIT-licensed, open-source, and the vendor states nothing leaves your machine. The ceiling appears fast: there is no multi-turn session management, no persistent storage of prior kontexts, and no support for AI platforms beyond ChatGPT and Claude. Teams running workflows across three or more models, or needing a searchable archive of past sessions, will hit that boundary quickly.
FreeOpen SourceSelf-hostedVerified Jul 5, 2026
65. LanceDB
Open-source embedded vector database for multimodal AI with billion-scale search on Lance columnar format.
PaidAPISelf-hostedVerified Apr 22, 2026
66. Latitude LLM
Latitude is an open-source AI agent monitoring platform that captures full conversation traces, clusters similar failures into triage-ready issue groups, and turns confirmed failure modes into automated evaluations that run against every new trace. The vendor states it ingests via OpenTelemetry, so teams already using OTEL pipelines point their existing setup at Latitude without reformatting data. Semantic search runs across 100% of traces — no sampling — which means finding 'frustrated users on a specific model version after a specific release' takes filters, not queries. The ceiling appears when your team needs the monitoring layer to also drive prompts or chain agents: that is not what this tool does.
PaidOpen SourceFree Trial · 30 days$99/monthAPISelf-hostedVerified Jun 24, 2026
67. LightRAG
The tool indexes documents into both a vector store and a graph of entities and relationships, then queries both at retrieval time — so a question about how two concepts relate pulls connected nodes, not just cosine-similar text. Self-hosting is first-class: the repo ships Dockerfiles, a docker-compose stack, and Kubernetes manifests, so you are not routing data through an external API. The graph construction step is slower than plain vector indexing, and at document-collection scale that latency becomes a real scheduling concern. Community reports on the GitHub issue tracker (195 open issues) suggest the surface area for edge cases is wide, meaning teams moving beyond the examples folder should plan for debugging time. For multimodal or highly structured corpora the graph extraction quality depends heavily on the LLM you point at it.
FreeOpen SourceAPISelf-hostedVerified Jul 2, 2026
68. llama.cpp
llama.cpp is a C/C++ inference engine that runs quantized LLMs entirely on local hardware, from an Apple Silicon laptop to an H100 cluster to a Jetson edge device, using the same binary and the same hand-tuned kernels across all of them. No API keys, no telemetry, no requests leaving the machine. It exposes an OpenAI-compatible server via `llama serve`, which means drop-in compatibility with tooling already pointed at OpenAI endpoints. The ceiling appears when you need the inference engine to do more than infer — there is no planning loop, no tool-calling orchestration, no agent layer built in. Teams building autonomous workflows bolt on a framework on top, which means they are maintaining two systems.
FreeOpen SourceAPISelf-hostedVerified Jun 9, 2026
69. LM Studio
LM Studio, built by Element Labs Inc., is a desktop and server runtime for running open-source LLMs — Qwen, Gemma, DeepSeek, gpt-oss, and others — entirely on local hardware, with no outbound API calls required. The GUI lets you download and chat with models in minutes; the headless CLI tool `llmster` extends the same runtime to Linux servers, cloud VMs, and CI pipelines with no interface overhead. An OpenAI-compatible API layer means existing code talking to OpenAI endpoints can be redirected to a local LM Studio server with minimal changes. The ceiling appears when you need the model to do something at scale: high-throughput production inference, fine-tuning, or multi-tenant serving — none of those are what this tool is built for.
PaidFree (home/work); Business $10–$20/user/month; Enterprise customAPISelf-hostedVerified Jun 9, 2026
70. LMCache
The library plugs into vLLM or TGI backends and stores KV cache tensors so that overlapping prompt prefixes — system prompts, document chunks, conversation history — are served from cache on subsequent requests. The vendor states 8–10x latency improvements for prompt caching workloads and 4–10x for RAG queries where the same document chunks appear across requests. The compression and streaming techniques described in the backing research (CacheGen, CacheBlend) are what make cache delivery fast enough to beat recomputation. The ceiling appears when your workload has little prompt overlap — unique user queries with no shared prefix — at which point the cache layer adds infrastructure without meaningful savings.
FreeOpen SourceSelf-hostedVerified Jun 18, 2026
71. Local RAG memory system
The server stores, retrieves, and versions memories using local ChromaDB, so context survives across sessions without touching any cloud service. You run it via Docker or Python, wire it into your MCP client once, and your assistant can recall preferences, project context, or past decisions on demand. Conflict detection flags when an incoming memory update collides with something already stored, so you are not silently overwriting context. The architecture fits solo developers and privacy-focused workflows well — it was built for exactly that. Where it strains: teams expecting multi-user memory sharing or production-grade scaling will find ChromaDB's local single-process model is not the right foundation.
FreeOpen SourceAPISelf-hostedVerified Jun 18, 2026
72. local-deep-research
The tool autonomously plans and executes multi-step research tasks: it queries sources, follows citations, synthesizes findings, and returns results with full attribution — all without a cloud handoff. The vendor reports ~95% on SimpleQA benchmarks using models like Qwen3-27B on a single RTX 3090, which gives you a concrete hardware target. It pulls from 10+ search backends including arXiv, PubMed, and private document collections. Where it breaks: running capable local models demands real GPU headroom, and teams without that hardware will either throttle to weaker models or route queries to cloud LLMs — at which point the privacy guarantee depends entirely on which cloud endpoint they configure. The 109 open issues and 210 open pull requests on GitHub signal an active but fast-moving codebase; production stability requires version pinning.
FreeOpen SourceAPISelf-hostedVerified Jun 9, 2026
73. LocalAI
LocalAI is a self-hosted, MIT-licensed stack that exposes an OpenAI-compatible REST API from your own hardware. Language model inference, image generation, audio, semantic search via LocalRecall, and autonomous agents via LocalAGI all run without a network call leaving your machine. The modular design pulls backends on demand, so you don't install inference engines you don't use. The wall appears at model selection and hardware sizing: you need at least 10GB of RAM and enough disk for the models you want to run, and the quality ceiling is set by what open-weight models can actually do. Teams needing GPT-4-class reasoning on constrained hardware eventually look elsewhere.
FreeOpen SourceAPISelf-hostedVerified Jun 9, 2026
74. Memori
The vendor states Memori classifies each chat turn into facts, preferences, rules, and summaries, then pulls targeted snippets at recall time rather than re-injecting full history. On the LoCoMo benchmark, the docs report 81.95% accuracy while cutting token usage by 95% versus full-context retrieval — a meaningful number if your cost problem is upstream of the model choice. The memory graph shows how entities connect across sessions, and every recall result ships with lineage explaining why that snippet was included, which matters when an enterprise audit asks why the agent said what it said. The ceiling appears when your retrieval logic needs fine-grained control the SDK's zero-configuration defaults don't expose — teams at that point are writing wrapper logic to compensate. Self-hosted deployment is available, so organizations with data-residency requirements are not locked into the cloud path.
Paid$19/monthAPISelf-hostedVerified Jun 9, 2026
75. MemoryOps
The project implements a full memory lifecycle — typed capture, policy evaluation before storage or retrieval, hybrid retrieval combining vector and structured lookup, deletion guarantees, and per-tenant isolation — as a self-hosted governance layer sitting between your AI assistant and whatever storage backends you run. Policy evaluation runs before memory is written or surfaced, which means you can enforce rules about what gets stored rather than cleaning up after the fact. The audit trail is first-class, not a log file bolted on. Where it breaks: this is a governance layer, not a memory provider with managed infrastructure, so your team owns the deployment, the backend wiring, and the operational burden. Early community adoption is thin — the repo reports five stars at the time of writing — which means when you hit an undocumented edge case, you are reading source code, not finding a Stack Overflow answer.
FreeOpen SourceSelf-hostedVerified Jun 22, 2026
76. MTPLX
The vendor states a 2.24× decode speedup on Qwen3-27B running on an M5 Max MacBook Pro, achieved by using the model's own built-in MTP heads as the drafter — no second model loaded, no external checkpoint to maintain. Acceptance is handled via Leviathan–Chen rejection sampling with a residual (p − q)+ correction, verified bit-exact against single-token autoregressive output. It serves an OpenAI- and Anthropic-compatible API, so downstream tooling like Claude Code, Cline, or the openai-python SDK connects without shims. The wall appears immediately if you leave Apple Silicon: the runtime is explicitly Apple Silicon only, and the custom Metal kernels have no CUDA path.
FreeOpen SourceAPISelf-hostedVerified Jun 9, 2026
77. Ollama
Ollama downloads open-source models like Llama 2 and Mistral and runs them on your own hardware—no API calls, no subscriptions, no data leaving your machine. The pitch is straightforward: you get inference without the per-token pricing or rate limits of cloud services. The catch is real: performance depends entirely on your CPU or GPU, and setup requires comfort with command-line tools and ~10GB of disk space per model. It's genuinely free, but you're trading convenience and speed for privacy and control.
PaidOpen Source$20/moAPISelf-hosted
78. Open-WebUI
Open WebUI is a self-hosted chat interface that connects to local models via Ollama, cloud providers like OpenAI and Anthropic, or any API-compatible endpoint — all from a single install that takes one command and under a minute. Your data stays on your infrastructure. The community layer lets teams browse, install, and share prompts, tools, and Python-based pipeline functions built by 448K other users, so you are not building every capability from scratch. Where it breaks: Open WebUI is a platform, not an agent system — teams that need autonomous multi-step task execution will hit that ceiling fast. Custom logic requires writing Python pipeline functions, which means a developer on the hook whenever the workflow changes.
PaidOpen SourceAPISelf-hostedVerified Jul 6, 2026
79. OpenRAG
OpenRAG is a modular framework for exploring Retrieval-Augmented Generation (RAG) techniques, built for transparency and rapid experimentation to develop document-grounded AI systems—fully ready for production-scale deployment. It uses Ray to parallelize chunking, embedding, and ingestion across CPUs and GPUs, enabling fast, scalable processing of large document sets, and can be deployed seamlessly on Kubernetes for distributed, production-grade workloads. Advanced loaders like Docling and Marker parse complex layouts with OCR-enhanced PDFs, and chunk contextualization significantly boosts retrieval relevance. The platform ships with fully OpenAI-compatible chat API for seamless integration with tools like LangChain, OpenWebUI, or N8N—no adapter work required. Built-in clustering auto-generates synthetic QA datasets from your indexed documents, and a local LLM scores each query-chunk pair to help you tune retrieval before production. Two friction points surface at scale: in collaborative systems where documents update hourly, embeddings are recomputed every time by vLLM, which is computationally expensive, and admin users cannot grant access to partitions they were not explicitly given access to—the admin role does not override partition-level access restrictions.
FreeAPISelf-hostedVerified May 7, 2026
80. OpenVINO™ Toolkit
Open-source toolkit for optimizing and deploying AI inference on Intel and multi-platform hardware.
FreeAPISelf-hostedVerified Apr 21, 2026
81. Orchid
Orchid sits between your agent and any API it talks to, capturing traffic into a local SQLite file — no cloud account, no SDK changes, no telemetry leaving your machine. The built-in web UI lets you step through a completed run, inspect every prompt, response, token count, and cost. The proxy also runs a built-in MCP server, so your IDE assistant in Cursor, VS Code, or Claude Code can query recorded traffic directly. Replay is deterministic and costs nothing in API fees. The ceiling appears when your team needs cross-service aggregation or production alerting — this tool is a local debugger, not an observability platform.
FreeOpen SourceSelf-hostedVerified Jun 25, 2026
82. OrgForge
OrgForge generates a deterministic, ground-truth corporate ecosystem: Confluence pages, JIRA tickets, Slack threads, Git PRs, Zoom transcripts, Zendesk tickets, Salesforce records, emails, and server telemetry — all parameterized to a target company shape or industry. Because the simulation is deterministic, the same seed produces the same dataset, so evaluation results are reproducible across runs. The ceiling appears when your evaluation scenario requires nuance from a specific real org's culture or data patterns — synthetic artifacts will not match those edge cases. Teams using OrgForge for RAG benchmarking get a controlled baseline; teams needing production-representative data for a specific enterprise client still have to build a separate data-collection pipeline.
FreeOpen SourceSelf-hostedVerified Jun 13, 2026
83. PandaProbe Cloud
The core loop is trace, eval, monitor: capture every span across a session, run research-grounded scoring against those traces, then schedule that scoring on a cron so regressions surface before users do. One-line instrumentation covers LangGraph, CrewAI, Google ADK, OpenAI Agents SDK, and others — so you are not writing custom middleware to get signal. The session-level evaluation is the differentiator; most observability tooling scores individual calls, not the drift that accumulates across a 40-step agent trajectory. Self-hosted deployment is available under Apache 2.0, which matters for teams whose data cannot leave their infrastructure. The free tier caps trace ingestion and session eval runs at counts that support experimentation but not sustained production load.
Paid$29/monthAPISelf-hostedVerified Jun 18, 2026
84. PII GUI
The app runs detection locally using on-device models, so nothing is uploaded at any point — no sign-up, no server round-trip, no cloud dependency. You review every flagged item in context before committing to a redaction, which means you catch the false positives before they become permanent holes in a legal document. Custom regex lets you add patterns the model won't know: internal case IDs, account number formats, bespoke identifiers. The export produces a PDF with sensitive text actually gone, not layered over. Where it breaks: single-file, single-session workflow with no batch processing described in the docs, so teams processing hundreds of support logs daily will hit a throughput ceiling fast.
FreeOpen SourceSelf-hostedVerified Jun 21, 2026
85. Pinokio
Pinokio is an open-source desktop launcher that wraps open-source AI tools — image generators, audio DAWs, TTS engines, video models — in one-click install scripts, so users never touch pip, conda, or a shell. The app store model means community-packaged scripts handle environment setup, GPU detection, and model downloads automatically. It runs on Windows, macOS, and Linux, with GPU support across NVIDIA, AMD, and Apple Silicon. The ceiling appears when you need to chain tools together in a real pipeline: Pinokio launches apps, it does not connect them. Teams that outgrow isolated launchers and need data passing between models end up writing the glue code themselves.
FreeOpen SourceSelf-hostedVerified Jul 3, 2026
86. PixelRAG
PixelRAG is an open-source retrieval framework that indexes document pages as images and searches over them using vision-language models, so structure that defeats text extraction — column layouts, embedded charts, dense tables — stays intact through the retrieval step. The hosted API requires no key and the pip-installable package supports self-hosted deployments, which means teams can run it locally without routing data through external services. Where it fits cleanly: Wikipedia-scale visual QA and any RAG pipeline where the page's visual structure carries meaning the text alone loses. Where it breaks: the screenshot-per-page approach trades token efficiency gains on visual content against higher compute per retrieved chunk, and the evidence base for how it performs past Wikipedia-scale collections is thin. Teams pushing beyond the documented use cases are largely on their own.
FreeOpen SourceAPISelf-hostedVerified Jun 28, 2026
87. PreFlight
PreFlight installs via npm and runs as a pre-commit gate, scanning AI-generated code for security vulnerabilities in auth flows, database logic, and SQL patterns — then offering deterministic or AI-assisted patches inline. It integrates with VS Code, Cursor, and MCP clients, so the scan happens in the environment where the AI code was written. The free tier caps patches at ten, which is sufficient for evaluation but stops short of daily use on an active codebase. Teams that exceed that ceiling without a pro key lose the fix-application step and are left with scan output only. The repo is open-source and self-hosted, so the scan never phones home.
PaidOpen Source$19/moSelf-hostedVerified Jun 23, 2026
88. Promptctl
promptctl is a CLI tool that brings Git-style versioning to LLM prompts: commit a prompt file, get a numbered version; diff two versions to see the exact text change; rollback to a previous version, which writes the revert as a new version rather than destroying history. The workflow maps directly to what engineers already do with code — commit, diff, rollback — so there is no new mental model to learn. The ceiling appears quickly: there is no hosted storage, no team sync, no API, and no integration with evaluation frameworks. Teams that outgrow local version history and need shared prompt state or automated regression testing will need to wire something else alongside it.
FreeOpen SourceSelf-hostedVerified Jun 25, 2026
89. PromptLayer
PromptLayer sits between your application and the LLM API, logging every request, tagging it to a prompt version, and giving engineers and non-technical collaborators a shared interface to iterate without touching code. The audit trail and A/B testing pipeline solve the 'who changed what and when' problem that kills rapid iteration on teams larger than two. The self-hosted deployment option exists for teams with data residency requirements. Where it hits a ceiling: the scraped page data available for this listing does not reflect PromptLayer's documented product — factual claims about specific integrations, provider support, or evaluation workflows cannot be sourced from the content retrieved.
FreeAPISelf-hostedVerified Jun 3, 2026
90. PromptShark
PromptShark is a local MITM proxy written in Go and C++ that sits between your agent and any OpenAI-compatible API endpoint. Every request and response pair is captured, logged, and surfaced in a real-time dashboard — no changes to your agent code, just a single base_url swap. The loop detector flags infinite tool-calling cycles automatically. The time-travel replay feature lets you re-run or edit any historical API step without firing a live request, which means no extra token spend during debugging. The self-hosted, MIT-licensed architecture means your prompts and API keys never leave your own infrastructure.
FreeOpen SourceAPISelf-hostedVerified Jun 18, 2026
91. RAGFlow
Open-source RAG engine with deep document understanding, hybrid search, and agentic workflow orchestration.
PaidOpen Source$29/moAPISelf-hostedVerified May 15, 2026
92. RiskKernel
Deployed as a single Go binary, it sits in front of your existing OpenAI, Anthropic, or LangChain stack via a one-variable proxy — no rewrite required. Every call is metered and checkpointed, so a killed or crashed run resumes from the last saved state instead of re-spending from zero. The human-approval gate routes irreversible tool calls for sign-off over CLI, web, or webhook before they fire, and the LLM cannot bypass it because the gate lives in compiled code, not a prompt. The hosted dashboard is private beta only; teams that need a UI today are self-managing.
FreeOpen SourceAPISelf-hostedVerified Jun 12, 2026
93. role-model
role-model is an open protocol and reference router runtime that routes LLM requests across endpoints using declared capability profiles, routing policy, and measured performance — then emits a structured decision artifact you can inspect after the fact. The router narrows candidates by role and task metadata, rejects endpoints that fail capability, locality, or budget checks, and scores what's left against latency, cost, and reliability evidence. The decision is deterministic enough to audit, not a hidden runtime guess. The baseline role set covers chat, code editing, code review, tool calling, and embeddings — so it fits mixed-workload deployments out of the box. Teams needing autonomous multi-step planning or tool loops will find this deliberately out of scope.
FreeOpen SourceAPISelf-hostedVerified Jun 30, 2026
94. Rootsign
RootSign is an open-source Python library that attaches tamper-evident provenance logging to AI agent actions — tool calls, API hits, database writes — capturing a verifiable record of what happened, in what order, and under whose authorization. The vendor describes it as the agent capture layer of a broader Agent Accountability Platform. It installs via pip and ships a Docker Compose quickstart for self-hosting, so the audit trail stays inside your infrastructure. The library integrates with LangGraph and CrewAI by wrapping agent actions at the point of execution. At low log volume the architecture holds; teams with high-throughput agents running thousands of tool calls per hour will hit questions the current documentation does not answer about storage scaling and query performance.
FreeOpen SourceSelf-hostedVerified Jun 20, 2026
95. Selvedge
Selvedge is a local MCP server that AI coding agents (Claude Code, Cursor, Copilot) call as they work, logging the reasoning behind every change into a SQLite file that lives next to your code under .selvedge/. Queries are entity-scoped — you ask about users.email or deps/stripe, not line numbers — so the answer surfaces in the same terms you search in. The vendor describes zero telemetry, no accounts, and no external servers; everything stays on disk. The wall appears when your team needs cross-repo provenance or wants to pipe this data into an existing observability stack — Selvedge emits records but does not integrate with those systems out of the box.
FreeOpen SourceSelf-hostedVerified Jun 1, 2026
96. SigmaShake
SigmaShake intercepts tool calls from agents running in Claude Code, Cursor, VS Code Copilot, and Gemini CLI, evaluating each action against a rule set before it executes. The vendor states decisions resolve in roughly 85 ms using deterministic native evaluation — no model inference, no GPU, no token spend. Rules follow an Allow/Ask/Deny pattern, where Ask routes the action to a human approval queue rather than blunting everything with a hard block. The desktop app installs in about 30 seconds with no admin rights; the CLI drops into any shell or CI hook chain. Self-hosting is supported, which means the guardrail layer stays offline and never sends your code or commands to a third-party model.
Paid$5/moSelf-hostedVerified Jun 18, 2026
97. Skillier.ai
Skillier sits between you and your AI client, detecting what domain you're working in and loading the relevant skill — finance modeling, legal reasoning, DevOps runbooks — into the context without you leaving the interface. The Lite version is MIT-licensed and runs offline, which matters for air-gapped environments where cloud-dependent tooling is a non-starter. The routing model hands control back through an AskUserQuestion prompt, so you confirm the skill selection rather than having it decided for you. That model works cleanly for single-domain sessions. Blended workflows — writing copy while checking financial assumptions, for instance — require you to manually re-route between skills, and the seams show.
PaidSelf-hostedVerified Jun 2, 2026
98. Spanlens
Spanlens sits in front of your LLM provider via a single baseURL change, recording every call's cost, latency, tokens, and full request-response body with no SDK rewrite required. Agent runs surface as waterfall span trees so you can identify the one step consuming 80% of wall-clock time. The model recommender flags GPT-4o calls that look like classification tasks and shows the cost delta if you swap — with numbers from your own traffic, not benchmarks. The eval and experiment layer lets you replay a fixed dataset across prompt versions before you ship, so quality regressions don't surprise you in production. PII scanning and anomaly detection run at log time, which matters when sensitive data crosses the wire at 3 a.m. with nobody watching.
PaidOpen Source$29/moAPISelf-hostedVerified Jun 9, 2026
99. SynapCores
The engine handles graph traversal, HNSW vector similarity, and in-database LLM inference inside a single MATCH statement, so the four-to-five round-trips that Pinecone plus Postgres plus an external reranker produce become one. The Community Edition ships with 161 ready-to-run recipes covering GraphRAG, fraud detection, document ingestion, and AutoML — each a runnable markdown file you can modify locally. The ceiling arrives at the infrastructure layer: multi-node clustering, Raft replication, and CDC ingest from MySQL or Postgres binlogs are paid-only features. Teams that outgrow a single host hit that wall before they hit a query performance problem. For single-host deployments, the binary wire protocol and B-tree indexes the vendor targets in a future release are not yet available.
PaidFree (Community Edition); Enterprise custom pricingAPISelf-hostedVerified Jun 9, 2026
100. Tenure
Where most memory systems rely on similarity search with soft boundaries, Tenure enforces hard scope isolation at the structural level: engineering beliefs stay in engineering sessions, Project A never bleeds into Project B. The vendor's benchmark claims a drift score of 0.00 against competing memory systems that score above 0.80. Retrieval latency is documented at 15ms with 1.0 precision. The self-hosted Helm install takes roughly 30 seconds and exposes an OpenAI-compatible endpoint, so existing clients require no code changes. The ceiling appears when your team needs managed infrastructure or enterprise support — neither is documented on the vendor site.
PaidAPISelf-hostedVerified Jun 11, 2026
101. Thunderbolt
Open-source, self-hosted enterprise AI client emphasizing data sovereignty and model choice.
PaidAPISelf-hostedVerified Apr 22, 2026
102. Trajeckt
The tool sits as a runtime gateway between your agent and its tools, evaluating the sequence of calls rather than any individual call in isolation. The vendor states enforcement runs deterministically in approximately 1.6ms, outside the model itself, so policy decisions cannot be overridden by prompt injection. It is open-source under Apache-2.0 and self-hostable via Docker or Cargo. Where it fits well is MCP-compatible setups that need sequence-level rules without building a custom policy engine. Where it breaks is anywhere you need probabilistic or context-aware rules — this is deterministic enforcement, and the constraints you do not define ahead of time do not get caught.
FreeOpen SourceAPISelf-hostedVerified Jul 1, 2026
103. vLLM
vLLM's core mechanism is PagedAttention, which the docs describe as a paged memory management approach for the KV cache — the part of GPU memory that normally fragments and wastes capacity at scale. Continuous batching sits on top of that, keeping the GPU fed instead of waiting for a fixed batch to fill. The result, per vendor benchmarks at perf.vllm.ai, is significantly higher throughput per GPU than naive serving setups. It exposes an OpenAI-compatible REST API, so existing client code needs no rewrite. The ceiling arrives when you need multi-node tensor parallelism beyond what your hardware topology supports, or when you're serving models on non-NVIDIA silicon — AMD ROCm and CPU paths exist, but community reports suggest NVIDIA CUDA gets the fastest fixes and the deepest optimization.
FreeOpen SourceAPISelf-hostedVerified Jun 9, 2026
104. Voker
Voker is a passive observability platform for conversational AI agents: it ingests chat session data, surfaces frustration patterns and knowledge gaps, and ties agent behavior to downstream metrics like conversion and retention. The self-hosted deployment path means your conversation data stays on your infrastructure — a hard requirement for many enterprise teams that competing SaaS observability tools cannot meet. The platform targets teams running at least 1,000 monthly sessions; below that threshold the pattern-detection signal is thin and the tooling is underutilized. Non-engineering teams can query agent insights without filing a ticket, which removes the bottleneck between product decisions and session data. Note: the scraped page content did not match Voker's product — factual claims here are drawn from the structured tool data provided.
PaidFree Trial · 30 days$80/moAPISelf-hostedVerified Jun 1, 2026
105. Xinference
Open-source library for unified deployment and serving of language, speech, and multimodal models across diverse hardware and infrastructure.
FreeOpen SourceAPISelf-hostedVerified May 6, 2026
Listings on this page are sourced and verified by the AIDiveForge data pipeline. AIDiveForge is editorially independent — inclusion and rank are not for sale. Labeled ads are separate.