Kitaru

Most teams testing a cheaper model or a degraded retriever have two bad options: re-run in production and pay the cost and the risk, or write synthetic test cases that never quite match reality. Kitaru takes a third path — it wraps the calls and steps your agent SDK already exposes and stores them as replayable checkpoints, so a replay is reading back a faithful record of what actually happened, not reconstructing it from scratch. The core workflow is three lines: instrument your agent with a Kitaru adapter, run it in production, then call `client.executions.replay()` with an execution ID and a single override — a different model, a mocked tool return, a degraded retriever result, or a dropped reranker.

The differentiating mechanic is durable execution scoped to agent runs. Kitaru’s adapters for OpenAI Agents SDK and Claude Agent SDK are thin wrappers around the runner objects you already use — `KitaruRunner` replaces `Runner`, `KitaruClaudeRunner` replaces the direct `query` call. The checkpointing strategy is configurable at the call level. Replay reads those checkpoints back, so the what-if reflects the actual model and tool call sequence from the original run, not a re-execution under different conditions.

Kitaru fits teams running agents in production who need an experimentation layer without a separate staging environment or synthetic fixtures. It is open source under Apache 2.0 and self-hosted, with ZenML Pro available separately as a managed control plane. Where it breaks: if your agent’s correctness depends on state that the adapter never intercepts — database writes, external webhook calls, or SDK internals the wrapper sits outside of — replay diverges from reality and the experiment loses its value. Teams hitting that boundary either instrument more aggressively at the SDK level or accept that some failure modes require production reruns regardless.