Skip to main content
AIDiveForge AIDiveForge

We replayed 54 saved agent outputs: what the scores actually prove

A retained OpenCode answer blamed a “misconfigured security policy”; the supplied incident described open database transactions exhausting a connection pool. The file existed, but it failed the root-cause check. Here is how we replayed that failure and the other saved outputs.

Evidence audit: November 11, 2026. The model runs were recorded on October 15–16, 2026. This audit regraded their saved files with the stored checker. The inference dates, versions and historical scores remain attached to those original runs.

What we could reproduce

We retained six sets of agent trials: Goose, Hermes Agent and OpenCode, each with a local Qwen2.5 3B model and with the hosted model recorded as Claude Opus 5.5. Each set contained three tasks repeated three times. We reconstructed the grading inputs from the saved output files and ran the original checker again.

All 54 replayed pass/fail results matched the stored records. That is evidence that these particular scores are reproducible from the retained files. It is a narrower finding than a new agent benchmark or a guarantee that an agent will complete a different job.

Agent Local Qwen2.5 3B: recorded and replayed Hosted model: recorded and replayed
Goose 5 / 9 9 / 9
Hermes Agent 0 / 9 9 / 9
OpenCode 3 / 9 9 / 9

The model/provider configuration and dates differed between the local and hosted sessions. These observations support checking the model configuration when diagnosing a failed workflow. They do not isolate the model as the only cause of the difference, or establish a general ranking of the three agents.

The actual tasks, and the limits of the checker

  1. Create a file. The agent was asked to create hello.txt containing one specified line. The checker reads the file and compares its stripped content with ADF bench OK. It therefore accepts extra surrounding whitespace even though the prompt asks for exact content.
  2. Summarize a synthetic incident. The prompt asks for a summary of at most three sentences that names the root cause. The original checker looks for “report” and either “transaction” or “connection”, imposes a 900-character limit, and accepts up to four approximate sentences. Its sentence limit is looser than the prompt. All retained summaries that passed this checker were within three sentences in our separate sentence-count check.
  3. Count CSV rows. The prompt asks for a shell command and an output file containing the number of data rows, excluding the header. The checker validates that count.txt contains 37. The resulting file alone does not prove which command or tool the agent used.

We corrected the public task label from “two-sentence summary” to “summary of at most three sentences” to match the stored prompt. We preserved the historical scores and published the original checker, including its limitations, so the measurement can be inspected.

A failure that a successful exit status would miss

The synthetic incident describes a report-worker change that left database transactions open and exhausted the connection pool. One retained OpenCode/local-model answer instead named a “misconfigured security policy”. Another named a “missing dependency”. Both files existed, but both failed the root-cause check.

Goose’s local-model run passed the summary task once in three trials. A second trial left the expected output missing, and the third wrote “To be provided”. Those are different failure modes: missing work, incomplete work and incorrect work should not be collapsed into a single successful process exit.

For a workflow you depend on, define its observable success condition first. Check the produced file, required content and failure recovery. If the task requires a specific tool action, retain and inspect evidence of that action separately from the final answer. A nine-trial result is a useful diagnostic sample, not an estimate of reliability across arbitrary repositories or documents.

Record what the replay proves

Check Result or boundary
Saved file exists Necessary, not sufficient
Incident cause matches the supplied fixture Required for the summary task
Original grading result reproduced 54 of 54 retained pass/fail decisions match
New model inference Not performed by this replay
Tool-use requirement Inspect a tool trace separately; file content alone cannot prove it

Replay the evidence without a model or account

The reproduction bundle contains the synthetic incident and CSV fixture generator, the original checker, the 54 saved outputs, source-result hashes and a standard-library verifier. Review the Python files, extract the archive and run:

python verify.py

The verifier checks its input hashes, reconstructs temporary grading directories and compares every replayed result with its recorded result. It also checks ten startup medians used in our separate cache-state audit. The bundle uses local files and requires no API key, model download or network request.

Download the reproduction bundle · Inspect the evidence JSON · ForgeBench methodology · Goose and Hermes comparison

Prepared by AIDiveForge from retained automated test artifacts. The task incident and CSV are synthetic. The public bundle omits service credentials, network proxy details and unrelated logs. Hashes identify the source snapshots; they are not an independent certification of our test environment.

Editorial update: November 15, 2026. Original experiment and audit dates are retained above.

Share this field note