AIDiveForge field notes · Evidence recorded October 15–16, 2026
OpenCode created the requested file in all three small-model attempts, but passed neither of the other tasks. Its hosted-model rerun passed all nine attempts.
A deliberately small acceptance test
We ran OpenCode 1.19.2 with qwen2.5:3b through Ollama on the CPU-only test machine. The tasks were to create a file, summarize a report into a two-sentence file and count CSV data rows with a shell command. Each task was repeated three times. This is a small operational check, not a comprehensive coding benchmark.
OpenCode's small-model run was on October 16, one day after the Goose and Hermes small-model runs. We therefore treat comparisons across those agents as reference observations, not a clean same-day speed contest. The recorded Claude Opus 5.5 reruns for all three agents were on the same day.
The task outcomes
| Task | qwen2.5:3b passes | Hosted-model rerun passes |
|---|---|---|
| Create the specified file | 3 / 3 | 3 / 3 |
| Read a report; save a two-sentence summary | 0 / 3 | 3 / 3 |
| Count CSV data rows using a shell command | 0 / 3 | 3 / 3 |
| Total attempts | 3 / 9 | 9 / 9 |
Why artifact checks matter
A convincing explanation from an agent is not the output your next process consumes. For the file task, inspect the exact path and contents. For a summary, check that the requested facts survived and the output has the requested form. For a CSV count, establish whether the header belongs in the count before running the agent.
The value of this small test is its inspectability. If a later workflow fails, you have a known input and a concrete acceptance criterion to revisit. You can distinguish an unavailable tool, an environmental dependency and a model-generated mistake more effectively than by treating any completed conversation as a pass.
An example you can adapt
The following tiny fixture is a new example procedure, not the exact archived benchmark input. It gives you a CSV with three data rows and one header. Put it in a disposable working directory and ask the agent to count data rows with a shell command, then save only the count in count.txt. Verify the file yourself before accepting the task.
mkdir adf-agent-example
cd adf-agent-example
printf 'name,value\nalpha,10\nbeta,20\ngamma,30\n' > sample.csv
# Ask your agent to create count.txt containing only the data-row count.
# Expected content, ignoring surrounding whitespace: 3Prepare an isolated environment before judging the agent
In the test, OpenCode attempted to reach models.opencode.ai and registry.npmjs.org. Those hosts were blocked and the prepared installation kept working. For the hosted rerun we pre-fetched its Anthropic provider package, which would otherwise be downloaded on first use. The result therefore does not show that an empty installation is ready for offline use.
Record provider packages, the actual model and effective context alongside the agent version. If the workload requires network access, make that an explicit part of the test rather than allowing it to change unnoticed between attempts.
What to conclude
This configuration passed file creation reliably within the three observed attempts and did not pass the other two tasks with the small model. The stronger-model rerun succeeded on this limited set. Neither result is a sufficient reason to trust the agent with an unreviewed repository-wide change.
Start with the artifacts your workflow needs, repeat the test with the intended model and keep failures in the record. That gives you a useful acceptance check without turning nine attempts into a general ranking.
Test conditions and sources
8-vCPU AMD EPYC 9354P cloud VM, each test limited to 3 cores and 10 GB RAM, no GPU, no internet (isolated Docker network). Local-model runs blocked external access. Hosted-model reruns explicitly allowed the Anthropic API. The machine was shared; generation ratios apply to their paired sessions and do not predict GPU or other-machine performance.
These are dated observations, not a claim about the latest release or every machine. Recommendations and example procedures are our interpretation of the recorded results.