AIDiveForge field notes · Evidence recorded October 15–16, 2026
Goose completed five of nine attempts with our small local model and Hermes completed none. With the recorded hosted model, both completed all nine. The model and context limitations are essential to the result.
Three simple tasks, repeated three times
Our task set asked each agent to create a file with specified text, read a report and write a two-sentence summary, and count the data rows in a CSV with a shell command. Each task had three attempts, making nine attempts per agent. We graded the files left behind rather than trusting a successful exit code or a claim in the conversation.
The local runs used qwen2.5:3b through Ollama on three CPU cores. Goose was version 1.54.0. Hermes was v0.22.1, commit 450ad50. The model's effective context was 32,768 tokens even though the client requested 65,536. This was below the 64K context requested by the Hermes documentation cited in the test record, and the 3B model was far below its recommended model size.
Show both model conditions
| Agent | qwen2.5:3b | Recorded Claude Opus 5.5 rerun |
|---|---|---|
| Goose | 5 / 9 attempts | 9 / 9 attempts |
| Hermes Agent | 0 / 9 attempts | 9 / 9 attempts |
What failed in the small-model runs
Goose created the requested file on all three attempts. It passed the report task once and the CSV task once. The test notes describe model mistakes such as treating a text file as an image or counting the CSV header as a data row. Goose itself exited cleanly in all nine attempts, demonstrating why process completion was not an adequate success criterion.
In the Hermes runs the local model called the agent's dispatcher instead of the intended file and shell tools. It completed none of the nine tasks. That is a useful observation about this model-agent configuration. It does not establish that Hermes is generally incapable of those tasks, as the hosted-model rerun makes clear.
Dependency readiness is another variable
The first hosted Hermes attempt failed because Anthropic support was not installed and the isolated environment could not fetch it. The test team pre-installed it using Hermes' installer and reran the tasks. The nine-of-nine hosted result belongs to the prepared environment. Leaving out that preparation would make the comparison difficult to reproduce.
The local and hosted conditions also differed in permitted network access. External requests were blocked for the local-model tests. The hosted reruns explicitly allowed the Anthropic API. Those reruns help investigate the role of the model, but they are not an experiment that changes only one perfectly isolated variable.
A selection test for your own workflow
Before choosing an agent, define the smallest useful task it must finish with the model you can actually run. Check the artifacts, repeat the task and record the effective context. Then test a stronger model if failures suggest tool selection or instruction-following problems. Keep the agent version and task inputs fixed as far as possible.
These nine attempts cover very small file tasks. They do not assess large-repository coding, security boundaries, long-horizon reliability or every feature of either agent. Our recommendation is to test the complete model-agent configuration against your acceptance criteria rather than treating this limited task count as a universal ranking.
Test conditions and sources
8-vCPU AMD EPYC 9354P cloud VM, each test limited to 3 cores and 10 GB RAM, no GPU, no internet (isolated Docker network). Local-model runs blocked external access. Hosted-model reruns explicitly allowed the Anthropic API. The machine was shared; generation ratios apply to their paired sessions and do not predict GPU or other-machine performance.
These are dated observations, not a claim about the latest release or every machine. Recommendations and example procedures are our interpretation of the recorded results.
- Goose: task results and hosted rerun
- Hermes Agent: task results, dependencies and limitations
- Existing Goose/Hermes comparison