AIDiveForge field notes · Evidence recorded October 15–16, 2026
Ollama capped our requested 65,536-token context at 32,768 for qwen2.5:3b. The effective setting mattered when interpreting the agent results.
The requested setting was not the observed setting
In our qwen2.5:3b agent tests, clients asked Ollama for a context of 65,536 tokens. The server capped it at the model's trained length of 32,768 tokens. The test record therefore describes a 32,768-token effective context, even though the request contained the larger number.
This is a result from the recorded model and runtime version. It is not a statement that every runtime or every model handles a larger context request in the same way. The important habit is to inspect the configuration that actually ran rather than copying a requested value into the benchmark description.
Three quantities to keep separate
| Quantity | Our observation | Why it matters |
|---|---|---|
| Client's requested context | 65,536 tokens | What the configuration asked the server to use |
| Server's effective context | 32,768 tokens | The condition used in interpreting the local runs |
| Agent's fixed prompt | About 1,150 Goose; about 11,500 Hermes tokens | Consumes part of the available input budget |
| Task conversation and tool output | Varied during execution | Also needs space; not the same as fixed prompt |
Why an agent comparison needs this information
The Hermes documentation cited in our test record asked for at least a 64K context and a model much larger than 3B. Our effective local configuration did not satisfy those conditions. Hermes completed none of the nine small-model task attempts, but later completed all nine with the recorded hosted model. The context limitation belongs next to the score.
Goose completed five of nine attempts with the local model; OpenCode completed three of nine in its next-day local run. All three completed the limited nine-attempt set in the hosted-model reruns. These observations support investigating the complete model-agent setup. They do not identify context length as the sole cause of the local failures.
Check effective settings before a long test
A small task may fail because the model chooses the wrong tool even when the input fits. A long task may fail for different reasons. Saving the effective context narrows the list of explanations, but it is not a substitute for inspecting the actual failure and output artifacts.
- Record the exact model identifier, quantization, runtime version and requested context.
- Inspect the runtime's loaded-model state and logs after the model is ready.
- Keep the effective context in the experiment record if it differs from the request.
- Estimate the space consumed by fixed instructions and realistic tool output.
- Repeat the intended task with controlled inputs before extrapolating to a longer workflow.
A reporting standard worth keeping
Configuration files describe intent. Logs and loaded-model state provide evidence of execution. A benchmark that reports both is easier to reproduce and less likely to turn an unsupported setup into a product verdict.
For capacity planning, use the same discipline with CPU allocation and memory. Record the limits that were enforced, not only what the process could see. Our thread-setting and memory guides show how those other boundaries changed the interpretation of this same small-model environment.
Test conditions and sources
8-vCPU AMD EPYC 9354P cloud VM, each test limited to 3 cores and 10 GB RAM, no GPU, no internet (isolated Docker network). Local-model runs blocked external access. Hosted-model reruns explicitly allowed the Anthropic API. The machine was shared; generation ratios apply to their paired sessions and do not predict GPU or other-machine performance.
These are dated observations, not a claim about the latest release or every machine. Recommendations and example procedures are our interpretation of the recorded results.
- Ollama test: requested and effective context
- Hermes test conditions
- Goose task conditions
- Ollama generation API options