AIDiveForge field notes · Evidence recorded October 15–16, 2026
In our three-core container, Ollama started eight threads. Explicitly matching the thread setting to the available cores changed generation time substantially.
The allocation and the default disagreed
We ran Ollama 0.40.5 on an eight-vCPU AMD EPYC cloud VM, but limited each test container to three CPU cores and 10 GB of RAM. The model was qwen2.5:3b, a Q4_K_M GGUF of roughly 1.9 GB. There was no GPU. Ollama still started eight threads inside that container. A process can see a host topology that does not match the CPU time its container is allowed to consume.
Across the three recorded runs, generation with the default setting was 3.3 to 11.9 times slower than with num_thread explicitly set to three. That is a range from this test, not an expected speedup for your computer. Shared-host scheduling and CPU steal can change the size of the effect. The useful finding is the mismatch worth checking, not the largest number in the range.
A comparison you can make on your own server
Keep the model, prompt, prediction limit, context and container quota fixed. Warm the model before timing. Alternate the two thread settings so one configuration does not always benefit from running second. Save the API response rather than timing only the shell process: model loading, prompt evaluation and generation are different parts of the request.
The following example assumes Ollama is already running locally and the model is already available. It submits the same small request with three and eight threads. It is a diagnostic procedure, not a reproduction of every input in our archived benchmark. Do not copy our three-thread setting onto a machine with a different CPU allocation.
for threads in 3 8 3 8 3 8; do
curl --fail --silent http://localhost:11434/api/generate \
-H 'Content-Type: application/json' \
-d "{\"model\":\"qwen2.5:3b\",\"prompt\":\"Explain a CSV header in one sentence.\",\"stream\":false,\"options\":{\"num_thread\":$threads,\"num_ctx\":4096,\"num_predict\":128}}"
printf '\n'
doneRead the timing fields separately
Ollama's response exposes load_duration, prompt_eval_duration and eval_duration, with duration values in nanoseconds. Use eval_count and eval_duration together when comparing generation rates. A request that produced fewer tokens can finish earlier without generating tokens faster. Also inspect the actual running process and container limits; a requested option alone is not proof that the intended configuration was used.
We found similar thread-allocation mismatches in llama.cpp and LocalAI. Their observed gaps came from separate runs, so they should not be used to rank the three runtimes against one another. In the LocalAI run CPU steal was especially high. A paired test with the same inputs is the appropriate next step once you have identified a mismatch.
When changing threads will not answer the problem
A GPU-bound workload, model swapping under memory pressure, a very long prompt or several concurrent requests can have a different bottleneck. Check those conditions before treating this as a universal tuning recipe. The strongest result is a repeatable improvement on your actual workload, with the answers still meeting the same quality requirement.
Our practical recommendation is to record the CPU quota and observed worker count beside every local-model benchmark. It makes a surprising result explainable and prevents a default setting from quietly becoming a hardware comparison.
Test conditions and sources
8-vCPU AMD EPYC 9354P cloud VM, each test limited to 3 cores and 10 GB RAM, no GPU, no internet (isolated Docker network). Local-model runs blocked external access. Hosted-model reruns explicitly allowed the Anthropic API. The machine was shared; generation ratios apply to their paired sessions and do not predict GPU or other-machine performance.
These are dated observations, not a claim about the latest release or every machine. Recommendations and example procedures are our interpretation of the recorded results.