Skip to main content
AIDiveForge AIDiveForge

Cold and warm cache: auditing ten startup medians

Four archived sessions below show eight cold/warm medians. The linked reproduction bundle checks ten medians in total, including an earlier Ollama pair.

Evidence audit: November 11, 2026. We recomputed these medians from the three individual trials retained for each condition in our October 15–16 runtime experiments. This is an audit of the saved measurements, with their original dates and conditions.

Compare the two conditions within each row

“Cold” in these records means a fresh runtime process with the model file evicted from the filesystem cache. “Warm cache” means a fresh process with the model file still in the filesystem cache. Neither condition means a second request to an already loaded model. Each result below is the median of three trials.

Runtime and archived session What was timed Cold median Warm-cache median
Ollama, Oct 16 at 08:43 UTC Wall time of the first generation request 7.800 s 5.014 s
llama.cpp, Oct 16 at 08:33 UTC Container start to first answer 11.826 s 6.180 s
llama.cpp, Oct 16 at 10:41 UTC Container start to first answer 6.598 s 6.217 s
LocalAI, Oct 16 at 10:34 UTC Container start to first answer 8.710 s 8.802 s

The Ollama row and the other runtime rows start their clocks at different points. Reading across the runtime names as if this were one comparable speed ranking would mix request latency with process startup. Our paired comparisons retain their own session and method notes for that reason.

The median leaves important behavior out of sight

Retained cold trials Individual seconds Median
Ollama, Oct 16 08:43 UTC 13.770; 7.343; 7.800 7.800 s
llama.cpp, Oct 16 08:33 UTC 77.002; 11.826; 7.688 (rounded) 11.826 s

These three-point records expose the slow observations; they do not estimate how often such delays recur.

Ollama’s three cold first-request times in the October 16 session were 13.770, 7.343 and 7.800 seconds. Its warm-cache times were 5.035, 5.014 and 4.974 seconds. The two medians summarize those trials, but they hide the much slower first cold observation.

The earlier llama.cpp session makes the same point more sharply: its cold start-to-answer observations were approximately 77.002, 11.826 and 7.688 seconds. The median was 11.826 seconds. Reporting that median alone would leave a reader unaware of the 77-second observation.

Our machine was a shared CPU-only cloud VM. The tests used resource limits of three cores and 10 GB of RAM and no GPU. Scheduling, filesystem caching and startup work can influence these small samples. The records do not establish how frequently the long observation would recur, or predict performance on a reader’s desktop.

Warm cache was not a universal improvement

LocalAI’s warm-cache median was slightly higher than its cold median in this session: 8.802 versus 8.710 seconds. That small difference is not evidence that a warm filesystem cache makes LocalAI intrinsically slower. It shows why the condition, individual observations and sample size matter more than a broad “warm is always faster” claim.

The two llama.cpp sessions also produced different cold medians on the same shared test machine. Treat each session as its own set of observations. Combining the fastest number from one session with a competitor’s number from another would discard the paired-test constraint we use for runtime comparisons.

A practical startup checklist

  • State whether the model file is cached and whether the runtime process is fresh.
  • Name the event that starts the timer: process launch, a model-load command or an inference request.
  • Keep time to first answer separate from time to finish the whole response.
  • Retain individual trials alongside the median, including slow observations.
  • Compare runtimes within the same planned session and measurement definition. Report unsupported comparisons as unsupported.

For an application that starts a model on demand, process startup is part of the user wait. For a service that keeps the model loaded, subsequent-request behavior is a different experiment. Choose the measurement that matches the intended workflow before using a startup number to make an operational decision.

Inspect and recompute the records

The reproduction bundle includes both the values summarized here and an earlier Ollama cold/warm pair, for ten checked medians in total. Its verifier recomputes the medians from the individual observations and checks them against the stored summaries with a 0.01-second rounding tolerance. It also replays the 54 archived agent outputs.

python verify.py

Download the reproduction bundle · Inspect the underlying records · Ollama and llama.cpp paired comparison · llama.cpp and LocalAI paired comparison · Testing method

AIDiveForge analysis of retained automated measurements. Three observations per condition do not establish a population latency distribution. These startup times are CPU-only observations; they are not generation-speed measurements or GPU benchmark results.

Editorial update: November 15, 2026. Original experiment and audit dates are retained above.

Share this field note