{
 "machine": "8-vCPU AMD EPYC 9354P cloud VM, each test limited to 3 cores and 10 GB RAM, no GPU, no internet (isolated Docker network)",
 "speed_caveat": "Times in seconds are from this CPU-only machine; a desktop with a GPU will be much faster. The VM is shared, so text-generation speed is shown only as a ratio between tools run back to back, not as tokens per second.",
 "size_note": "Docker image size is the on-disk size from docker image ls for every tool.",
 "tools": {
  "ollama": {
   "slug": "ollama",
   "kind": "runtime",
   "tested_utc": "2026-10-16T08:43:09Z",
   "source_file": "results/ollama_20261016T084309Z.json",
   "version": "0.40.5",
   "license": {
    "spdx": "MIT",
    "url": "https://github.com/ollama/ollama/blob/main/LICENSE"
   },
   "install": "Official Docker image, one command",
   "installer_s": null,
   "disk_gb": 9.37,
   "disk_label": "Docker image size on disk",
   "disk_source": "docker image ls, ollama/ollama:latest (sha256:5d3a2f25f982), read 2026-10-16",
   "outbound_blocked": [],
   "notes": [
    "Inside a container limited to 3 CPU cores, Ollama still started 8 threads. Across our three runs that made text generation 3.3x to 11.9x slower than with num_thread set to 3, the cores the container has.",
    "Ollama capped the context at the model's trained length (32,768 tokens for qwen2.5:3b) when we asked for 65,536."
   ],
   "cold_start": [
    {
     "label": "Server start to first answer, model file not cached",
     "s": 7.9
    },
    {
     "label": "Model load time reported by Ollama, model file not cached",
     "s": 7.1
    }
   ],
   "steal_pct": 7.2,
   "mem_gb": 2.2,
   "model": "qwen2.5:3b (Q4_K_M GGUF, 1.9 GB)"
  },
  "lmstudio": {
   "slug": "lm-studio",
   "kind": "runtime",
   "tested_utc": "2026-10-15T08:01:48Z",
   "source_file": "results/lmstudio_20261015T080148Z.json",
   "version": "llmster 0.0.25-1",
   "license": {
    "spdx": "proprietary",
    "url": "https://lmstudio.ai/app-terms",
    "note": "LM Studio app/llmster is closed source (free to use under LM Studio terms); the lms CLI repo lmstudio-ai/lms is MIT"
   },
   "install": "Vendor one-line install script (curl | bash)",
   "installer_s": 53.9,
   "disk_gb": 4.68,
   "disk_label": "Docker image size on disk",
   "disk_source": "docker image ls, adfb/lmstudio:latest, read 2026-10-15, at build; image removed since",
   "outbound_blocked": [
    "extensions.lmstudio.ai",
    "lmstudio.ai",
    "versions-prod.lmstudio.ai"
   ],
   "notes": [
    "The app and its headless server (llmster) are closed source and free under LM Studio's terms; the lms command-line tool is MIT.",
    "It tried to reach lmstudio.ai, versions-prod.lmstudio.ai and extensions.lmstudio.ai. We blocked all of them and it still loaded and served the model.",
    "The lms load command took 44.6 s on a cold start, but its llama.cpp engine needed only 8.1 s of that to load the model: about 34 s (33 to 36 s) passed between the load request and the engine starting. Its log does not show what it waited on."
   ],
   "cold_start": [
    {
     "label": "Model load time from its engine's log, model file not cached",
     "s": 8.1
    },
    {
     "label": "lms load command, model file not cached",
     "s": 44.6
    }
   ],
   "steal_pct": 16.1,
   "mem_gb": 2.0,
   "model": "qwen2.5:3b (Q4_K_M GGUF, 1.9 GB)"
  },
  "llamacpp": {
   "slug": "llama-cpp",
   "kind": "runtime",
   "tested_utc": "2026-10-16T08:33:49Z",
   "source_file": "results/llamacpp_20261016T083349Z.json",
   "version": "build 11681 (commit 5e03a1ff4)",
   "license": {
    "spdx": "MIT",
    "url": "https://github.com/ggml-org/llama.cpp/blob/master/LICENSE"
   },
   "install": "Official Docker image (llama.cpp:server), one command",
   "installer_s": null,
   "disk_gb": 1.21,
   "disk_label": "Docker image size on disk",
   "disk_source": "docker image ls, ghcr.io/ggml-org/llama.cpp:server (sha256:e83cbdc66e69), read 2026-10-16",
   "outbound_blocked": [],
   "notes": [
    "Like Ollama, the llama.cpp server started 8 threads inside a 3-core container by default; text generation was 7.6x slower than with --threads 3.",
    "The first cold start after pulling the image took 74 s; the two after it took 9 s and 4 s. We show the median.",
    "It made no DNS lookups during the test (we log every lookup)."
   ],
   "cold_start": [
    {
     "label": "Server start to first answer, model file not cached",
     "s": 8.7
    }
   ],
   "steal_pct": 13.6,
   "mem_gb": 1.6,
   "model": "qwen2.5:3b (Q4_K_M GGUF, 1.9 GB)"
  },
  "localai": {
   "slug": "localai",
   "kind": "runtime",
   "tested_utc": "2026-10-16T10:34:02Z",
   "source_file": "results/localai_20261016T103402Z.json",
   "version": "v4.11.0",
   "license": {
    "spdx": "MIT",
    "url": "https://github.com/mudler/LocalAI/blob/master/LICENSE"
   },
   "install": "Official Docker image, plus its llama-cpp backend",
   "installer_s": null,
   "disk_gb": 1.2,
   "disk_label": "Docker image size on disk",
   "disk_source": "docker image ls, localai/localai:v4.11.0 (sha256:f1bc435659b9), read 2026-10-16",
   "outbound_blocked": [
    "index.localai.io",
    "quay.io",
    "raw.githubusercontent.com"
   ],
   "notes": [
    "LocalAI's Docker image does not include the engine that runs the model. Its llama-cpp backend is a separate download (local-ai backends install llama-cpp, 165 MB on disk, from quay.io), and LocalAI's log warned that it installed it without signature verification. On October 16, 2026 the image pull took about 13 s and the backend install about 4 s on our server's connection.",
    "At every start it tried to load its model gallery from index.localai.io, raw.githubusercontent.com and quay.io. We blocked all three; it logged a warning and kept serving.",
    "By default it started 8 threads inside a 3-core container, and text generation was 6.1x slower than with threads set to 3 in the model's config. The VM's CPU steal was high during that run (29%), so treat the size of that gap as approximate."
   ],
   "cold_start": [
    {
     "label": "Server start to first answer, model file not cached",
     "s": 6.4
    }
   ],
   "steal_pct": 11.5,
   "mem_gb": 1.7,
   "model": "qwen2.5:3b (Q4_K_M GGUF, 1.9 GB)"
  },
  "goose": {
   "slug": "goose",
   "kind": "agent",
   "tested_utc": "2026-10-15T07:05:30Z",
   "source_file": "results/goose_20261015T070530Z.json",
   "version": "1.54.0",
   "license": {
    "spdx": "Apache-2.0",
    "url": "https://github.com/aaif-goose/goose/blob/main/LICENSE"
   },
   "install": "Vendor one-line install script",
   "installer_s": null,
   "disk_gb": 0.71,
   "disk_label": "Docker image size on disk",
   "disk_source": "docker image ls, adfb/goose:v1.54.0 (sha256:e2247236b20c), read 2026-10-16",
   "outbound_blocked": [
    "models.dev"
   ],
   "notes": [
    "With qwen2.5:3b, every failure was a model mistake (reading a .txt file as an image, counting the CSV header row). Goose itself exited cleanly on all 9 runs. With Claude Opus 5.5 it passed all 9.",
    "It tried to reach models.dev on every run, with both models. We blocked it and Goose kept working."
   ],
   "model": "qwen2.5:3b via Ollama",
   "context_tokens": 32768,
   "passed": 5,
   "total": 9,
   "tasks": [
    {
     "label": "Create a file with given text",
     "passed": 3,
     "runs": 3,
     "median_s": 10.5
    },
    {
     "label": "Read a report and write a 2-sentence summary",
     "passed": 1,
     "runs": 3,
     "median_s": 33.1
    },
    {
     "label": "Count the data rows in a CSV with a shell command",
     "passed": 1,
     "runs": 3,
     "median_s": 15.1
    }
   ],
   "fixed_prompt_tokens": 1150,
   "api_model": {
    "model": "claude-opus-5-5",
    "model_name": "Claude Opus 5.5",
    "provider": "Anthropic API",
    "tested_utc": "2026-10-16T09:54:55Z",
    "source_file": "results/goose-opus_20261016T095454Z.json",
    "version": "1.54.0",
    "passed": 9,
    "total": 9,
    "tasks": [
     {
      "label": "Create a file with given text",
      "passed": 3,
      "runs": 3,
      "median_s": 8.1,
      "median_model_calls": 2
     },
     {
      "label": "Read a report and write a 2-sentence summary",
      "passed": 3,
      "runs": 3,
      "median_s": 10.3,
      "median_model_calls": 3
     },
     {
      "label": "Count the data rows in a CSV with a shell command",
      "passed": 3,
      "runs": 3,
      "median_s": 11.6,
      "median_model_calls": 3
     }
    ],
    "model_calls": 26,
    "model_calls_approx": false,
    "tokens_per_call": 2700,
    "tokens_per_task": 7800,
    "cache_read_pct": 88,
    "cost_usd": 0.14,
    "cost_note": "agent-reported tokens at Anthropic's list price, all 9 runs",
    "allowed_host": "api.anthropic.com:443",
    "denied_hosts": [
     "models.dev"
    ],
    "api_key_found_in_logs_files": 0
   }
  },
  "hermes": {
   "slug": "hermes-agent",
   "kind": "agent",
   "tested_utc": "2026-10-15T07:09:29Z",
   "source_file": "results/hermes_20261015T070929Z.json",
   "version": "v0.22.1 (commit 450ad50)",
   "license": {
    "spdx": "MIT",
    "url": "https://github.com/NousResearch/hermes-agent/blob/main/LICENSE"
   },
   "install": "Vendor one-line install script (browser and computer-use extras skipped)",
   "installer_s": 151.5,
   "disk_gb": 4.17,
   "disk_label": "Docker image size on disk",
   "disk_source": "docker image ls, adfb/hermes:450ad5055eb3, read 2026-10-15, at build; image removed since",
   "outbound_blocked": [
    "api.github.com",
    "hermes-agent.nousresearch.com",
    "openrouter.ai",
    "portal.nousresearch.com",
    "raw.githubusercontent.com"
   ],
   "notes": [
    "With a 3B model, Hermes completed none of the 9 tasks: the model called Hermes' tool dispatcher instead of the file and shell tools. With Claude Opus 5.5 driving it, Hermes passed all 9, so the 3B result is about model size, not a verdict on Hermes.",
    "Hermes installs its Anthropic support from PyPI on first use; offline it fails until pre-installed. Our first Claude run failed every task that way, so we pre-installed it in the image with Hermes' own installer and ran again.",
    "In the 3B test Ollama capped the context at 32,768 tokens (we asked for 65,536), below the 64K Hermes' docs ask for. Hermes ran anyway without a warning we could see.",
    "With the local model, every run tried to reach GitHub, OpenRouter and Nous Research hosts (all blocked). With Claude Opus 5.5 it contacted only Anthropic's API."
   ],
   "model": "qwen2.5:3b via Ollama",
   "context_tokens": 32768,
   "passed": 0,
   "total": 9,
   "tasks": [
    {
     "label": "Create a file with given text",
     "passed": 0,
     "runs": 3,
     "median_s": 177.4
    },
    {
     "label": "Read a report and write a 2-sentence summary",
     "passed": 0,
     "runs": 3,
     "median_s": 156.1
    },
    {
     "label": "Count the data rows in a CSV with a shell command",
     "passed": 0,
     "runs": 3,
     "median_s": 100.0
    }
   ],
   "fixed_prompt_tokens": 11500,
   "api_model": {
    "model": "claude-opus-5-5",
    "model_name": "Claude Opus 5.5",
    "provider": "Anthropic API",
    "tested_utc": "2026-10-16T10:08:06Z",
    "source_file": "results/hermes-opus_20261016T100805Z.json",
    "version": "v0.22.1 (commit 450ad50)",
    "passed": 9,
    "total": 9,
    "tasks": [
     {
      "label": "Create a file with given text",
      "passed": 3,
      "runs": 3,
      "median_s": 18.8,
      "median_model_calls": 2
     },
     {
      "label": "Read a report and write a 2-sentence summary",
      "passed": 3,
      "runs": 3,
      "median_s": 24.8,
      "median_model_calls": 3
     },
     {
      "label": "Count the data rows in a CSV with a shell command",
      "passed": 3,
      "runs": 3,
      "median_s": 22.3,
      "median_model_calls": 3
     }
    ],
    "model_calls": 24,
    "model_calls_approx": true,
    "tokens_per_call": 17700,
    "tokens_per_task": 47100,
    "cache_read_pct": 94,
    "cost_usd": 0.27,
    "cost_note": "agent-reported tokens at Anthropic's list price, all 9 runs",
    "allowed_host": "api.anthropic.com:443",
    "denied_hosts": [],
    "api_key_found_in_logs_files": 0
   }
  },
  "opencode": {
   "slug": "opencode",
   "kind": "agent",
   "tested_utc": "2026-10-16T09:33:55Z",
   "source_file": "results/opencode_20261016T093354Z.json",
   "version": "1.19.2",
   "license": {
    "spdx": "MIT",
    "url": "https://github.com/sst/opencode/blob/dev/LICENSE"
   },
   "install": "Vendor one-line install script",
   "installer_s": 3.1,
   "disk_gb": 0.57,
   "disk_label": "Docker image size on disk",
   "disk_source": "docker image ls, adfb/opencode:latest (sha256:db3c6f6ae3f0), read 2026-10-16",
   "outbound_blocked": [
    "models.opencode.ai",
    "registry.npmjs.org"
   ],
   "notes": [
    "Its qwen2.5:3b run was on October 16, a day after Goose's and Hermes Agent's, so read that row of the comparison table as a reference, not a same-day result. All three Claude Opus 5.5 runs were on the same day. With Claude Opus 5.5, OpenCode passed all 9 tasks.",
    "It tried to reach models.opencode.ai and registry.npmjs.org with both models (102 lookups over 9 runs with the local model). We blocked both and OpenCode kept working. For the Claude run we pre-fetched its Anthropic provider package into the image, which OpenCode otherwise downloads on first use.",
    "In the 3B test Ollama capped the context at 32,768 tokens (we asked for 65,536)."
   ],
   "model": "qwen2.5:3b via Ollama",
   "context_tokens": null,
   "passed": 3,
   "total": 9,
   "tasks": [
    {
     "label": "Create a file with given text",
     "passed": 3,
     "runs": 3,
     "median_s": 20.8
    },
    {
     "label": "Read a report and write a 2-sentence summary",
     "passed": 0,
     "runs": 3,
     "median_s": 37.7
    },
    {
     "label": "Count the data rows in a CSV with a shell command",
     "passed": 0,
     "runs": 3,
     "median_s": 26.2
    }
   ],
   "api_model": {
    "model": "claude-opus-5-5",
    "model_name": "Claude Opus 5.5",
    "provider": "Anthropic API",
    "tested_utc": "2026-10-16T10:01:40Z",
    "source_file": "results/opencode-opus_20261016T100140Z.json",
    "version": "1.19.2",
    "passed": 9,
    "total": 9,
    "tasks": [
     {
      "label": "Create a file with given text",
      "passed": 3,
      "runs": 3,
      "median_s": 11.0,
      "median_model_calls": 2
     },
     {
      "label": "Read a report and write a 2-sentence summary",
      "passed": 3,
      "runs": 3,
      "median_s": 16.3,
      "median_model_calls": 3
     },
     {
      "label": "Count the data rows in a CSV with a shell command",
      "passed": 3,
      "runs": 3,
      "median_s": 10.9,
      "median_model_calls": 3
     }
    ],
    "model_calls": 25,
    "model_calls_approx": false,
    "tokens_per_call": 10500,
    "tokens_per_task": 29200,
    "cache_read_pct": 95,
    "cost_usd": 0.19,
    "cost_note": "agent-reported tokens at Anthropic's list price, all 9 runs",
    "allowed_host": "api.anthropic.com:443",
    "denied_hosts": [
     "models.opencode.ai",
     "registry.npmjs.org"
    ],
    "api_key_found_in_logs_files": 0
   }
  },
  "mtplx": {
   "slug": "mtplx",
   "kind": "runtime",
   "tested_utc": "2026-10-15T08:20:58Z",
   "source_file": "results/mtplx_20261015T082058Z.json",
   "version": "2.12.2",
   "license": {
    "spdx": "Apache-2.0",
    "url": "https://github.com/youssofal/MTPLX/blob/main/LICENSE"
   },
   "install": "pip install mtplx",
   "installer_s": null,
   "disk_gb": null,
   "disk_label": "Docker image size on disk",
   "disk_source": null,
   "outbound_blocked": [],
   "notes": [
    "pip install works on Linux and the CLI starts, but generation needs Apple Silicon (MLX/Metal). We have no Apple Silicon test machine yet, so we have no speed numbers for it."
   ],
   "measurable": false,
   "reason": "Vendor states \"Requirements: Apple Silicon (M1 or newer), macOS 14+\" and \"Not a CUDA project\"; inference is MLX/Metal only. This box is Linux x86_64 with no GPU."
  }
 },
 "pairs": [
  {
   "a": "ollama",
   "b": "lm-studio",
   "kind": "runtime",
   "tested_utc": "2026-10-15T08:01:48Z",
   "source_files": [
    "results/ollama_20261015T081212Z.json",
    "results/lmstudio_20261015T080148Z.json"
   ],
   "note": "Same GGUF model file, same 3 cores, measured back to back in one session. Speed is shown for each of our three prompt lengths: Ollama's lead in generation speed shrinks as the prompt gets longer. The two start-up rows are different measurements: Ollama's is the first 1-token answer from a freshly started server (the model loads on that request); LM Studio's is the lms load command, most of which passed before its engine started loading the model.",
   "rows": [
    {
     "label": "Model load time from the runtime's own log, model file not cached (s)",
     "a": 5.9,
     "b": 8.1,
     "lower_better": true
    },
    {
     "label": "Start-up, not the same measurement: Ollama = first answer from a fresh server; LM Studio = lms load command (s)",
     "a": 6.7,
     "b": 44.6,
     "lower_better": true,
     "compare": false
    },
    {
     "label": "Text generation speed, 53-token prompt, relative",
     "a": 1.38,
     "b": 1.0,
     "lower_better": false,
     "unit": "x"
    },
    {
     "label": "Text generation speed, 441-token prompt, relative",
     "a": 1.22,
     "b": 1.0,
     "lower_better": false,
     "unit": "x"
    },
    {
     "label": "Text generation speed, 723-token prompt, relative",
     "a": 1.0,
     "b": 1.0,
     "lower_better": false,
     "unit": "x"
    },
    {
     "label": "Prompt reading speed, 53-token prompt, relative",
     "a": 1.32,
     "b": 1.0,
     "lower_better": false,
     "unit": "x"
    },
    {
     "label": "Prompt reading speed, 441-token prompt, relative",
     "a": 1.77,
     "b": 1.0,
     "lower_better": false,
     "unit": "x"
    },
    {
     "label": "Prompt reading speed, 723-token prompt, relative",
     "a": 1.58,
     "b": 1.0,
     "lower_better": false,
     "unit": "x"
    },
    {
     "label": "Memory used by the model process (GB)",
     "a": 2.2,
     "b": 2.0,
     "lower_better": true
    },
    {
     "label": "Docker image size on disk (GB)",
     "a": 9.37,
     "b": 4.68,
     "lower_better": true
    }
   ]
  },
  {
   "a": "ollama",
   "b": "llama-cpp",
   "kind": "runtime",
   "tested_utc": "2026-10-16T08:33:49Z",
   "source_files": [
    "results/ollama_20261016T084309Z.json",
    "results/llamacpp_20261016T083349Z.json"
   ],
   "note": "Same GGUF model file, same 3 cores, measured back to back in one session. The VM's CPU steal was higher during the llama.cpp run (13.6% vs 7.2%), so differences under about 10% are within noise.",
   "rows": [
    {
     "label": "Server start to first answer, model file not cached (s)",
     "a": 7.9,
     "b": 8.7,
     "lower_better": true
    },
    {
     "label": "Text generation speed, 53-token prompt, relative",
     "a": 1.07,
     "b": 1.0,
     "lower_better": false,
     "unit": "x"
    },
    {
     "label": "Text generation speed, 441-token prompt, relative",
     "a": 1.05,
     "b": 1.0,
     "lower_better": false,
     "unit": "x"
    },
    {
     "label": "Text generation speed, 723-token prompt, relative",
     "a": 1.17,
     "b": 1.0,
     "lower_better": false,
     "unit": "x"
    },
    {
     "label": "Prompt reading speed, 53-token prompt, relative",
     "a": 1.06,
     "b": 1.0,
     "lower_better": false,
     "unit": "x"
    },
    {
     "label": "Prompt reading speed, 441-token prompt, relative",
     "a": 1.11,
     "b": 1.0,
     "lower_better": false,
     "unit": "x"
    },
    {
     "label": "Prompt reading speed, 723-token prompt, relative",
     "a": 1.07,
     "b": 1.0,
     "lower_better": false,
     "unit": "x"
    },
    {
     "label": "Memory used by the model process (GB)",
     "a": 2.2,
     "b": 1.6,
     "lower_better": true
    },
    {
     "label": "Docker image size on disk (GB)",
     "a": 9.37,
     "b": 1.21,
     "lower_better": true
    }
   ]
  },
  {
   "a": "llama-cpp",
   "b": "localai",
   "kind": "runtime",
   "tested_utc": "2026-10-16T10:34:02Z",
   "source_files": [
    "results/llamacpp_20261016T104105Z.json",
    "results/localai_20261016T103402Z.json"
   ],
   "note": "Same GGUF model file, same 3 cores, same 4,096-token context, measured back to back in one session; LocalAI runs the model through its own llama-cpp backend. CPU steal: 12.8% during the llama.cpp run, 11.5% during LocalAI's, so differences under about 10% are within noise. LocalAI's image size does not include that backend, a separate 165 MB download. The llama.cpp numbers here come from a second llama.cpp run that day, not the one in the Ollama comparison.",
   "rows": [
    {
     "label": "Server start to first answer, model file not cached (s)",
     "a": 3.5,
     "b": 6.4,
     "lower_better": true
    },
    {
     "label": "Text generation speed, 53-token prompt, relative",
     "a": 0.98,
     "b": 1.0,
     "lower_better": false,
     "unit": "x"
    },
    {
     "label": "Text generation speed, 441-token prompt, relative",
     "a": 0.96,
     "b": 1.0,
     "lower_better": false,
     "unit": "x"
    },
    {
     "label": "Text generation speed, 723-token prompt, relative",
     "a": 0.99,
     "b": 1.0,
     "lower_better": false,
     "unit": "x"
    },
    {
     "label": "Prompt reading speed, 53-token prompt, relative",
     "a": 1.22,
     "b": 1.0,
     "lower_better": false,
     "unit": "x"
    },
    {
     "label": "Prompt reading speed, 441-token prompt, relative",
     "a": 0.99,
     "b": 1.0,
     "lower_better": false,
     "unit": "x"
    },
    {
     "label": "Prompt reading speed, 723-token prompt, relative",
     "a": 0.94,
     "b": 1.0,
     "lower_better": false,
     "unit": "x"
    },
    {
     "label": "Memory used by the model process (GB)",
     "a": 1.6,
     "b": 1.7,
     "lower_better": true
    },
    {
     "label": "Docker image size on disk (GB; LocalAI's backend is a separate download)",
     "a": 1.21,
     "b": 1.2,
     "lower_better": true,
     "compare": false
    }
   ]
  },
  {
   "a": "goose",
   "b": "hermes-agent",
   "kind": "agent",
   "tested_utc": "2026-10-15T07:05:30Z",
   "source_files": [
    "results/goose_20261015T070530Z.json",
    "results/hermes_20261015T070929Z.json"
   ],
   "note": "Same model (qwen2.5:3b on 3 CPU cores), same three tasks, three runs each, graded on the files left behind. Ollama capped the context at 32,768 tokens for both, below the 64K Hermes' docs ask for, and a 3B model is far below the model size Hermes asks for, so read the task score as 'which agent copes with a small local model', not overall quality.",
   "rows": [
    {
     "label": "Tasks passed with qwen2.5:3b (of 9)",
     "a": 5,
     "b": 0,
     "lower_better": false
    },
    {
     "label": "Fixed prompt sent with every model call (tokens)",
     "a": 1150,
     "b": 11500,
     "lower_better": true
    },
    {
     "label": "Docker image size on disk (GB)",
     "a": 0.71,
     "b": 4.17,
     "lower_better": true
    },
    {
     "label": "Outside hosts it tried to reach",
     "a": 1,
     "b": 5,
     "lower_better": true
    }
   ]
  }
 ],
 "agent_api": {
  "title": "Same three tasks, Claude Opus 5.5 driving each agent",
  "agents": [
   "goose",
   "hermes-agent",
   "opencode"
  ],
  "tested_utc": "2026-10-16T09:54:55Z",
  "source_files": [
   "results/goose-opus_20261016T095454Z.json",
   "results/hermes-opus_20261016T100805Z.json",
   "results/opencode-opus_20261016T100140Z.json"
  ],
  "rows": [
   {
    "label": "Tasks passed with Claude Opus 5.5 (of 9)",
    "values": [
     9,
     9,
     9
    ],
    "lower_better": false
   },
   {
    "label": "Median time: Create a file with given text (s)",
    "values": [
     8.1,
     18.8,
     11.0
    ],
    "lower_better": true
   },
   {
    "label": "Median time: Read a report and write a 2-sentence summary (s)",
    "values": [
     10.3,
     24.8,
     16.3
    ],
    "lower_better": true
   },
   {
    "label": "Median time: Count the data rows in a CSV with a shell command (s)",
    "values": [
     11.6,
     22.3,
     10.9
    ],
    "lower_better": true
   },
   {
    "label": "Model calls, all 9 runs",
    "values": [
     26,
     24,
     25
    ],
    "lower_better": true,
    "compare": false
   },
   {
    "label": "Tokens per task, input + output (approx.)",
    "values": [
     7800,
     47100,
     29200
    ],
    "lower_better": true
   },
   {
    "label": "API cost, all 9 runs (US$, list price)",
    "values": [
     0.14,
     0.27,
     0.19
    ],
    "lower_better": true
   },
   {
    "label": "Outside hosts it tried to reach besides the API (blocked)",
    "values": [
     1,
     0,
     2
    ],
    "lower_better": true
   },
   {
    "label": "For reference: tasks passed with qwen2.5:3b on 3 CPU cores (of 9)",
    "values": [
     5,
     0,
     3
    ],
    "lower_better": false,
    "compare": false
   }
  ],
  "note": "All three ran on October 16, 2026, one after another, with Claude Opus 5.5 (claude-opus-5-5) through Anthropic's API and each agent's default settings: three runs per task, each in a fresh container, graded on the files left behind. The containers had no internet; a proxy let them reach api.anthropic.com and nothing else. Times include the round trips to Anthropic's API, so they show the agent and the API together, not this machine's speed. Tokens come from each agent's own output (Hermes Agent's model-call count is approximate); cost is those tokens at Anthropic's list price. The qwen2.5:3b row is from the earlier local-model runs (Goose and Hermes Agent on October 15, OpenCode on October 16)."
 }
}