How We Test AI Tools

We install each tool ourselves, run it under the same conditions as the tools it competes with, and publish only what we measured. This page explains exactly how, so you can judge how far to trust a number on this site.

The test machine

8-vCPU AMD EPYC 9354P cloud VM, each test limited to 3 cores and 10 GB RAM, no GPU, no internet (isolated Docker network). There is no GPU, so every result is a CPU result. A desktop or laptop with a GPU, or an Apple Silicon Mac, will be much faster in absolute terms. That is why we compare tools to each other on the same machine instead of quoting tokens per second for yours.

Who runs the tests: ForgeBench

ForgeBench is the name we give our automated test bench, not a person. It is a set of scripts that installs each tool, runs the same tests on it and records the results. Emrich Enterprises LLC built it, chooses which tools to test, reads every result before it is published and is responsible for what this site says. When a page says Tested by ForgeBench, that means the numbers came from this harness on the date shown.

Isolation

Every tool runs in its own Docker container limited to 3 CPU cores, 10 GB of memory and 512 processes, with no privileged access.
Test containers sit on an internal network with no internet and no access to anything else on the host. Downloads (installers, model files) happen on a separate network before the test starts.
A logging DNS resolver answers every lookup with “not found” and records the name. That list is what we publish as “outside hosts it tried to reach”.

Local model runtimes (Ollama, LM Studio, llama.cpp and LocalAI)

Same model file. All runtimes load the identical GGUF file (qwen2.5:3b, Q4_K_M, 1.9 GB), linked rather than copied.
Install. The vendor’s official image or install script, timed.
Cold start. Three times from a fresh container with the model file dropped from the operating system’s cache, and three times with it cached. We report the median time from start to the first answer.
Speed. Three fixed prompts (about 50, 440 and 720 tokens), each run three times, 128 new tokens, temperature 0, one request at a time. Every run starts with a different line so no runtime can reuse a cached prompt. We report the median and publish speed only as a ratio between tools measured back to back in the same session, because the machine is a shared cloud VM.
Threads. We pin threads to the 3 cores the container has, and also measure each runtime’s own default once, because several runtimes ignore the container limit and get much slower.
Memory. Peak memory of the model process, not counting the operating system’s file cache.

AI agents (Goose, Hermes Agent and OpenCode)

Every agent uses the same local model through Ollama, with the same context size and thread count. We then run the same tasks again with a frontier model (below).
Three small tasks: create a file with exact text, read a report and write a short summary that names the root cause, and count rows in a CSV with a shell command. Each task runs three times, each time in a fresh container and an empty working folder.
We grade the files left behind, never what the agent says it did. A task passes only if the output file exists and is correct.
A 3-billion-parameter model is small. Agents designed for much larger models can fail here for reasons that have nothing to do with their quality, and we say so on the page when that is the case.

Agents with a frontier model

Same tasks, same grading. The same three tasks, three runs each in a fresh container, graded on the files left behind.
Model. Claude Opus 5.5 through Anthropic’s API, with each agent’s default settings, so the agent is what varies rather than the model. We publish each agent’s token use and the API cost at list price.
Still no internet. The containers stay on the isolated network. Their only way out is a small proxy we run that allows one destination, api.anthropic.com on port 443, and refuses everything else. It passes the encrypted connection through without decrypting it. The hosts it refused are listed on each page.
Checked before every pass. An isolation probe confirms that direct connections from the test network are still blocked everywhere and that the proxy reaches only api.anthropic.com.
The API key is never stored in an image. It is handed to each container only when the test runs, and afterwards we search the logs and work folders for it to confirm it was not written anywhere.

What we do not do

We do not publish a number we did not measure, and we do not mix vendor benchmarks into our results.
We do not test on GPUs or Apple Silicon yet. Tools that need them are marked as not tested.
Sponsorship and affiliate links have no effect on what we test or what we publish.

Tools tested so far

Raw results for every tested tool, including versions, dates and per-run numbers: adf-bench.json.

Pages we have not tested

Tool pages without a We installed and ran it section summarise the tool’s own site and documentation. We do not give them scores, and we are replacing them with tested pages one tool at a time.

Corrections and re-tests

New version out, or a number that looks wrong? Email [email protected]. Vendors can ask for a re-test; it is free and does not change how we report.

Last reviewed

This method was last reviewed on October 16, 2026.