AutoYou
Freemium
Users wanting fully private local AI
llama.cpp
FreeOpen SourceAPISelf-Hosted
The open-source engine that many local AI apps are built on. Use it directly as a command-line tool or as an OpenAI-compatible server.
Similar tools
LocalAIFreeLocalAI is an MIT-licensed server that exposes an OpenAI-compatible REST API from your own…
AutoYouFreemiumAutoYou runs a local AI server on your Windows, macOS, or Linux machine, then lets your phone…
LM-Kit OneFreemiumThe vendor describes LM-Kit One as a self-hosted AI application server that exposes OpenAI…
Atlas Inference EngineFreeThe vendor page benchmarks Atlas at 3.1x the decode throughput of vLLM on Nvidia DGX Spark…
Get This Tool
License: MIT Any use incl. commercial
Local-run terms: MIT license permits unrestricted use, modification, and distribution for any purpose including commercial applications, provided original license and copyright notice are retained.
Screenshots 4
Pricing
- Model
- Free
Summary
llama.cpp is the MIT-licensed C/C++ engine that runs GGUF models on your own hardware. We ran the official llama.cpp:server Docker image (build 11681) on a CPU-only server with a 3B model (qwen2.5:3b). It had the smallest footprint of the runtimes we tested: a 1.21 GB image, about 1.6 GB of memory with the model loaded, and not a single DNS lookup during the test. Its server speaks the OpenAI chat API, so existing OpenAI code can use it by changing the base URL. Back to back on the same model file and cores, Ollama generated text 5 to 17% faster, though the VM was busier during the llama.cpp run. Like Ollama, it started 8 threads in a 3-core container by default, which made generation 7.6 times slower until we passed --threads 3. The trade-off is that it's an engine, not a model manager: you find and download GGUF files yourself.
Bottom line: Pick it when you want the engine itself, with full control over every setting and nothing phoning home. Pick an app like Ollama if you'd rather have model downloads and setup handled for you.
We installed and ran it
- How we installed it
- Official Docker image (llama.cpp:server), one command
- Docker image size on disk
- 1.21 GB
- Model we used
- qwen2.5:3b (Q4_K_M GGUF, 1.9 GB)
- Server start to first answer, model file not cached
- 8.7 s
- Memory used by the model process
- 1.6 GB
- Like Ollama, the llama.cpp server started 8 threads inside a 3-core container by default; text generation was 7.6x slower than with --threads 3.
- The first cold start after pulling the image took 74 s; the two after it took 9 s and 4 s. We show the median.
- It made no DNS lookups during the test (we log every lookup).
Measured against Ollama
| Measured | llama.cpp | Ollama |
|---|---|---|
| Server start to first answer, model file not cached (s) | 8.7 | 7.9 |
| Text generation speed, 53-token prompt, relative | 1x | 1.07x |
| Text generation speed, 441-token prompt, relative | 1x | 1.05x |
| Text generation speed, 723-token prompt, relative | 1x | 1.17x |
| Prompt reading speed, 53-token prompt, relative | 1x | 1.06x |
| Prompt reading speed, 441-token prompt, relative | 1x | 1.11x |
| Prompt reading speed, 723-token prompt, relative | 1x | 1.07x |
| Memory used by the model process (GB) | 1.6 | 2.2 |
| Docker image size on disk (GB) | 1.21 | 9.37 |
Same GGUF model file, same 3 cores, measured back to back in one session. The VM's CPU steal was higher during the llama.cpp run (13.6% vs 7.2%), so differences under about 10% are within noise.
Measured against LocalAI
| Measured | llama.cpp | LocalAI |
|---|---|---|
| Server start to first answer, model file not cached (s) | 3.5 | 6.4 |
| Text generation speed, 53-token prompt, relative | 0.98x | 1x |
| Text generation speed, 441-token prompt, relative | 0.96x | 1x |
| Text generation speed, 723-token prompt, relative | 0.99x | 1x |
| Prompt reading speed, 53-token prompt, relative | 1.22x | 1x |
| Prompt reading speed, 441-token prompt, relative | 0.99x | 1x |
| Prompt reading speed, 723-token prompt, relative | 0.94x | 1x |
| Memory used by the model process (GB) | 1.6 | 1.7 |
| Docker image size on disk (GB; LocalAI's backend is a separate download) | 1.21 | 1.2 |
Same GGUF model file, same 3 cores, same 4,096-token context, measured back to back in one session; LocalAI runs the model through its own llama-cpp backend. CPU steal: 12.8% during the llama.cpp run, 11.5% during LocalAI's, so differences under about 10% are within noise. LocalAI's image size does not include that backend, a separate 165 MB download. The llama.cpp numbers here come from a second llama.cpp run that day, not the one in the Ollama comparison.
Test machine: 8-vCPU AMD EPYC 9354P cloud VM, each test limited to 3 cores and 10 GB RAM, no GPU, no internet (isolated Docker network). Times in seconds are from this CPU-only machine; a desktop with a GPU will be much faster. The VM is shared, so text-generation speed is shown only as a ratio between tools run back to back, not as tokens per second. Docker image size is the on-disk size from docker image ls for every tool. Local AI tools we tested · How we test · raw data
Find your fit
Before you choose llama.cpp
More options in Local Inference Runtimes. Compare pricing, platforms, and features side by side.
LM-Kit One
Freemium
Companies needing data-residency compliance
Field notes
All field notes →- Ollama vs llama.cpp: a paired test with the same 3B model
Our controlled pair showed modest generation differences and different process-memory and Docker-image footprints. It did not establish a universal runtime winner.
- LocalAI vs llama.cpp: the differences our paired test could support
The generation results were close enough to treat as noise on our shared VM. Startup boundaries and LocalAI's separately installed backend mattered more to the operational comparison.
Pros
- OpenAI-compatible server, so code written for the OpenAI chat API can use a local model by changing the base URL.
- Its server handles several users at once with continuous batching, on by default.
- Small footprint in our test: about 1.6 GB of memory for a 3B model, and no DNS lookups.
Cons
- It is an engine, not a model manager: you pick and download GGUF files yourself, where apps like Ollama do that for you.
- In containers it does not size threads to the CPU limit; with 3 cores, generation was 7.6 times slower until we passed --threads 3.
- Answer quality is limited by the open-weight model you can run on your hardware.
About
- Platforms
- Linux, macOS, Windows, Android, ChromeOS, iOS, Web (WebGPU)
- API Available
- Yes
- Self-Hosted
- Yes
- Last Updated
- June 9, 2026
Best For
Who it's for
- Developers who want direct control over how a local model runs
- Self-hosted servers that need an OpenAI-compatible API
What it does well
- Serve a local model behind an OpenAI-compatible API
- Run GGUF models on CPU-only machines
- Tune threads, context and memory settings yourself
Integrations
Compare llama.cpp
Frequently Asked Questions
- Is llama.cpp free?
- llama.cpp offers a free option. Hosting, model usage, or optional services may have separate costs; check the pricing details and official terms.
- Is llama.cpp open source?
- Yes. llama.cpp is open source — the source repository is at https://github.com/ggml-org/llama.cpp.
- Does llama.cpp have an API?
- Yes. llama.cpp exposes a developer API. See the official documentation at https://llama.app for details.
- Can I self-host llama.cpp?
- Yes. llama.cpp supports self-hosting on your own infrastructure.
- When was llama.cpp released?
- llama.cpp was first released in 2023.
- What platforms does llama.cpp support?
- llama.cpp is available on: Linux, macOS, Windows, Android, ChromeOS, iOS, Web (WebGPU).
People also compared
Curated lists that include this category
llama.cpp is the open-source C/C++ engine that runs language models in the GGUF format on your own hardware. It is MIT-licensed and developed at github.com/ggml-org/llama.cpp. Other local AI tools build on it; LocalAI, for example, runs models through its llama-cpp backend.
You can use it three ways: build or download the binaries, run the official Docker server image, or install the Llama app and CLI from llama.app, which its page says comes from the llama.cpp team and Hugging Face. The server speaks the OpenAI chat API, so code written for that API can point at it. Its server documentation lists continuous batching and parallel decoding for several users at once, both on by default.
We ran the official llama.cpp:server Docker image on a CPU-only server with a 3B model (qwen2.5:3b, the same file we used for Ollama and LocalAI). The model used about 1.6 GB of memory, and the server made no DNS lookups during the test. Our measured results, including back-to-back comparisons with Ollama and LocalAI, are in the test section on this page.
One setting matters in containers: by default it started 8 threads inside a 3-core container, and text generation was 7.6 times slower than with –threads 3. If you limit CPUs, set the thread count to match.
Contribute
- Work on llama.cpp? Claim this listing.
- Embed a badge for your README or site.
Sign in to rate this tool or suggest an edit.