Cuey
Freemium
Power users comparing multiple LLMs daily
SOCBench
FreeOpen Source
Vendor claims about AI detection performance are untestable until you run the same model on the same labeled traffic and compare it against ground truth — which almost nobody publishes. SOCBench is the open benchmark built to close that gap for cybersecurity operations.
Similar tools
CueyFreemiumCuey sits inside your existing ChatGPT, Claude, or Gemini session and routes the same prompt…
AgiRankerFreeThe tool lets you rank frontier models on a 0–100 AGI Score, explore per-domain breakdowns…
oqoqoPaidOqoqo runs agent evaluation experiments on managed cloud infrastructure, spinning up sandboxed…
ClientCodedFreemiumClientCoded runs adversarial multi-turn conversations against your agent, scores every exchange…
Pricing
- Model
- Free
Summary
The published detection module runs three frontier LLMs across four analyst personas against 1,205 labeled NetFlow units, scoring each provider-persona pair on F1 (per-flow, per-pair, per-host), verdict accuracy, cost per alert, latency, and completion rate. That scoreboard lets you stop trusting marketing and start comparing models on telemetry that resembles what a real monitoring tier sees. The ceiling is visible immediately: detection is live, but triage, investigation, threat hunting, detection engineering, and threat intelligence are all roadmap items. If your evaluation need extends beyond binary flow classification, SOCBench does not cover it yet — and the roadmap carries no committed dates.
Bottom line: Pick SOCBench when you need a reproducible, auditable basis for choosing which LLM to put in your detection pipeline; shelve it when the SOC capability you are evaluating is anything other than network-flow classification.
Find your fit
Before you choose SOCBench
More options in LLM Evaluation & Benchmarks. Compare pricing, platforms, and features side by side.
AgiRanker
Free · Open source
Researchers tracking verifiable AI capability trends
Pros
- Four-dimension scoring (efficacy, cost, latency, reliability) on every provider-persona pair, so a model that wins on F1 but doubles your alert cost or fails to return structured output on a measurable fraction of runs does not slip through unnoticed.
- Four analyst personas applied to the same evaluation units, which means you see whether a model's performance is stable across roles or collapses when the prompt changes — the kind of variance that only shows up after you have deployed.
- Published labeled NetFlow corpus alongside the results, so you can audit the evaluation data, reproduce the run independently, or extend the corpus to cover traffic patterns from your own environment.
- Open-source and contribution-ready architecture, which means teams with domain-specific telemetry can add evaluation units rather than waiting for a vendor to add their use case.
- Free with no paywalled results, so you get the full scoreboard without a sales call — which removes the version of vendor bias where you only see the numbers the vendor chose to share.
Cons
- Detection on NetFlow data is the only live capability. If the SOC task you need to evaluate is alert triage, multi-step investigation, threat hunting, detection engineering, or threat intelligence, none of those have published results — teams with those needs are writing their own evaluation framework.
- The benchmark covers three specific frontier models at the time of publication; any model your team is evaluating that is not in that set requires you to run the evaluation harness yourself, which means reading the methodology docs and instrumenting a new provider integration before you get a single number.
- No self-hosted option and no API surface means SOCBench is a reference benchmark you run against, not a service you call — teams expecting a hosted evaluation endpoint or a CI-integrated scoring service will need to build that wrapper themselves or adopt a competitor evaluation platform that offers it.
About
- API Available
- No
- Self-Hosted
- No
- Last Updated
- July 8, 2026
Best For
Who it's for
- AI model evaluation in cybersecurity
- Benchmarking detection performance
- Reproducible LLM comparisons on NetFlow data
What it does well
- Compare LLMs on SOC detection tasks
- Measure AI efficacy, cost, latency, and reliability in cyber operations
- Contribute to or extend the open SOC benchmark
Frequently Asked Questions
- Is SOCBench free?
- SOCBench offers a free option. Hosting, model usage, or optional services may have separate costs; check the pricing details and official terms.
- Is SOCBench open source?
- Yes. SOCBench is open source.
Curated lists that include this category
AI vendors move faster than the evaluations designed to check them, and SOCBench is DeepTempo’s answer to that problem. The benchmark runs candidate AI systems through labeled NetFlow data and scores them on four dimensions that actually decide whether a tool ships to production: efficacy (F1, verdict accuracy), cost (USD per alert, token economics), latency (MTTD, P50, P95), and reliability (completion rate, defect rate). Each provider is tested under four analyst personas — soc_analyst, threat_analyst, adversary_hunter, detection_engineer — over the same 1,205 evaluation units, producing a per-(provider × persona) result set rather than a single aggregate score that hides persona-level variance.
The differentiating feature is the multi-dimensional scoring grid applied uniformly across providers. Most internal model evaluations pick one metric — usually F1 — and stop there. SOCBench forces cost and latency into the same table, so a model that scores highest on detection but costs three times more per alert or returns malformed output at a non-trivial rate gets that tradeoff made visible rather than buried.
The benchmark fits teams that are vetting LLMs for a detection or monitoring workload and want a reproducible comparison they can cite. It breaks as soon as the SOC capability under evaluation moves beyond flow classification: triage, investigation, threat hunting, detection engineering, and threat intelligence are all scoped roadmap items with no published results. Teams whose evaluation needs span those areas will be building their own evaluation harness — at which point SOCBench is reference material, not a complete answer.
The project is open source, grew out of DeepTempo’s validation work on their LogLM model and the open-source AI-SOC Vigil, and is explicitly structured for external contributions. The labeled NetFlow corpus is published alongside the benchmark, so teams can extend the dataset or adapt the methodology to other log types.
Contribute
- Work on SOCBench? Claim this listing.
- Embed a badge for your README or site.
Sign in to rate this tool or suggest an edit.