SOCBench
Pricing
- Model
- Free
Summary
Vendor claims about AI detection performance are untestable until you run the same model on the same labeled traffic and compare it against ground truth — which almost nobody publishes. SOCBench is the open benchmark built to close that gap for cybersecurity operations.
The published detection module runs three frontier LLMs across four analyst personas against 1,205 labeled NetFlow units, scoring each provider-persona pair on F1 (per-flow, per-pair, per-host), verdict accuracy, cost per alert, latency, and completion rate. That scoreboard lets you stop trusting marketing and start comparing models on telemetry that resembles what a real monitoring tier sees. The ceiling is visible immediately: detection is live, but triage, investigation, threat hunting, detection engineering, and threat intelligence are all roadmap items. If your evaluation need extends beyond binary flow classification, SOCBench does not cover it yet — and the roadmap carries no committed dates.
Bottom line: Pick SOCBench when you need a reproducible, auditable basis for choosing which LLM to put in your detection pipeline; shelve it when the SOC capability you are evaluating is anything other than network-flow classification.
Community Performance Report Card
No community ratings yet. Be the first to rate this tool!
Community Benchmarks Community
Sign in to submit a benchmarkNo community benchmarks yet. Be the first to share a real-world data point.
Pros
Sign in to edit- Four-dimension scoring (efficacy, cost, latency, reliability) on every provider-persona pair, so a model that wins on F1 but doubles your alert cost or fails to return structured output on a measurable fraction of runs does not slip through unnoticed.
- Four analyst personas applied to the same evaluation units, which means you see whether a model's performance is stable across roles or collapses when the prompt changes — the kind of variance that only shows up after you have deployed.
- Published labeled NetFlow corpus alongside the results, so you can audit the evaluation data, reproduce the run independently, or extend the corpus to cover traffic patterns from your own environment.
- Open-source and contribution-ready architecture, which means teams with domain-specific telemetry can add evaluation units rather than waiting for a vendor to add their use case.
- Free with no paywalled results, so you get the full scoreboard without a sales call — which removes the version of vendor bias where you only see the numbers the vendor chose to share.
Cons
Sign in to edit- Detection on NetFlow data is the only live capability. If the SOC task you need to evaluate is alert triage, multi-step investigation, threat hunting, detection engineering, or threat intelligence, none of those have published results — teams with those needs are writing their own evaluation framework.
- The benchmark covers three specific frontier models at the time of publication; any model your team is evaluating that is not in that set requires you to run the evaluation harness yourself, which means reading the methodology docs and instrumenting a new provider integration before you get a single number.
- No self-hosted option and no API surface means SOCBench is a reference benchmark you run against, not a service you call — teams expecting a hosted evaluation endpoint or a CI-integrated scoring service will need to build that wrapper themselves or adopt a competitor evaluation platform that offers it.
Community Reviews
Sign in to write a reviewNo reviews yet. Be the first to share your experience.
About
- API Available
- No
- Self-Hosted
- No
- Last Updated
- 2026-07-08T20:44:07.495Z
Best For
Who it's for
- AI model evaluation in cybersecurity
- Benchmarking detection performance
- Reproducible LLM comparisons on NetFlow data
What it does well
- Compare LLMs on SOC detection tasks
- Measure AI efficacy, cost, latency, and reliability in cyber operations
- Contribute to or extend the open SOC benchmark
Discussion Community
Sign in to commentNo discussion yet. Sign in to start the conversation.
Compare SOCBench
Spotted incorrect or missing data? Join our community of contributors.
Sign Up to ContributeCommunity Notes & Tips Community
Sign in to contributeBe the first to contribute. General notes, observations, gotchas, and tips from people who use this tool day-to-day.
Frequently Asked Questions
- Is SOCBench free?
- Yes — SOCBench is fully free to use. There is no paid tier.
- Is SOCBench open source?
- Yes. SOCBench is open source.
Hours Saved & ROI Stories Community
Sign in to contributeBe the first to contribute. Concrete time/cost savings, with context. e.g. "Cut my code review backlog from 4h to 45m per week."
Curated lists that include this category
AI vendors move faster than the evaluations designed to check them, and SOCBench is DeepTempo’s answer to that problem. The benchmark runs candidate AI systems through labeled NetFlow data and scores them on four dimensions that actually decide whether a tool ships to production: efficacy (F1, verdict accuracy), cost (USD per alert, token economics), latency (MTTD, P50, P95), and reliability (completion rate, defect rate). Each provider is tested under four analyst personas — soc_analyst, threat_analyst, adversary_hunter, detection_engineer — over the same 1,205 evaluation units, producing a per-(provider × persona) result set rather than a single aggregate score that hides persona-level variance.
The differentiating feature is the multi-dimensional scoring grid applied uniformly across providers. Most internal model evaluations pick one metric — usually F1 — and stop there. SOCBench forces cost and latency into the same table, so a model that scores highest on detection but costs three times more per alert or returns malformed output at a non-trivial rate gets that tradeoff made visible rather than buried.
The benchmark fits teams that are vetting LLMs for a detection or monitoring workload and want a reproducible comparison they can cite. It breaks as soon as the SOC capability under evaluation moves beyond flow classification: triage, investigation, threat hunting, detection engineering, and threat intelligence are all scoped roadmap items with no published results. Teams whose evaluation needs span those areas will be building their own evaluation harness — at which point SOCBench is reference material, not a complete answer.
The project is open source, grew out of DeepTempo’s validation work on their LogLM model and the open-source AI-SOC Vigil, and is explicitly structured for external contributions. The labeled NetFlow corpus is published alongside the benchmark, so teams can extend the dataset or adapt the methodology to other log types.