Back to Paper
cs.CRcs.AI
Local ID: 2604.19533v3
AI Summary: gemma4:e4b
Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOps
By Alankrit Chona, Igor Kozlov, Ambuj Kumar
Revision History Timeline
v14/21/2026
4/21/2026
“13 pages, 3 figures, 5 tables. Complete benchmark and hunt traces available on request”
v24/22/2026
4/22/2026
“13 pages, 3 figures, 5 tables. Complete benchmark and hunt traces available on request”
v34/23/2026
4/23/2026
“Updated leaderboard with newer models”
★ Version indexed in ExplorerComparing v2 vs v3
Green = Added • Red = Removed
Title Comparison
Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOps
Authors Comparison
No author changes.
v2 Comment
“13 pages, 3 figures, 5 tables. Complete benchmark and hunt traces available on request”
v3 Comment
“Updated leaderboard with newer models”
Abstract Word Diff
We introduce the Cyber Defense Benchmark, a benchmark for measuring how well large language model (LLM) agents perform the core SOC analyst task of threat hunting: given a database of raw Windows event logs with no guided questions or hints, identify the exact timestamps of malicious events. The benchmark wraps 106 real attack procedures from the OTRF Security-Datasets corpus - spanning 86 MITRE ATT&CK sub-techniques across 12 tactics - into a Gymnasium reinforcement-learning environment. Each episode presents the agent with an in-memory SQLite database of 75,000-135,000 log records produced by a deterministic campaign simulator that time-shifts and entity-obfuscates the raw recordings. The agent must iteratively submit SQL queries to discover malicious event timestamps and explicitly flag them, scored CTF-style against Sigma-rule-derived ground truth. Evaluating five frontier models - Claude Opus 4.6, GPT-5, Gemini 3.1 Pro, Kimi K2.5, and Gemini 3 Flash - on 26 campaigns covering 105 of 106 procedures, we find that all models fail dramatically: the best model (Claude Opus 4.6) submits correct flags for only 3.8% of malicious events on average, and no run across any model ever finds all flags. We define a passing score as >= 50% recall on every ATT&CK tactic - the minimum bar for unsupervised SOC deployment. No model passes: the leader clears this bar on 5 of 13 tactics and the remaining four on zero. These results suggest that current LLMs are poorly suited for open-ended, evidence-driven threat hunting despite strong performance on curated Q&A security benchmarks.