GPT-5 and Gemini 3.1 Pro sit in the mid-price range at $1.07 and $1.85 per run. Both plateau around 2% flags and stop climbing. Throwing more budget at them does not help, because the limit is not token cost, it is that the agent believes it has completed its task and has no work left to do.

For practitioners evaluating models for production SOC use, this has two practical implications. First, you cannot assume a model that is two-thirds of the price gets you two-thirds of the detection – the curve does not work that way. Second, if you are buying on cost alone, you are buying an agent that gives up.

Why LLMs Are Great at Offense and Poor at Defense

Frontier models are rapidly getting more and more capable at finding and exploiting software vulnerabilities. So why does the same model class fail when you ask it to defend? The asymmetry is structural, not a matter of training data. Offense has a clear goal and a fast feedback loop. Did the exploit land? Did the shell open? Did I get root? The model knows when it has won. Every iteration refines toward that signal, and the environment cooperates – errors surface immediately, successful paths compound.

Defense has none of that. The goal is to "find every malicious event in this telemetry" as this is the only way to understand the scale of an attack and stop the attacker. The challenge is you have no idea how many there are – the ground truth is invisible. Signal-to-noise collapses because normal enterprise activity looks a lot like reconnaissance until you have context. Every wrong hypothesis costs you queries without telling you that you were wrong. You cannot know when you have won, only when you have run out of budget.

Scale does not fix this. A bigger context window does not fix this. A more capable reasoning chain does not fix this. These are all things frontier models already do well, and it was not enough. 

Why Past AI Security Benchmarks Don't Tell You This

Ask a frontier LLM what MITRE ATT&CK technique maps to LSASS memory dumping, and it will answer correctly. Ask it to look at ten thousand Sysmon events and tell you whether anyone is dumping LSASS right now, and the answer is very different. 

This is the gap in prior cybersecurity benchmarks. CTI-Bench uses multiple choice. CyBench runs CTF puzzles in a Kali container. CyberSOCEval asks MCQ questions about sandbox malware reports. CTI-REALM is agentic but measures whether a model can convert a threat report into a Sigma rule, not whether it can find an attacker in logs it has never seen. Most of these are gameable by training on the test set.

The Cyber Defense Benchmark was built to be difficult to game. It replays real Windows telemetry — Sysmon and Security event logs — captured from controlled lab environments running various attack vectors including Empire, Covenant, Mimikatz, and Rubeus. Each campaign run mutates with seeded randomization hostnames, IP addresses, user identities, domain SIDs, timestamps, and other log fields that could be used to connect pieces of an attack into a coherent attack picture. No two runs present identical data. The model receives a threat intelligence briefing and a SQL-queryable log database, then gets 50 queries to find the attacker. Scoring is deterministic: exact timestamp matches against ground truth. No LLM judges, no subjective rubrics. 

The Harness Is the Product

Here is the finding buried under the failure: the gap between 46% average per-tactic performance and 95% verdict disposition accuracy in production is closeable. At Simbian we know because we have closed it.

Simbian's AI SOC Agent achieve 95% accuracy in production enterprise environments for SecOps cyber defense. The model doing the reasoning is often the same as one of the models on the leaderboard above. What is different is what surrounds it.

In SecOps, the scaffolding around the model, what we call the harness, is the key to better security. Context about the organization's assets, users, baseline behavior, and previous incidents. Deterministic retrieval that grounds every hypothesis in real data. Agentic loops structured around investigation patterns that analysts actually use, not ReAct-for-its-own-sake. Tool access calibrated to what the agent needs at each step. An assessment loop that ensures the LLM keeps generating and evaluating hypotheses until the task is complete. Cost controls so the agent can afford to hunt the way a good hunter actually hunts. Simbian’s AI solutions provide this scaffolding. 

This is why the benchmark results should not be read as "AI cannot do defense." They should be read as "an LLM alone is not a defender, and that distinction matters commercially." A model is a component. An agent is a product. An AI-powered and AI-ready SOC need both.

What This Means for Your SOC

If you are evaluating AI for security operations, three things follow from the benchmark:

If you want the practical version of this argument, you can book a demo with Simbian and put it against your own telemetry.

AI-armed attacks will continue to redefine the threat landscape. Now we need to extend AI models so that they are equally effective at defending our organizations. This benchmark is one step in helping us understand how to do that.

Frequently Asked Questions

Q: What is the Cyber Defense Benchmark? A: It is the first scalable AI security benchmark that places an agent in front of real attack telemetry and asks it to autonomously hunt. Eleven frontier LLMs were tested across 884 runs covering 105 attack procedures and 93 MITRE ATT&CK sub-techniques. Agents receive a threat briefing plus a SQL-queryable log database and submit exact timestamps of malicious events. Scoring is deterministic, with no LLM judges.

Q: Why did every frontier LLM fail? A: Defense is structurally harder for an LLM than offense. Offense has a clear goal and fast feedback – did the exploit work? Defense has unknown ground truth, thousands of malicious events hidden inside far more noise, and no signal that you have found everything. Raw reasoning capability is not enough without a harness that grounds the agent in organizational context and guides investigation.

Q: Can a cheaper model work for production SOC use? A: Not on its own. The cost-quality relationship is a cliff. Gemini 3 Pro at $1.85 per run finds 2% flags; Opus 4.6 at $17.89 finds 4.5%. Mid-priced models plateau around 2% flags because the model thinks it has completed its work and requires no additional queries. In production, cheaper models only work inside a harness that compensates for their reasoning limits.

Q: Is the benchmark Windows-only? A: For v1.0, yes. All procedures are sourced from Windows endpoints using Sysmon and Security event logs. Linux, macOS, cloud, and network telemetry are planned for future versions. The current library covers 93 ATT&CK sub-techniques across 13 of 14 tactics. The known gap is Reconnaissance, which v1.0 does not yet include, and is planned to be created using Simbian PenTest agent.

Q: Will future frontier models crack this benchmark? A: Some ground will be made up as every frontier release is more capable than the last. But the structural asymmetry between offense and defense does not disappear with more parameters. The path to 95% accuracy in production will continue running through better harnesses, not through raw LLM progress alone. The benchmark will keep raising the ceiling as models improve.

Q: Where can I see the full results? A: The full leaderboard, methodology, per-procedure detection rates, and Pareto frontier analysis are published on the Cyber Defense Benchmark page. Additional models and runs are being added.

Every frontier model on the current leaderboard is going to get better. The benchmark will get harder. The practical question for SOC leaders is not which model wins next quarter — it is whether the agent you deploy today has the harness to get from 46% to 95%. On April 29, we are walking through the full results and the techniques that close that gap in a session called "Claude and OpenAI Will Change Security — Just Not the Way You Think." Seats are limited.