Loading...
Loading...
Defensive LLM cybersecurity caps at 44.5% on the best-performing MITRE ATT&CK tactic, per the Cyber Defense Benchmark (Simbian, 2026). The leader was Anthropic's Opus 4.6. Exfiltration was the worst-covered tactic. The same model triple-run on the same alert returns three different verdicts. Cost varies up to 17× across evaluated models with no correlation to accuracy. A raw LLM in front of the alert queue does not work.
Most LLM cybersecurity writing stops at the attack surface: prompt injection, training-data poisoning, model exfiltration. That work matters, and the OWASP LLM Top 10 captures it well. The question this brief answers is the other half: how well do frontier LLMs do the defensive work of a SOC, triage, investigation, response, when you put them in front of real alert telemetry?
The answer, on the current frontier, is poorly. Five LLM cybersecurity failure modes show up consistently in the data, and they do not get smaller as models get bigger. They get rearranged.
Simbian's Cyber Defense Benchmark (CDB) is a defense-side LLM evaluation built on real attack telemetry. It covers 13 of the 14 MITRE ATT&CK tactics, 105 procedures chained into full kill-chains of varying complexity, and deterministic ground truth, meaning every claim a model makes ("this IP was the attacker," "this is exfiltration") can be checked against the simulated environment.
The benchmark was built for one reason. Most LLM evaluations measure offensive capability (capture-the-flag, exploit-writing) because offense has clear, gradeable goals. Defense does not. A defender who never gets breached has nothing to show. CDB creates the missing scoreboard for defensive work.
The numbers, as of the most recent cohort: the best frontier model is Anthropic Opus 4.6, and its best-tactic coverage is 44.5%. Worst-tactic coverage sits near zero on several tactics, with Exfiltration consistently the weakest category across the cohort. The pass threshold is 50% on every tactic. Number of frontier models clearing it: zero.
The leading model gets fewer than half of one tactic right, on its best day, on a clean test. Production conditions are noisier than the benchmark, not cleaner. The honest framing for any defensive LLM cybersecurity conversation starts there.
Exfiltration deserves its own line. Data leaving the environment is the tactic boards measure security on, the one regulators hand down fines for, and the one current frontier models are weakest at detecting. An AI SOC that handles everything except exfiltration has the wrong everything. The benchmark is direct about which everything that is.
An earlier April 2026 CDB run reported 46% on the leader. The 44.5% figure reflects the most recent cohort with a fresh set of models added. Both are within the broader claim: no frontier model clears 50% on any tactic.
Run the same alert with the same telemetry through the same LLM three times. The answer is not the same answer three times.
In the benchmark runs, identical inputs to identical models return verdicts that swing between critical, benign, and expected, no action needed, sometimes on consecutive runs, on the same model, against the same telemetry. This is a structural property of probabilistic systems, not a bug to be patched.
The implication for security operations is direct:
The fix is structural: every verdict has to be checked against the underlying data the model claims it saw, before the analyst ever sees it.
LLMs are very good at writing. They produce SOC-style verdicts that read like a senior analyst wrote them: well-formatted, well-reasoned-sounding, MITRE-mapped, with a clean confidence score. Open the verdict and the IP address it cites never appeared in the traffic. The endpoint it names was decommissioned eighteen months ago. The MITRE technique it mapped to does not match the activity it described.
A human analyst, when uncertain, communicates uncertainty. Pace slows. Hedging shows up. The reader gets a signal. An LLM hedges only when asked to hedge. By default, fluent prose comes out at the same confidence regardless of how grounded the underlying claim is. This is the structural challenge behind LLM hallucinations applied to a defensive context.
In a SOC, three failure shapes follow:
Detection here is the same as for the coin-flip problem: do not take the verdict at face value. Make the architecture verify each named artifact against the actual data the model claims to have seen. Verdicts that fail verification get demoted; analysts see only the verdicts whose claims survive the check.
Cost is the part of the conversation that breaks budgets quietly. There are credible reports of organizations burning through full-year AI budgets in a single quarter as defensive LLM usage scaled.
The benchmark data is direct about why naive scaling does not pay back. Across the cohort of evaluated models:
The right framing is a Pareto curve, not a ladder. At a $10 per-investigation budget, there is one sensible model choice up through about $15. Past $15, options open: a more expensive frontier model, or the cheaper model running longer with more steps. The optimal point shifts with the task. A phishing triage and a lateral-movement investigation do not warrant the same model spend.
The practical rule:
The previous four failures look like engineering problems. The fifth is the structural one that explains why the others persist on the current frontier.
Offensive LLMs work because the goal is concrete. Get the data out. Deface the page. Move laterally to the database. The model does not need to understand the defender's environment; it needs to execute against a goal it can verify. Research has consistently found that LLMs perform well when goals are clearly specified.
Defense has no equivalent. The best defensive outcome is nothing happened. If an organization runs for two years without a breach, no one can say whether the security program is excellent or the adversaries simply did not probe its weakest point. There is no ground truth to optimize against. The model has no signal to learn from.
That is why a bigger model does not solve the problem. A more capable model still faces a goal it cannot grade. The frontier-LLM-versus-frontier-LLM benchmark race delivers attack capability much faster than it delivers defensive capability, because the attacker's grader is built in (did the data leak?) and the defender's grader has to be constructed.
"AI does not actually simplify anyone's life. It's a sharper knife that everybody gets, the offensive side gets it, defenders get it, your competition gets it. So the question is: what can you do that is beyond what everybody else is doing?"
That quote is the build-versus-buy frame. A raw frontier LLM gives every defender the same blade. Edge belongs to whoever wraps that blade in something the others do not have.
The same Opus 4.6 that caps at 44.5% on CDB raw clears 95% on the same benchmark when wrapped in a harness. The model did not change. The architecture around it did.
Four components compose the harness, and each addresses one of the failure modes above.
Every LLM claim ("this IP was the attacker," "this lateral movement crossed these hosts") gets checked against the actual data the model says it saw. Claims that fail verification get demoted. This is the structural answer to fluent hallucinations and to coin-flip inconsistency. Both are caught at the gate, not in the verdict.
The harness grounds every investigation in the organization's own context: network topology, IP ranges, asset criticality, identity tiers, past verdicts, scheduled maintenance, approved pentest windows. The Context Lake gives the model the missing defensive grader: was this activity expected, given everything else this organization knows?
The harness picks model, depth, and time budget per task. Triage tasks run on cheaper models; high-value investigations get frontier spend. The Pareto curve gets respected at runtime, which is what keeps the bill bounded as alert volume scales.
The harness reflects on its own results. On one integration that started at 0% query accuracy because the model had never seen the query language, the loop reached near-100% on the task within a day of running, without engineering tickets. This is self-improving, not self-driving: humans keep containment authority and escalation calls; agents handle the mechanical work.
Better frontier models (Opus 4.6, Mythos, Fable 5, GPT 5.5) raise this ceiling further. They do not replace the harness. They multiply it.
If the benchmark is the test, an AI SOC Agent on the harness is what passes it. A companion piece, What is an LLM Harness, and why does your SOC need one?, walks through the architecture in full. This brief stops at the failure side, on purpose.
Buyers evaluating LLM cybersecurity for the SOC in the next twelve months will be looking at a wave of "AI SOC" products built on the same frontier models. The question is not which model the vendor uses. Almost every credible vendor uses one of three or four. The question is how the vendor handles the five failure modes above.
A short test:
A vendor who answers those five questions cleanly has built the harness. A vendor who answers them with adjectives has wrapped a frontier model and called it product. The benchmark is the difference.
Q: Are general-purpose LLMs enough for cybersecurity? No. LLM cybersecurity scores top out at 44.5% coverage on the best-performing MITRE tactic in the Cyber Defense Benchmark, with several tactics near zero. General-purpose LLMs lack the verifiable gates, organizational context, and reasoning architecture that defensive work requires. The model is useful only when wrapped in a harness that compensates for those gaps.
Q: Why do LLMs fail at defense if they succeed at offense? Offense has clear goals (data exfiltrated, system compromised) that the LLM can verify on its own. Defense has no equivalent ground truth, the best defensive outcome is that nothing happens, so the model lacks the grader it needs to optimize against. The asymmetry is structural, not a capability gap.
Q: Does a more expensive frontier model solve the SOC accuracy problem? No. The Cyber Defense Benchmark cohort shows up to a 17× cost spread with no correlation between cost and accuracy. The leading model still caps at 44.5%, and several less expensive models score above some frontier-tier models on key tactics. Spend should be routed per-task on a Pareto curve, not defaulted to the most expensive option.
Q: Are LLM hallucinations a real risk in the SOC? Yes. Defensive LLMs routinely cite IP addresses that never appeared in the traffic, endpoints that were decommissioned, and MITRE techniques that do not match the activity described. The prose reads as evidence-backed; the evidence is fabricated. Without verification gates, analysts trust polished verdicts more than they should.
Q: What is an LLM harness? A harness is the architecture around a base LLM that turns probabilistic output into reliable SOC behavior. The core components are verifiable gates against every claim, an organizational context layer (Context Lake), a reasoning engine that routes cost and time per task, and a self-improvement loop. Same model, harnessed, lifts CDB coverage from 44.5% to 95% in Simbian's evaluation.
Q: How do you measure whether an AI SOC vendor is actually working? Ask for third-party defensive benchmark scores, run-to-run consistency on identical alerts, examples of verdicts that lost claims to verification, the cost curve per task type, and evidence of self-improvement from prior runs. Adjectives are not answers. Numbers and traces are.
Better LLMs are coming this year and next. They will widen the offense-defense gap unless the defender's harness scales with them. The LLM cybersecurity decision in front of every SOC leader is not which frontier model to license. It is which architecture turns whatever model is current into reliable defensive work. Book a demo to see how Simbian's AI SOC Agent runs the harness on production telemetry, or watch the Why LLMs Fail in the SOC webinar replay for the full benchmark walkthrough.
