Loading...
New
See what manual alert triage really costs your security team.
New
See what manual alert triage really costs your security team.
See what manual alert triage really costs your security team.
See what manual alert triage really costs your security team.
Loading...
AI SOC agents usually stop investigating when their LLM decides it's finished, not when the evidence runs out. On Simbian's October 2026 Cyber Defense Benchmark, none of 30 LLMs clears 50% coverage on every MITRE ATT&CK tactic, and the leader, Opus 5, averages 45.1% of each attack's steps. The fix is a coverage-based stopping rule held outside the model.
Your AI SOC agent decided it was done. It never checked whether it was.
We've watched our own agent do it. On one incident, a shallow run called a malicious-domain lookup a true positive: bad domain reputation, repeated DNS sinkhole hits. Who owned the endpoint, which user was behind it, what process made the request? Never answered. A deeper run on the same incident spotted that the flagged IP was an internal DNS resolver answering for many downstream clients, so the lookup couldn't show which downstream client, if any, was compromised. That run closed the case as inconclusive.
The shallow run stopped doubting itself while it still had questions left. This time the result was a true-positive verdict the evidence didn't support. Turn the evidence around and the same habit closes a real intrusion as benign.
AI SOC agents that let a general-purpose LLM decide when they're done usually quit before the evidence runs out, and nothing on your dashboard will tell you. On Simbian's October 2026 Cyber Defense Benchmark, nearly every hunt trace we publish ends with the model choosing to stop, long before it has found the whole attack.
The benchmark (arXiv 2604.19533) hands each model a threat briefing, a SQL database of real Windows attack telemetry, and a 50-query budget. A separate cap limits turns (any agent step counts as one). On each turn the model can run a query and see at most 10 rows, submit timestamps of events it thinks are malicious, or call give_up to say it's finished.
The opening moves look like decent hunting. Here are turns one and five from an Opus 4.6 run in the paper, where the model sizes up the data and then scans for noisy tradecraft:
-- Turn 1. How much data is there?
SELECT COUNT(*), MIN(TimeCreated), MAX(TimeCreated) FROM logs
-- 134,867 records, about 1 hour 51 minutes of activity
-- Turn 5. Look for loud tradecraft
SELECT TimeCreated, EventID, Image, CommandLine FROM logs
WHERE EventID='1' AND (CommandLine LIKE '%encoded%'
OR CommandLine LIKE '%bypass%' OR CommandLine LIKE '%invoke%')
-- 16 new malicious events found
Keyword scans like that find the loud steps fast, and the models rarely follow them with the narrower queries that would find the rest.
It's our benchmark, and we sell a fix, so check the traces yourself on the Cyber Defense Benchmark page. Across 1,314 runs in the October 2026 edition, even the leader, Opus 5, ends its hunts having found 45.1% of each attack's steps, on average. Then it calls give_up, which in a SOC queue is a closed case.
From the alert queue, a premature closure by an AI SOC agent looks exactly like a finished investigation, with a clear verdict and tidy reasoning. The questions that decide whether there was a compromise never got asked, and often nothing in the output records which ones were skipped.
Take an ordinary phishing alert.
The confidence-based agent parses the email and checks the sender, the linked domains, and the attachment hash against every threat intelligence feed it has. Nothing is known-bad. The write-up reads well, and the case closes as benign.
The coverage-based agent gets to the same point and keeps going, because questions on its list are still open. Did anyone click the link? Do DNS and EDR logs show any host resolving the domain or running the attachment? Did the recipient's account sign in from somewhere new afterwards? Did the same message land in other mailboxes? The case closes only when each is answered or recorded as open.
Confidence-based vs coverage-based stopping in AI SOC agents.
| Question | Confidence-based stopping | Coverage-based stopping |
|---|---|---|
| When does it stop? | When the model feels sure of a verdict | When the questions that could change the verdict are answered |
| What decides "done"? | The model's own judgment | A list held outside the model |
| What does a reviewer see? | A confident narrative | The evidence checked and the questions still open |
| What happens on a novel alert? | Stops at the first plausible story | Keeps going until the list is closed |
| How does a model upgrade show up? | Shorter investigations nobody notices | A change in what gets checked, visible in the report |
LLM overconfidence ends investigations early because nothing in a defensive task tells an AI SOC agent how much of the attack is left. No alert fires when the last malicious event is found, so the model stops when the case feels finished, and LLMs feel finished too soon.
Alankrit Chona, my co-founder and Simbian's CTO, describes the pattern this way:
"The models confuse confidence for coverage — they decide they're done long before they've exhausted the hypotheses. In a real SOC, that's a missed intrusion."
Barkan, Black, and Sourbut found the same thing. In their ICLR 2026 paper, every LLM they tested was overconfident, and "[o]n multi-step agentic tasks, the overconfidence of several frontier LLMs worsens as they progress through the tasks". A threat investigation is that kind of task, so the deeper the model goes, the surer it is likely to get, with or without new evidence.
A model upgrade can make an AI SOC agent stop sooner. In three of the seven release pairs we track on the October 2026 Cyber Defense Benchmark, Sonnet 5, Opus 4.8, and Haiku 5.5 each run fewer turns than their predecessors and cover less of the attack. Haiku 5.5 drops from 35 turns a hunt to 7.
Each pair comes from one lab and ran the same campaigns under the same rules, which is as close to a controlled experiment as this data gets. That isn't very close, because a newer model changes more than when it stops. For Opus, the earlier model is 4.6, two releases back, and most of the drop came at 4.7.
Release pairs on the Cyber Defense Benchmark (October 2026 edition): mean agent turns per hunt and mean Coverage Score.
| Newer model | Turns (newer) | Coverage (newer) | Earlier model | Turns (earlier) | Coverage (earlier) |
|---|---|---|---|---|---|
| Sonnet 5 | 45 | 19.9% | Sonnet 4.6 | 59 | 36.9% |
| Opus 4.8 | 34 | 32.7% | Opus 4.6 | 52 | 44.5% |
| Haiku 5.5 | 7 | 9.3% | Haiku 4.5 | 35 | 17.6% |
| GPT-6 Luna | 31 | 21.6% | GPT 5.6 Luna | 15 | 15.7% |
| GLM 5.3 | 54 | 33.9% | GLM 5.2 | 57 | 35.0% |
| Grok 4.7 | 52 | 37.6% | Grok 4.6 | 53 | 41.3% |
| GPT 5.6 Sol | 47 | 41.0% | GPT 5.5 | 46 | 37.4% |
A turn is any agent step, so turns can run past the 50-query budget. Coverage Score is the share of each attack's steps that the agent backs with at least one correctly submitted malicious-event timestamp.
The other four pairs don't move that way. GPT-6 Luna runs longer and finds more, while GLM 5.3, Grok 4.7, and GPT 5.6 Sol run about as many turns as their predecessors did.
Kimi K3 is the awkward case for this argument. It sits outside the seven pairs, runs fewer turns than Kimi 2.7 Code, and still covers more, so turn count alone won't tell you much. Across all 30 models, though, the agents that keep querying usually find more of the attack, provided the extra queries test something.
Anthropic's own numbers show what a nudge is worth. Its 2026 Claude Haiku 5.5 system card reports two ExploitBench runs of Haiku 5.5. One was plain, and the other used a harness that injected a keep-trying prompt whenever the model "voluntarily stopped short of the budget." With the nudge, its capability score rose from 38% to 49%.
Re-test on your own alerts every time the model underneath changes. On most dashboards, Haiku 5.5's drop from 35 turns to 7 would look like a speed-up.
Running an AI SOC agent longer doesn't fix early stopping on its own. Extra turns add investigation depth only when the agent spends them testing a specific hypothesis. Nemotron 3 Super is the cautionary case. It runs some of the longest hunts in the October 2026 Cyber Defense Benchmark and still finishes near the bottom on coverage.
Agents that rarely quit fail differently. SIABench is a 2026 benchmark from Concordia University and Defence Research and Development Canada. It tested LLMs on security incident analysis and found that "[n]ewer models, such as Claude-4.5-Sonnet and GPT-5, rarely give up." GPT-5 gave up on 0.44% of questions and got 14.94% wrong. That's about one question in seven answered confidently and incorrectly.
Exfiltration is the weakest MITRE ATT&CK tactic for most of the 30 models on the October 2026 Cyber Defense Benchmark, and even the leader, Opus 5, clears the 50% coverage bar on only 7 of 13 tactics. The models that run the longest hunts miss Exfiltration too.
That's the stage where data leaves the building. An agent running on its own sense of done has no reason to look there unless Exfiltration is on a list it has to close.
Nobody gets paged for a false negative, so a miss like that can sit for years. CISA's February 2024 Volt Typhoon advisory describes actors "maintaining access and footholds within some victim IT environments for at least five years."
AI SOC agents should decide they're done with a coverage-based stopping rule: a completion criterion, held outside the LLM, that ends an investigation only when every hypothesis that could change the verdict has been tested and every relevant MITRE ATT&CK tactic has been checked.
In Simbian's AI SOC Agent, we built five pieces around the model to enforce that rule and make it checkable. In an earlier run of the benchmark (the April 2026 harness run, independently verified by a global MSSP), the best frontier LLM alone reached 46% coverage, and Simbian's harness reached 95% on the same scenarios.
A planner maps out the investigation first. An assessment loop then keeps the model generating and testing new hypotheses well past the first narrative that sounds complete enough to close on.
Every verdict ships with a step-by-step reasoning trace, so a reviewer can see what was queried and spot a thin case before signing off on it.
A second agent, the critic, watches the investigation as it runs, judges what's going right and wrong, and feeds corrections back to catch premature verdicts.
The agent treats its first answer as a hypothesis to test. In the DNS incident, the deeper run worked out that the flagged IP was a resolver and named the resolver logs as the place to find the client that made the request.
Every investigation ends in a case report whose summary lists what remains unresolved, so the gaps are the first thing a reviewer reads. That report, and your analyst's call on it, goes into the Context Lake, the shared memory every Simbian Agent reads from and writes to, so the next investigation starts from what this one learned. That's the loop behind Simbian's Self-Improving Defense.
In production, Simbian investigates and responds to every threat, on average under 4 minutes from detection to response. The customer's own analysts approve 95% of the responses it proposes. Their call, not ours.
Test an AI SOC agent for early stopping in the proof of concept, on your own alerts, by checking what makes it stop and what it skipped. Most demos skip this part of AI agent evaluation and show you MTTR, which on a shallow investigation only measures how quickly the agent stopped.
Q: What are AI SOC agents? AI SOC agents are LLM-driven AI agents for security that investigate alerts the way an analyst would. They query SIEM, EDR, identity, and email logs, test hypotheses, and return a verdict with a reasoning trace. How far they investigate before that verdict depends on the stopping rule they run under.
Q: Why do AI SOC agents stop investigating early? Most AI SOC agents let the LLM decide when the work is done, and the LLMs tested are overconfident. A 2026 ICLR paper by Barkan, Black, and Sourbut found several frontier models grow more overconfident as agentic tasks progress. Nothing in an investigation signals the last malicious event was found, so the model's own doubt is the only brake.
Q: How do AI SOC agents decide an investigation is complete? SOAR playbooks end when the last scripted step runs. Most agentic AI SOC tools leave it to the LLM, and the agent stops when the model is satisfied. A coverage-based stopping rule replaces that judgment with criteria held outside the model, ending the investigation only when every verdict-changing hypothesis is tested and every relevant MITRE ATT&CK tactic is checked.
Q: Do AI SOC agents miss real threats? Yes, and a closed case raises no alert, so the misses stay hidden. On Simbian's October 2026 Cyber Defense Benchmark, no model clears 50% coverage on every MITRE ATT&CK tactic, the leader covers under half of each attack's steps, and Exfiltration scores lowest for most models. Regular hunts behind the agent's closed cases are how you find them.
Q: Does a high confidence score mean the investigation was complete? No. A confidence score is the model's guess that it's right, and it says nothing about which logs the agent queried. A 2026 study by Kaddour and colleagues found some agents that succeed only 22% of the time predict 77% success. Ask for the trace and the unresolved questions behind every verdict.
Q: How do you safely validate the automated workflows of AI SOC agents? Run a proof of concept on your own alerts, with your analysts approving every response. Replay alerts where the obvious explanation is wrong, run the same alert several times, check that each closed case lists what the agent skipped, and hunt behind a sample of its dismissals. Repeat whenever the underlying model changes, since newer models can stop earlier.
Pick a case your agent closed last week and find out what it didn't check.
Then Book a Demo and bring an alert your current tool closed as benign. We'll show you what our harness finds.
