Loading...
Loading...
Alert triage automation is the one job independent testing says AI SOC agents actually do well, scoring 98% on a labelled alert set. Full incident analysis is another matter, at 8 of 25 solved end to end. Four failure modes recur across eleven models, and no vendor page names them.
Google Cloud, Microsoft, CrowdStrike, EY, Prophet Security, Dropzone, Anomali, Deepwatch and Stellar Cyber all publish a page explaining the agentic SOC. I read the ones I could fetch in full, in September 2026. Prophet, CrowdStrike and Anomali source every claim to their own products. So does Microsoft's page, which carries two corporate vice-president bylines. Stellar Cyber does cite outside sources, though they are frameworks and market research, namely MITRE ATT&CK, NIST SP 800-207 and a Gartner quadrant.
On the pages I could fetch, none publishes a failure rate.
So the definition is settled and the failure data is silent. Buyers take the first on faith and the second on trust, because there isn't anything else to take it on. Two 2026 benchmarks broke that silence. One of them let the agent drive real forensic tools against live evidence. The architecture argument lives at why the model is the easy part.
SIABench, published in March 2026 by Concordia University and Defence Research and Development Canada, tested eleven models across 25 full incident scenarios, 229 investigative questions and 135 alert-triage scenarios, with the agent driving live forensic tools itself. The authors describe it as the first systematic evaluation of eleven major LLMs on security incident analysis.
The agent had to run Volatility, Tshark and Oledump on a real command line, read what came back, and decide what to run next. Scenarios are drawn from the material used to train junior security operations center analysts. They are curated. The tooling is production-grade. Results split hard along one seam.
Alert triage is the work of sorting incoming security alerts into real threats and noise, historically a human reading a queue. Alert triage automation hands that work to AI SOC agents, which investigate every incoming alert, close the benign ones and escalate the real threats, with no playbook authored in advance for that alert type. Automated alert triage of the older kind ran on rules; AI alert triage reasons from the evidence each query returns. It replaces the SOAR model, where a human writes a branch for each scenario and everything unanticipated escalates to a senior analyst.
On SIABench's triage sets it works. GPT-5 reached 98% on TII-SSRC-23, calling 48 of 50 true positives and all 50 false positives correctly. Claude-4.5-Sonnet reached 96%. That's the job most AI SOC agents are sold to do, and the measurement supports the pitch.
Full incident analysis tells a different story. Claude-4.5-Sonnet led at 8 of 25 scenarios solved end to end. GPT-5, the strongest triage performer, managed 5. On the single hard memory-forensics scenario, no model in the benchmark produced a complete solve, and the best answered about half of its fifteen questions. Defense Evasion was the weakest MITRE ATT&CK tactic across the top four models at 51.28% to 74.07%, plausibly because evasion rewards reverse-engineering judgment over speed.
One result deserves more attention than it has had. On the second triage dataset, CIC-IDS2017, GPT-5 posted 97.1% overall while its true-positive rate sat at 80% — four of five, on a five-alert true-positive sample. The metric carries one bit of resolution. A headline percentage computed over thirty-five alerts, thirty of them benign, says almost nothing about whether real threats get caught. Ask for recall, precision, and the denominator under each.
The researchers put it bluntly: "Rushing adoption risks overlooking LLM effectiveness in SIA tasks and may ultimately degrade security rather than improve it." They place these systems at the tier of work junior analysts handle today, and note that higher-tier work leaves limited scope for full automation.
AI SOC agents fail in four measured ways: they build on their own wrong answer, quit an investigation early, contain a host before gathering evidence, and fabricate findings outright. Only 29% of teams feel they can deploy agentic AI securely, on Cisco's 2026 State of AI Security, and the failure data below suggests that caution is well calibrated.
Model choice moves the give-up rate sixtyfold and the wrong-answer rate fivefold. For a SOC team, picking the right one pays more than any other decision on this page.
| Model (SIABench, 2026) | Full incidents solved (of 25) | Triage, TII-SSRC-23 | Gave up (% of questions) | Wrong answer (% of questions) |
|---|---|---|---|---|
| Claude-4.5-Sonnet | 8 | 96% | 1.61% | 11.47% |
| GPT-5 | 5 | 98% | 0.44% | 14.94% |
| Claude-3.5-Sonnet | 4 | Not evaluated | 7.4% | 23.6% |
| GPT-4o | 3 | Not evaluated | 25.8% | 19.2% |
| DeepSeek-Reasoner | 1 | 89% | 6.6% | 43.2% |
| Gemini-1.5-Pro | 1 | Not evaluated | 26.2% | 35.8% |
| o3-mini | 1 | Not evaluated | 0.9% | 58.1% |
| GPT-4o-mini | 1 | 75% | 24.9% | 46.3% |
| Llama-3.1-405B | 0 | Not evaluated | 11.4% | 42.62% |
| Llama-3.1-70B | 0 | Not evaluated | 9.2% | 40.17% |
| Llama-3.1-8B | 0 | Not evaluated | 8.3% | 23.1% |
| SIABench ran alert triage on four models only. Give-up and wrong-answer rates are percentages of all 229 investigative questions, not shares of failures. |
Cascading error is the failure mode SIABench ranks first most often.
One wrong answer early becomes the input to every step after it. The paper calls this question dependency and ranks it the number-one failure cause for seven of the eleven models tested.
For a SOC team, that's the expensive one. The output still arrives looking like a finished alert investigation, with a narrative, a verdict and a confidence score. The error sits four pivots back, in step three.
AI SOC agents abandon investigations at wildly different rates. Gemini-1.5-Pro abandoned 26.2% of all questions and GPT-4o 25.8%. On the answer side, o3-mini returned a wrong answer on 58.1% of questions and GPT-4o-mini on 46.3%. Claude-4.5-Sonnet sat at 1.61% and 11.47%.
On wrong answers alone, o3-mini and Claude-4.5-Sonnet differ by a factor of five.
AI SOC agents contain hosts before they have checked the evidence. A January 2026 preprint, OpenSec, fed incident response agents deliberately misleading evidence and measured what they did next. It is single-author work from a commercial AI company, run over 40 episodes per model, so weigh it accordingly. Every model identified the real threat when it acted. Restraint was the problem.
Across four frontier models, only 37.5% to 54.2% of containment actions followed a fetch of trusted evidence about the target. Between 45% and 82.5% of episodes contained at least one incorrect containment action. One model contained something in 100% of episodes, by the fourth step on average; another waited until step ten and roughly halved its false-positive rate.
The threat gets caught either way. How much of your estate gets caught alongside it lands on your change-freeze policy and your on-call rota.
AI SOC agents hallucinate entire investigations. The SIABench authors record models that "fabricate answers or create fictional simulations without any actual evidence." An investigation that never happened, written up as though it did.
Evidence traces stop being optional at that point. You should be able to see which query returned what, and run it again.
No. The harness is the scaffolding around the model that manages tool output, execution state and reasoning between actions. Adding a summarizer module alone moved one model by 12 to 32 points. Changing the model moves them further. Partial solving runs from 7.03% on Llama-3.1-8B to 81.70% on Claude-4.5-Sonnet, and full solves from zero to 8 of 25.
A summarizer module managing verbose tool output moved Claude-3.5-Sonnet by 12.52 to 31.93 points across four task families; without it that model hit context-limit errors on seven tasks, and GPT-4o left 11 of 25 scenarios unfinished. Multi-state execution lifted Claude-3.5-Sonnet 24.1 points on memory forensics. GPT-5 gained 13.95. Reasoning between actions was worth another 23.4 points for Claude-3.5-Sonnet and 19.85 for GPT-4o.
Simbian has argued that the harness is the product. On this evidence that argument is half right, and the half it gets wrong matters. The authors found scaffolding mattering less as models improved, writing that "newer and more advanced models may require progressively less scaffolding." The summarizer made GPT-5 slightly worse on two of four task families. If that trend holds, the engineering advantage any AI SOC vendor holds today shrinks with every frontier release, and we have no data showing ours is exempt.
Vendors build those thirty points. They rent the frontier model underneath, and no comparison page publishes the difference.
Simbian's AI SOC Agent aims its harness at the failure data directly. Cascading error has the widest blast radius in the dataset, so a critique skill reads the main Agent's work while the investigation is still running and pushes corrections back mid-flight. Context Lake™ holds what is true about a specific environment, consulted during the investigation, so one entry covers a whole pattern and nobody maintains a playbook of per-address exceptions.
Across 300+ enterprise environments, 95% of the responses Simbian's AI SOC Agent proposes are approved by the customer's own analysts. Their call, not ours. In the other 5%, the Agent gets additional context that improves the next response. Self-improving, not self-driving.
Evaluate AI SOC agents by replaying alerts your team has already closed and scoring the agent against your own dispositions. Vendor benchmarks run on vendor data, which isn't your data. Five measures are cheap to instrument in your SIEM or EDR, whether that is Splunk, Microsoft Sentinel or CrowdStrike Falcon:
Note what is missing from that list. MTTR is the easiest number for a vendor to move and the least informative once an agent runs 24/7, because an agent that closes fast and wrong beats your analysts on every dashboard you own.
Then ask any vendor for the give-up rate and one investigation that ended inconclusive. A good senior analyst says "I do not have enough evidence" most weeks. Watch whether the agent ever does.
Q: How good is AI SOC agent alert triage? The strongest model in SIABench scored 98% on the TII-SSRC-23 alert set and 97.1% on CIC-IDS2017, but its true-positive recall on the second set was 80% across only five true positives. Headline scores and recall diverge sharply on imbalanced alert data, so ask any vendor for both figures and the sample size behind them.
Q: What is the difference between an AI SOC agent and SOAR? SOAR executes a playbook a human wrote in advance and stalls when an alert matches no branch, so coverage stops where the author's imagination stopped. An AI SOC agent chooses each investigative query from what the last one returned and reaches a verdict with no playbook for that alert type. Independent 2026 testing scored the best triage performer at 98%, and the best full-incident performer, a different model, at 8 of 25.
Q: Can AI SOC agents run a full investigation on their own? Not on the current evidence. The best model in independent 2026 testing solved 8 of 25 full incident scenarios end to end, and the researchers place these systems at the tier of work junior analysts handle, noting that higher-tier work leaves limited scope for full automation.
Q: Do AI SOC agents contain hosts without checking evidence first? Often, yes. The January 2026 OpenSec study found that across four frontier models, only 37.5% to 54.2% of containment actions were preceded by fetching trusted evidence about the target, with false-positive rates from 45% to 82.5%. Ask any vendor what share of its containment recommendations follow an evidence fetch.
Q: Do AI SOC agents hallucinate during investigations? Yes, and the independent research documents it directly, including models that fabricate answers or construct fictional analysis with no underlying evidence. That is the argument for an evidence trace on every verdict: you should be able to see which query returned what, and reproduce it.
Q: How do you evaluate an AI SOC agent? Replay alerts your team already closed and score the agent against your own dispositions, then measure five things: agreement with your analysts, recall and precision with their denominators, whether containment followed an evidence fetch, whether the agent knows when to stop, and how it behaves after a wrong first pivot.
Alert triage automation is measured. GPT-5 scored 98% on the TII-SSRC-23 alert set and Claude-4.5-Sonnet 96%, and that is the tier worth running with a supervisor on the queue. Full incident ownership runs to 8 of 25 scenarios, Claude-4.5-Sonnet's result and the best anyone has published. The vendors publishing definitions of this category publish neither number.
So take twenty alerts your team closed last month and replay them. Then ask the vendor to pick twenty of their own, and to show you the one their agent refused to call. When you want to run that against an Agent that shows its work, Book a Demo and bring your own alerts.
It hasn't earned the keys.
