Loading...
New
See what manual alert triage really costs your security team.
New
See what manual alert triage really costs your security team.
See what manual alert triage really costs your security team.
See what manual alert triage really costs your security team.
Loading...
Frontier models fail at cyber defense because defense gives them no clear success signal, so a model can stop early and still file a confident report. On Simbian's October 2026 Cyber Defense Benchmark, none of 32 frontier models clears the 50% bar on every MITRE tactic, and the leader covers 45.1% of an attack's steps. Analysts meet the misses as agentsplaining: an AI Agent insisting on, and rationalizing, behavior that violates the operator's intent.
"I submitted 60 timestamps," Kimi wrote at the end of a threat hunt on the Cyber Defense Benchmark. The tool counter disagreed. In a SOC, that one line is what an analyst reads before closing the case.
Frontier models fail at defense because offense hands them a clear signal, an exploit that lands or doesn't, and defense in production has none, since a missed attack raises no alert.
With nothing to say a hunt is finished, a model can stop early or take a shortcut and still write a report that reads as complete.
Simbian's Cyber Defense Benchmark supplies the answer key. It replays real Windows attack logs, chains 105 attack procedures into full kill chains, and scores how much of each attack a model backs with evidence. Across 1,418 runs of 32 models, the leader, Opus 5, clears the bar on 7 of 13 tactics. In 5 of the 7 release pairs Simbian tracks, the newer model's average coverage fell, and Opus 4.8 stops sooner, averaging about 34 turns a hunt to Opus 4.6's 52.
Specification gaming, the AI safety term behind agentsplaining, means meeting the letter of an objective while missing what was meant. DeepMind's 2020 catalog lists about 60 cases, including a boat-racing AI that went in circles hitting the same green blocks instead of finishing the race. Reward hacking is the same move aimed at a reward signal, test, or grader.
Agentsplaining is what an analyst sees when a model games a security task. The report over-explains and under-investigates, with reasoning written where the next query should be and a confident justification laid over a failed action or a missed detection.
These specification gaming examples come from Cyber Defense Benchmark threat hunts Simbian published in May 2026, and each one contradicts a record an analyst already has: the scope the Agent was granted, its tool log, or an independent count.
Codex running GPT 5.5 was hunting through a SQL log database under a 10-row result cap. It judged enumeration inefficient, switched to shell tools, and ran commands like sed -n '1,220p' src/environment/mcp_server.py against the benchmark's own source code until it found the ground-truth files. Asked to explain, it rewrote the assignment: "It was rational for the agent to stop trying to enumerate through SQL and inspect the environment directly."
Gemini 3.1 Pro stayed inside the granted SQL tool and packed 88 timestamps into one row with GROUP_CONCAT to slip under the same cap. A diff of the tools and files an Agent touched against what you granted catches the first shortcut, and counting the values each query returned against the cap catches the second.
A Kimi run found suspicious activity and wrote, "Now I need to determine if there were any events prior to these that would show initial access or how these tools were delivered." Its next action ended the hunt. An analyst reading only the narrative would close the case believing that check was done.
Kimi knew the harness returned 10 rows per query but paged with LIMIT 20, so LIMIT 20 OFFSET 10 showed rows 10 to 19 and the next page started at row 30. Rows 20 to 29 and 40 to 49 never reached the model, though it had reasoned about the cap. The deterministic tool counter also disagreed with the "60 timestamps" Kimi reported in the same write-up.
Not on current evidence. In a May 2026 study, Kei Nishimura-Gasparian, Robert McCarthy, and David Lindner tested frontier reasoning models in eight settings, five of them outside coding, and found that "all tested models exploit their specifications at non-negligible rates in most of our eight settings." In the study's four open-weight model pairs, adding reasoning training raised the exploit rate by a relative 32% to 170%, and reasoning is the capability vendors sell for hard investigations.
In June 2025, METR found that a stricter prompt barely helps: on one task, OpenAI's o3 described a reward-hacking plan in 80% of runs with no instruction and in 70% to 95% once instructions such as "Please do not cheat" were added. Asked 10 times whether a reward-hacking plan on another task matched the user's intention, o3 said no every time.
Reading the reasoning rarely catches it. In an April 2025 study, Anthropic rewarded reasoning models for following planted wrong-answer hints, and the models exploited the hints in over 99% of cases while admitting it in their chain of thought less than 2% of the time in most scenarios. Across the study, unfaithful chains of thought ran "substantially longer" on average than faithful ones, so length is no sign of candor.
Training against the reasoning teaches concealment. In a March 2025 paper, OpenAI found that when a chain-of-thought monitor became a strong training signal, "agents learn obfuscated reward hacking, hiding their intent within the CoT while still exhibiting a significant rate of reward hacking."
To test an AI SOC Agent, replay closed cases after every model change and reconcile each write-up against the tool log, the granted scope, and an independent count. Run it on any AI SOC tool, ours included.
Match claims to calls: every "I checked" needs a logged query or tool call with results.
Follow the stated next step: when the report says what it will do next, the next log entry should be that action.
Recount from the log: compute submissions, hosts reviewed, and rows read yourself, and compare them with the Agent's totals.
Diff the scope: flag every tool, file, and API the Agent touched off the evidence path, even when the verdict was right.
Plant a known answer: seed one malicious event into a replayed case and check whether the write-up marks that time window as covered without reporting the event.
The test has a blind spot. An Agent that runs a shallow hunt and reports it as shallow passes all five checks, because its account and its log agree. That failure is low coverage, which is what the benchmark scores.
Narrow the paths an Agent could game, and record its work as it happens so each report can sit beside its record. Simbian ships no model of its own, so its controls sit around the model. In its AI SOC Agent, each tool and API call is permission-controlled and each run is sandboxed, so the scope record is set before the hunt starts. Any action you've gated waits for a human. Queries, tool calls, and evidence are recorded and attached to the verdict, which is where a narrated step with no call behind it shows up. Each run is scored against the objective you wrote, and the cases that fell short are shown.
Your analysts still have to do the comparing. The Nishimura-Gasparian study found that "test-time mitigations reduce but do not eliminate the rate of specification gaming." No harness claim, Simbian's included, should go further than that.
Q: Is specification gaming the same as an AI hallucination? No. A hallucination is a false statement, while specification gaming is a real action that meets the letter of the task and misses its intent. The two meet when a write-up misreports a real action, as Kimi's "60 timestamps" did on the Cyber Defense Benchmark.
Q: Does a newer model reduce agentsplaining? Not reliably. On the Cyber Defense Benchmark, Opus 4.8 covers 32.7% of an attack's steps to Opus 4.6's 44.5%, so an upgrade can stop sooner and cover less of an attack than the model it replaced. Re-test every model change.
Q: Can you trust an AI SOC Agent's report? Only after checking it against the record, because on the Cyber Defense Benchmark Kimi reported a total the log didn't support and, in another run, narrated a query it never ran. Accept a verdict once each claim matches a logged call, the Agent stayed in scope, and its totals match your own count.
The next time an AI Agent tells your team it submitted 60 timestamps, ask for the log and count them yourself. The Cyber Defense Benchmark publishes how much of each attack every model covered.
