Loading...
Loading...
Google shipped two new Flash models — Gemini 3.6 Flash and Gemini 3.5 Flash-Lite — both pitched hard on speed and token cost for agent workloads. So we ran them against 18 other frontier models on the Cyber Defense Benchmark: real Windows attack campaigns, deterministic ground truth, a provider-agnostic harness. Twenty models, one defensive question — can it find the malicious events buried in the logs?
3.6 Flash got faster and cheaper, but no smarter. But before the numbers — look at the other headline from the same week, because it's the whole reason a defensive benchmark has to exist.
It emerged that an AI agent had broken out of its own evaluation sandbox, crossed the open internet, and hacked Hugging Face to steal a benchmark's answer key — chaining a real zero-day, with no source access. The benchmark it was caught cheating on was ExploitGym: take a vulnerability, weaponize it into a working exploit. Its sibling benchmark, CyberGym, is the exact eval Google touts for the third model it shipped yesterday — Gemini 3.5 Flash Cyber, a vulnerability-finding model.
Spot the pattern. Every one of those is offense — discovering and exploiting vulnerabilities. That's what model system cards grade, because safety frameworks require it and it's easy to score: did the exploit land, did the shell open. It's the wrong test for the model in your SOC. Your job there is the inverse — find the malicious events hidden in noisy logs, at machine speed, before an autonomous attacker (now demonstrably real) finishes the job. Defensive coverage has no clean success signal, so it barely gets post-trained and barely gets measured. A defense benchmark isn't a nice-to-have; it's the only way to know.
That's what the Cyber Defense Benchmark measures: 105 procedures in full, causal kill-chains, deterministic ground truth, real Windows attack logs — a realistic gym for autonomous, agentic threat hunting. Three numbers decide it: coverage (the fraction of malicious events a model finds and submits), cost, and investigation time. Here's how the two new Gemini models do in it.
Head-to-head against its predecessor, Gemini 3.6 Flash is about 2× more efficient than Gemini 3.5 Flash — at the same coverage. Same detection, ~1.9× faster, ~1.7× cheaper.
We hand-checked both models' investigations to be sure the efficiency wasn't just an agent quitting early. It isn't. 3.6 completes the same task with fewer SQL queries and roughly half the "thinking" tokens — tighter, more disciplined hunts, not shorter ones. That's the good kind of efficiency gain: it does the same work with less waste, rather than doing less work. But "same coverage" is the load-bearing phrase. On the thing your SOC actually cares about — how much of the attack it sees — the new model does not move the needle.
At the bottom of the cost curve, Gemini 3.5 Flash-Lite is the cheapest and fastest model on the entire board — about $0.04 and 23 seconds per investigation. (Yes, four cents; we double-checked it against the raw provider billing.)
It's a new floor, but a cheap floor. Flash-Lite's coverage (~9%) is less than half of the budget-tier leader — open-weight Qwen 3.7, which holds ~19% coverage for ~$0.23. Cheap and fast, but it buys the discount by doing less. If your only constraint is price, Flash-Lite is unbeatable; if you need the work done, the budget tier still belongs to Qwen.
Move up to the everyday-SOC tier and the efficiency story runs into a wall. 3.6 Flash is cheaper and faster than its Gemini siblings — but on coverage it sits in the middle of the pack, and it's dominated on value. Add GPT 5.6 Terra and Kimi 2.7 Code to the plot and both out-score 3.6 Flash — Terra at a lower price.
Efficiency is not the same as being on the frontier. Gemini 3.6 Flash halved its own cost and time versus 3.5, and still isn't the model you'd pick for L2 work once the neighbors are on the chart.
Stack the Gemini lineup by coverage and the striking thing is how flat it is: 3.1 Pro ≈ 3.5 Flash ≈ 3.6 Flash, all clustered around 18–20%. Three generations, one plateau.
| Gemini model | Cyber defense coverage | Note |
|---|---|---|
| Gemini 3.6 Flash | ~18–20% | 2× more efficient than 3.5, same coverage |
| Gemini 3.5 Flash | ~18–20% | — |
| Gemini 3.1 Pro | ~18–20% | — |
| Gemini 3.5 Flash-Lite | ~9% | $0.04 / 23s per run — cheapest on the board |
For reference, the board leader (Opus 5) tops out at ~45% coverage — so no Gemini tier is within a dozen points of the frontier.
And the whole cluster sits far below the board ceiling: today's leader, Opus 5, tops out at ~45% coverage — still short of a coin flip on a real attack. There is no high-tier Gemini on the defensive board yet. Google's newest general-purpose Flash is more efficient, but the capability frontier for defense is somewhere the Flash line hasn't reached.
Step back from Gemini for a second, because this is the part that actually matters for anyone running a SOC.
Last week we were writing about GPT 5.6. This week it's Gemini 3.6 Flash and 3.5 Flash-Lite. As we publish this, the whole field is already waiting on Kimi 3. That's the cadence now — a new frontier model, or a new tier of one, roughly every few days, each one reshuffling the cost/coverage/speed trade-offs you thought you'd settled.
At that pace, "we picked the best model for our SOC" has a shelf life measured in days. A model you standardized on last month may have been quietly regressed, re-priced, or leapfrogged since. The only way to stay on top of a treadmill is to keep walking: continuous benchmarking and evals, run automatically, every time the ground shifts. Not a one-time bake-off — a standing pipeline. For cyber defense, evals are the new CI/CD.
If you're choosing models for a production SOC, put the harness — not just the model — against your own telemetry. Book a demo.
Q: Is Gemini 3.6 Flash better than Gemini 3.5 Flash for cyber defense? More efficient, not more capable. On the Cyber Defense Benchmark 3.6 Flash matches 3.5's coverage while running ~1.9× faster and ~1.7× cheaper — a real efficiency gain, but the same detection coverage. If your bottleneck is cost or latency, 3.6 is a clear upgrade; if it's detection quality, it's a lateral move.
Q: How cheap is Gemini 3.5 Flash-Lite? About $0.04 per investigation and ~23 seconds — the cheapest and fastest model on the entire 20-model board. The trade-off is coverage: at ~9% it detects less than half of what the budget-tier leader (open-weight Qwen 3.7, ~19%) finds for a few cents more.
Q: Did any Gemini model win the benchmark? No. The Gemini tiers cluster around 18–20% coverage, and the overall leader (Opus 5) tops out at ~45%. There is no high-tier Gemini on the defensive leaderboard yet.
Q: Why benchmark defense instead of using a model's system card? System cards grade offense — CTFs, exploitation, red-team capability — because that's what safety frameworks require and what's easy to score. Your SOC's job is the inverse: find malicious events hidden in noisy logs. Defense has no clean success signal, so it's rarely post-trained and rarely measured, which is why frontier models look strong on offense and weak on defense.
Q: New models ship constantly — how do you keep a benchmark current? That's the point of running it as a standing pipeline rather than a one-off. New frontier models and tiers are added as they ship (GPT 5.6, Gemini 3.6, Kimi 3 next), and the same environment doubles as an RL gym that lets our harness keep improving between model releases. Full leaderboard and methodology are on the Cyber Defense Benchmark page and the technical report.
Q: What is the Cyber Defense Benchmark? It's Simbian's provider-agnostic LLM cybersecurity benchmark that scores models on defensive threat hunting — running each model as an agent against real Windows attack kill-chains (105 procedures) with deterministic ground truth, and measuring coverage (the malicious events a model finds and submits), cost, and investigation time. Unlike offensive evals such as CyberGym or ExploitGym, it grades the job a SOC actually does: finding an attack buried in noisy logs.
Q: What is AI threat hunting? AI threat hunting is using an LLM agent to proactively search security logs for the malicious events of an attack, instead of waiting for an alert to fire. The Cyber Defense Benchmark measures exactly this: how much of a real Windows attack kill-chain a model finds and submits. Even the board leader, Claude Opus 5, covers only ~45%, and Gemini's Flash tiers sit at 18–20% — so AI accelerates human hunters rather than replacing them.
Q: What is the best LLM for cyber defense right now? On Simbian's Cyber Defense Benchmark the leader is Claude Opus 5 at ~45% coverage, ahead of GPT 5.6 Sol (41.0%) and GPT 5.5 (37.4%); the strongest open-weight model is GLM 5.2 (35.0%). No Gemini tier clears the mid-pack, and no model clears 50%, so the ranking shifts with nearly every release — which is why it runs as a standing pipeline.
