Loading...
Loading...
While everyone debated how long open-weight models would take to catch the frontier labs, one of them just beat every publicly available model in blind human evaluations on web development. Kimi K3 (Moonshot) is #1 on WebDev Arena — decided by hundreds of thousands of blind, head-to-head human votes — over Claude Fable 5 and GPT 5.6 Sol Max.
That's a genuinely impressive result. But web coding isn't what a SOC runs on. So we did what we do with every new frontier model: we ran Kimi K3 on the Cyber Defense Benchmark — real Windows attack kill-chains, deterministic ground truth, a provider-agnostic harness, fully agentic threat hunting — against 21 other models. On defense, the web king lands #7 of 22 at 33% coverage. The gap between those two results is the whole point of this post.
On coverage — the fraction of malicious events a model finds and submits across a full attack kill-chain — Kimi K3 lands at #7, behind the frontier (Opus 5 at 45.1%, GPT 5.6 Sol at 41.0%) and, notably, behind open-weight leader GLM 5.2 at 35.0%.
| Rank | Model | Coverage | Type |
|---|---|---|---|
| 1 | Claude Opus 5 | 45.1% | Proprietary |
| 2 | Opus 4.6 | 44.5% | Proprietary |
| 3 | GPT 5.6 Sol | 41.0% | Proprietary |
| 4 | GPT 5.5 | 37.4% | Proprietary |
| 5 | Sonnet 4.6 | 36.9% | Proprietary |
| 6 | GLM 5.2 | 35.0% | Open-weight |
| 7 | Kimi K3 | 33.0% | Open-weight |
What's interesting is why it lands there — because it isn't a capability ceiling. When we hand-checked the investigations, Kimi K3's individual queries were the sharpest any Kimi has produced. The problem is that it stops working before the job is done.
If that pattern sounds familiar, it should. It's the same low-effort dip we flagged in Opus 4.7 (30.9%) and Opus 4.8 (32.7%) — capable models that under-invested and landed mid-pack — a dip Anthropic then fixed in Opus 5. Kimi K3 has the query quality to place much higher; it just talks itself into leaving the budget on the table.
Zoom out to the Kimi line and there's real, steady progress on defense: 13.4% → 26.7% → 33.0% from Kimi 2.6 to 2.7 Code to K3. K3 is the best defensive Kimi yet, comfortably ahead of its predecessors.
But hold that curve next to the WebDev Arena result and the contrast is stark. On web, K3 didn't just improve generation-over-generation — it leapt to #1 in the world, past every closed frontier model. On cyber defense, it improves but stays firmly mid-pack, still a dozen points under the frontier ceiling.
Why does the same model dominate one domain and plateau in the other? Because web has a built-in verifiable-reward loop and defense doesn't. Build the code, screenshot it in a browser, check it against the intent, redo if it's wrong — the reward is objective, and reinforcement learning with verifiable rewards (RLVR) scales beautifully against it. That's the recipe that pushed K3 to the top of web.
Frontier labs have no such gym yet. The RLVR environments that exist today are almost entirely offensive — CTF and Hack-The-Box style, where "did you capture the flag?" is trivially checkable. Defense is the hard case: verifying whether an agent found every malicious event buried in noisy logs is genuinely difficult, and it's barely been built. So models keep getting graded on offense and rewarded for offense — and it shows up as exactly this kind of plateau on the defensive job.
If you're choosing an open-weight model to run defensive work today, Kimi K3 isn't it — GLM 5.2 wins on all three axes that matter.
| Metric | GLM 5.2 | Kimi K3 |
|---|---|---|
| Coverage | 35.0% | 33.0% |
| Cost per investigation | $2.12 | $2.43 |
| Investigation time | ~8 min | ~23 min |
GLM 5.2 delivers higher coverage (35.0% vs 33.0%), costs less per investigation ($2.12 vs $2.43 at Standard list pricing), and finishes faster. Kimi K3 currently runs ~23 minutes per investigation, which may be a symptom of high demand — we started benchmarking as soon as the model was released.
But zoom out from the head-to-head and there's a trend for open weights: the gap to proprietary is closing on defense. Both GLM 5.2 (35.0%) and Kimi K3 (33.0%) now edge out Opus 4.7 (30.9%) and Opus 4.8 (32.7%). And this fits a pattern we've documented before — the offense-vs-defense asymmetry. On offensive cyber tasks, US frontier models lead the open-weight frontier by 3.1× (UK AISI/CAISI, safeguards off). On defense, that same lead is just 1.24×. Defense is exactly where the frontier's advantage is thinnest.
The catch: closing the last of that gap won't come from bigger general-purpose models. Kimi K3 is proof — a 2.8-trillion-parameter web champion that still plateaus on defense. It'll take dedicated investment in RL training and verifiable-reward gyms built for defense, the same recipe that made K3 win the web.
The recipe that made Kimi K3 win the web is exactly what defense is missing: reinforcement learning against a verifiable-reward gym. That's what we build. The Cyber Defense Benchmark you've just read is the evaluation instantiation of that gym — a scalable, deterministic environment of real attack kill-chains with ground truth. It's precisely what lets us test any LLM's defensive skill today; run in reverse, the same environment generates the verifiable rewards to train one. We offer that as post-training data to fine-tune LLM defensive capabilities — the RLVR gym for defense that doesn't otherwise exist.
This isn't new ground for us. A year ago we published the first AI SOC LLM benchmark — the earliest public attempt to grade LLMs on real defensive SOC work instead of offensive CTFs. The Cyber Defense Benchmark is the mature form of that thesis, and it's why we can say with data — not vibes — that a 2.8-trillion-parameter web champion still catches only a third of a real attack.
If you're training an LLM for defense, or choosing one to run in your SOC, reach out. We have the gym, the training data, and a year's head start on the exact problem this post keeps circling back to.
Q: Is Kimi K3 good at cyber defense? It's the best Kimi model yet (33% coverage, up from 26.7% for Kimi 2.7 Code), but it lands #7 of 22 on the Cyber Defense Benchmark — behind the frontier (Opus 5 45.1%, GPT 5.6 Sol 41.0%) and behind open-weight leader GLM 5.2 (35.0%). Its individual queries are strong; it just quits most hunts early.
Q: Kimi K3 is #1 on WebDev Arena — why isn't it #1 on defense? Because web has a verifiable-reward loop (build → screenshot → check → redo) that RL with verifiable rewards scales against, and defense doesn't have an equivalent gym yet. General web/coding capability doesn't transfer to catching intrusions in noisy logs — which is exactly why a defense-specific benchmark exists.
Q: Which open-weight model is best for a SOC — Kimi K3 or GLM 5.2? GLM 5.2. It beats Kimi K3 on coverage (35.0% vs 33.0%), cost ($2.12 vs $2.43 per investigation, Standard list pricing), and speed. Both open-weight models now edge out Anthropic's Opus 4.7 and 4.8 on defense, so the proprietary lead here is thin — just 1.24×, versus 3.1× on offense.
Q: Why did Kimi K3 score lower than its query quality suggests? It quit early on 16 of its 26 hunts — averaging 42 of a 50-query budget — declaring the investigation "complete" while it had found only ~5% of the malicious events. It's a persistence and calibration problem, the same low-effort dip we saw in Opus 4.7 / 4.8 and that Opus 5 corrected. Nudging it in-harness to keep going recovers much of the gap.
Q: How is coverage measured? Coverage is the fraction of malicious events across a full, causal attack kill-chain that the agent both finds and submits — 105 procedures, real Windows attack logs, deterministic ground truth. Full methodology is in the technical report and on the Cyber Defense Benchmark page.
Q: Is Kimi K3 open source? Yes — Kimi K3 is an open-weight flagship from Moonshot AI, so you can download and self-host the weights, which is why it runs in the open-weight tier of the Cyber Defense Benchmark. On defense it lands #7 of 22 at 33% coverage, behind fellow open-weight model GLM 5.2 (35.0%) — so being open doesn't yet make it the best open-weight pick for a SOC.
Q: What is the Cyber Defense Benchmark? It's Simbian's provider-agnostic LLM cybersecurity benchmark that scores models on defensive threat hunting — running each model as an agent against real Windows attack kill-chains (105 procedures) with deterministic ground truth, and measuring coverage (the malicious events a model finds and submits), cost, and investigation time. It grades the job a SOC actually does — finding an attack buried in noisy logs — instead of the offensive CTF tasks most evals score.
Q: What is AI threat hunting? AI threat hunting is using an LLM agent to proactively search security logs for the malicious events of an attack, instead of waiting for an alert to fire. The Cyber Defense Benchmark measures exactly this: how much of a real Windows attack kill-chain a model finds and submits. Even the board leader, Claude Opus 5, covers only 45.1% and Kimi K3 catches ~33%, so AI accelerates human hunters rather than replacing them.
