On-demand webinar · 48 min
The LLM Security Test:
Three Open-Weight Models Beat Claude.
GLM-5.2, DeepSeek 4 Pro and Kimi K3 each beat Claude Opus 4.8 on 100,000 real attack events. Not one of the 27 models tested reached the pass mark. Watch the full session for the scoreboard, and the selection test to run before your next model decision.
Ambuj Kumar
Founder & CEO
Alankrit Chona
Chief Technology Officer
Watch the recording
Open weight models vs. frontier, scored on defense
27 models dropped into 100,000+ real attack events with no alert, no hint and no question. Scored on how much of a multi-stage intrusion each one reconstructs.
| Rank | Model | Provider | Attack-chain coverage | Median cost per hunt |
|---|---|---|---|---|
| 1 | Opus 5 | 45.1% | $3.76 | |
| 2 | Opus 4.6 | 44.5% | $2.71 | |
| 3 | Grok 4.6 | 41.3% | $1.58 | |
| 4 | GPT 5.6 Sol | 41.0% | $4.03 | |
| 5 | Grok 4.7 | 37.6% | $1.44 | |
| 6 | GPT 5.5 | 37.4% | $4.50 | |
| 7 | Sonnet 4.6 | 36.9% | $2.21 | |
| 8 | GLM 5.2 | 35.0% | $2.01 | |
| 9 | DeepSeek 4 Pro 0813 | 34.0% | $0.97 | |
| 10 | Kimi K3 | 33.0% | $2.38 | |
| 11 | Opus 4.8 | 32.7% | $1.68 | |
| 12 | Opus 4.7 | 30.9% | $1.65 |
11 more models
All 27 models scored on attack-chain coverage and median cost per hunt, weights marked open or closed — on the public Cyber Defense Benchmark.
View the Full LeaderboardCoverage is the mean fraction of the attack chain reconstructed across 1,132 runs. Cost is median USD per hunt. Passing requires 50%+ coverage on every one of 13 MITRE tactics; the leader clears it on 7. Source: arXiv 2604.19533.
What the LLM security leaderboard shows
1,132 runs against deterministic ground truth, no LLM-as-judge. The LLM cybersecurity benchmark behind arXiv 2604.19533.
- 0Of 27 models passed
- 3Open-weight models above Opus 4.8
- 100K+Attack events per environment
How to score open weight models for the SOC
Why the intelligence index misleads
Qwen 3.8 27B and GPT-5.6 Luna both score 52 on Artificial Analysis. On real attack logs, one reconstructs a third less than the other. General capability predicts nothing here.
The defensive test you can run in a week
What to substitute for the leaderboard score everyone quotes, using your own logs and the models you are already paying for.
Where self-hosting triples the bill
Per-token math is what makes a local model look cheap. Price per investigation instead and the reasoning tokens surface immediately.
Score coverage, not confidence
Every model tested called its investigation finished before exhausting the hypothesis space. Several did it while still under their query budget.
Frontier took the podium. Open weight took everything else.
GLM 5.2, DeepSeek 4 Pro and Kimi K3 land 8th, 9th and 10th of 27, ahead of Opus 4.8, Opus 4.7, Sonnet 5 and GPT-5.6 Luna.
The open vs. closed leaderboard, yours to keep
27 models scored on attack-chain coverage and median cost per hunt, weights marked open or closed, sourced to arXiv 2604.19533 and built for a procurement conversation.
Get the open vs. closed LLM security leaderboard
Recorded 29 September with the team behind the Cyber Defense Benchmark. Bring the model shortlist you are about to sign off.