Train models that can actually defend.
Why frontier LLMs fail at cyber defense.
Offense is saturated and recall-heavy. Defense is abductive: you reconstruct an unknown attacker's intent from noise. Almost none of that reasoning exists in pretraining.
A breach is a path hidden in a haystack of benign logs. Models go blind exactly where it matters.
Offense is mostly solved
Deductive, recall-heavy work: exploits, code, CTFs. The web is full of signal, and models already score well.
Defense is different in kind
Abductive. You reconstruct an unknown attacker's intent from a haystack of benign-looking logs, and almost none of that reasoning lives in pretraining.
It's a data + environment problem
You can't read defensive reasoning off the web, and you can't crowdsource it. It has to be grown with RL against verifiable rewards, which needs an environment that can score an investigation right or wrong.
No frontier model passes the Cyber Defense Benchmark.
The leader stalls at 44.5% MITRE ATT&CK coverage against a 50% passing bar. The category is wide open.
frontier models pass the Cyber Defense Benchmark
best model's MITRE ATT&CK coverage (passing bar: 50%)
MITRE ATT&CK tactics the benchmark measures, end to end
The shape collapses on the tactics that define a breach — lateral movement, credential access, exfiltration.
Every bar; not one crosses the 50% line.
Offensive benchmarks saturate in months. This one stays unsolved, and every model still goes blind where real breaches happen.
Training data, RL environments, and evaluations.
Closing the gap takes three things a lab can't build alone: verified security trajectories, the Holodeck RL environment, and the published Cyber Defense Benchmark.
Verified security trajectories
Real investigations and attack chains, decomposed into gradeable tasks across threat hunting, detection engineering, triage, and offensive security. Every unit ships with verified ground truth, machine-checkable wherever the task allows.
See the task taxonomy →Holodeck
A deterministic, Gymnasium-compatible gym where a base model learns to hunt and is scored on machine truth.
How Holodeck works →The Cyber Defense Benchmark
The published eval, formalized as your north-star metric. It resists memorization and shows exactly where a model fails.
Read the benchmark →We build data per skill, not per use case.
Each use case breaks into discrete skills we can grade independently.
Threat Hunting
- hypothesis generation
- query generation
- evidence correlation
- lead / pivot expansion
- attack attribution
- timeline reconstruction
Detection Engineering
- detection authoring
- coverage-gap analysis
- false-positive tuning
- rule validation
- data-source mapping
- detection-as-code review
Triage & Investigation
- alert enrichment
- prioritization
- root-cause analysis
- disposition (TP / FP)
- response actions
- escalation
Offensive / Pentest
- recon
- exploitation
- privilege escalation
- lateral movement
- impact
- reporting
Each category is scored independently, so we produce targeted data and measure a model skill by skill. It extends to tool-use and integration trajectories (agents driving security connectors) for agentic data, not just text.
Holodeck: a deterministic RL environment with verifiable rewards.
A Gymnasium-compatible gym that scores security investigations against machine truth. No reward model to drift, no LLM judge to game.
Any base model drops in
A Gymnasium-compatible interface: observe (threat briefing + prior results) → act (query / submit / give-up) → reward. No bespoke harness to build.
Verifiable reward
In Holodeck, ground truth is deterministic, so the reward is exact and unfakeable. You can't game it the way you can a learned reward model.
Real attacker behavior, modeled at org scale
Deterministic replay with seeded mutation. Same seed, identical run. Reproducible and reusable, like a versioned artifact you check into your stack.
Compatible with the RLVR (reinforcement learning with verifiable rewards) toolchain your team already uses. Tasks, tools, and reward functions ship as first-class, verifiable artifacts.
Where judgment is needed, bring expert human validation, not crowd labels.
Deterministic rewards cover the machine-checkable tasks. The judgment calls they can't cover go to people. Our network of career SecOps analysts (threat hunters, detection engineers, and incident responders with decades in the field) reviews and grades your tasks and model outputs.
Grade model outputs on real security tasks
Scored by analysts who run these investigations in production.
Preference & RLHF data
Expert rankings of competing outputs, for reward modeling.
Judgment-task ground truth
The calls that can't be mechanically verified: severity, escalation, reasoning quality.
Held-out human validation
An independent, human-graded set to catch reward hacking (anti-Goodhart).
Holodeck automates the verifiable reward. Human validation covers what it can't. Priced hourly, for deep security expertise, not general crowd work.
Generalist data vendors staff doctors, lawyers, and bankers. We staff the people who actually run security operations.
You can't crowdsource security data. We build it.
Grounded in real operations
Our attack and organization models are built on real attacker telemetry, informed by operating security through the world's largest MDRs (managed detection and response providers). Not fabricated from scratch, and not crowd-labeled.
Verified & clean to license
Every trajectory carries verified ground truth; the artifact is our own IP. The moat is the modeling and the verification, not hoarded data.
Built by people who've shipped both security and infrastructure
Founders from Fortanix, NVIDIA, and Twitter; authors of the Cyber Defense Benchmark.
Built to slot into your post-training stack.
JSONL and verifiers / Inspect-compatible tasks, Gymnasium-compatible environments, and RLVR-ready rewards. Nothing bespoke to build.
Two ways to work with us.
Data provider
Verified security training data, the Holodeck RL environment, and the Cyber Defense Benchmark. You train on it.
Human validation
Our network of career SecOps analysts grades your tasks and model outputs: the judgment calls a machine can't verify.
Questions from research teams
Bring a base model. We'll measure the lift together.
A bounded first experiment: baseline on the Cyber Defense Benchmark, train in Holodeck, measure the delta.