Loading...
Loading...
On cybersecurity defense, GPT 5.6 Sol is the best model OpenAI has ever shipped, and it still fails. Simbian's Cyber Defense Benchmark runs 20 LLMs — GPT 5.6 Sol, Terra, Luna, and 17 others — through 1,002 investigations of real attacks across all 13 MITRE ATT&CK tactics. Sol finishes second at 41% coverage, and no model clears the 50% pass mark on every tactic. To catch an attacker loose in your logs, you can't use the LLMs powering ChatGPT alone.
OpenAI shipped GPT 5.6 on July 9 and pitched its top-tier model, Sol, as the most capable cybersecurity AI it has released. That is a big claim, so we tested it. Every frontier model that ships gets run through the Cyber Defense Benchmark (arXiv 2604.19533): the agent is dropped into a haystack of 100,000-plus security events and scored on how much of a real, multi-stage attack it can reconstruct from the logs. It is the closest thing we have to the job a SOC analyst or threat hunter does at two in the morning. GPT 5.6 Sol is the best any OpenAI model has ever done on it. It also joins every other model in failing it.
▶ Watch the webinar — Why LLMs Fail in the SOC A 30-minute walk through the Cyber Defense Benchmark: how every frontier model is scored, the per-tactic breakdowns, and why no LLM clears 50% coverage on real attack logs.
The launch coverage makes GPT 5.6 look like a real jump on security. But everything that actually got measured is offense. On ExploitBench, the security firm Irregular found Sol a little sharper than GPT 5.5 while it kept choking on well-defended targets and couldn't carry an attack from start to finish. Turned loose on live Chromium and Firefox code, it spotted bugs but never chained them into a working exploit. The U.K.'s AI Safety Institute then showed that even that restraint jailbreaks away. Every one of those tests asks the same question: can the model find a weakness and weaponize it?
Defending a SOC is the mirror image of that, and it is the harder half. You don't start with a target. You start with noise, tens of thousands of events that are almost all boring, and you have to prove an attacker is hiding somewhere inside them. That means linking a credential-access event to the lateral movement that follows it three hours later, then telling the real intrusion apart from ten thousand false alarms that look exactly like it. Finding a bug in a quiet lab and catching an adversary loose in a live network are not the same skill, and the second one is what a security team is actually paying for. It is the skill the Cyber Defense Benchmark measures.
The benchmark does not grade on a curve. Twenty frontier models, 1,002 investigations, more than 100,000 events apiece, every model reasoning through real Windows attack telemetry with SQL queries and chaining what it finds across 105 MITRE ATT&CK procedures. To pass, a model has to hold 50% coverage on every tactic. GPT 5.6 Sol comes in second at 41%, the strongest defensive score OpenAI has ever posted, and it clears only six of the 13 tactics. Second place is still a failing grade.
| Rank | Model | Coverage | Median cost | Tactics passing |
|---|---|---|---|---|
| 1 | Opus 4.6 | 44.5% | $2.71 | 7 of 13 |
| 2 | GPT 5.6 Sol | 41.0% | $4.03 | 6 of 13 |
| 3 | GPT 5.5 | 37.4% | $4.50 | 2 of 13 |
| 4 | Sonnet 4.6 | 36.9% | $2.21 | 2 of 13 |
| 5 | GLM 5.2 | 35.0% | $2.01 | 1 of 13 |
| 6 | Opus 4.8 | 32.7% | $1.69 | 1 of 13 |
Anthropic's Opus 4.6 has held the top spot for five months, and Sol doesn't take it, though it closes most of the gap and leaves GPT 5.5 behind. So OpenAI is clearly moving in the right direction. What refuses to move is the bottom line: not one of the 20 models passes. This is not a leaderboard creeping toward solved. It is a wall no one has climbed.
Falling short would sting less if it were cheap. It is the opposite. Sol runs the second-most-expensive and the slowest investigations of anything we tested: a median of $4.03 per case — only GPT 5.5, at $4.50, costs more — and 690 seconds each, the longest on the board. Opus 4.6 scores three and a half points higher for $2.71, which means Sol charges you roughly 50% more to cover less ground. Down at the far end, Google's Gemini 3 Flash returns 14.6% coverage for sixteen cents. If you were hoping price tracked quality here, the benchmark has bad news: the two are barely related.
Pouring in more money and compute buys steadily less coverage, because staying with an investigation, pulling the next query when the trail goes cold instead of quitting, is not what these models were trained to do. It has been the pattern since the benchmark's first run. If the plan for cybersecurity is to run the biggest, newest GPT and let it grind, the data has already priced it out: you pay the most and still miss more than half the attack.
GPT 5.6 did not ship as one model but as three: Sol the flagship, Terra the middle tier, and Luna the fast and cheap one. On defense, calling them the same product is generous.
| Model | Coverage | Median cost | Overall rank |
|---|---|---|---|
| GPT 5.6 Sol | 41.0% | $4.03 | #2 |
| GPT 5.6 Terra | 22.5% | $0.82 | #9 |
| GPT 5.6 Luna | 15.7% | $0.40 | #15 |
Sol at #2 and Luna at #15 sit almost the whole leaderboard apart, with Terra stranded in between at 22.5%, barely more than half of Sol's coverage. "Use GPT for cybersecurity" is not really a decision, then: the variant you happen to pick swings your coverage by more than 25 points, and even the best of the three, at full price, doesn't pass. Which GPT you run swings your results enormously. Whether it is a GPT at all barely registers.
Step back and this stops looking like a GPT problem and starts looking like a defense problem. Offensive AI benchmarks routinely see frontier models clear 80% and up. Defensive coverage on real logs tops out at 44.5%. The gap is structural, not a fluke of one bad run. Offense is a scavenger hunt with a prize at the end: find one working path and you win. Defense is an open-ended investigation with no answer key, where you have to carry a hypothesis across dozens of steps, keep querying when nothing obvious surfaces, and know the difference between clearing a tactic and simply losing the thread.
That last part is where LLMs come apart. Running these evaluations, we have watched a model announce "no further signs of compromise" at step three of an investigation with a dozen steps still to go. They narrate instead of query, call an investigation finished long before it is, and post near-zero on whole tactics they never even looked at. None of this shows up on a coding test or a chat leaderboard, which is exactly why "most advanced cybersecurity AI" and "safe to point at your alert queue" are two very different claims.
Which is the point the whole benchmark exists to make: the model is the least interesting part of an AI defender. What the leaderboard scores is a bare LLM with nothing around it, and the best of those manages 44.5%. Give that same model a defensive harness, something that decides which query to run next, a second pass that catches the early quit, a memory of the environment it is working in, and a MITRE scoreboard tracking which tactics are still open, and the score jumps to production grade. On this same benchmark, the Simbian harness reaches 95%, verified by a global MSSP.
That gap is the reason the AI SOC Agent was built the way it was, on a defensive harness from the first commit, back when the industry still called this kind of thing a wrapper. The assumption underneath it is blunt: don't hand GPT, Claude, or Gemini the keys to your SOC and hope for the best. Make the model one supervised part of a system built to investigate, self-improving, not self-driving, one that gets sharper with every case instead of starting from scratch at the next alert. The model you choose still matters. It is just not what decides whether the attacker walks out with your data.
Is ChatGPT good for cybersecurity? For a lot of an analyst's day, parsing a log dump, sketching a detection rule, or explaining an unfamiliar technique, the LLMs behind it genuinely are. For the part that matters most, catching the attack, GPT 5.6 Sol is a real step forward and still not something you can point at your queue and call a SOC. The model is a component. The system you build around it is the product.
Q: Is ChatGPT good at cybersecurity? For assistive work like drafting a script, summarizing a threat report, or walking through an unfamiliar technique, GPT 5.6 is genuinely useful. For the core defensive job of finding and investigating an attack in live telemetry, the data says no: on Simbian's Cyber Defense Benchmark, GPT 5.6 Sol reaches 41% coverage and fails the 50% pass bar, right alongside every other frontier model tested.
Q: Can ChatGPT detect cyber attacks? Up to a point. The LLMs powering ChatGPT are good at flagging individual pieces of suspicious activity, but detection means stitching those pieces into one coherent attack across the kill chain, and that is where they slip. On the Cyber Defense Benchmark, GPT 5.6 Sol clears only 6 of the 13 MITRE ATT&CK tactics and misses the 50% pass bar. Spotting a clue is not the same as confirming an intrusion.
Q: Can ChatGPT do a SOC analyst's job? Not on its own. The analyst's real work is sustained, multi-step investigation through noisy logs with no answer key, and that is exactly where every frontier model stalls. Give a model a harness that plans its next move, second-guesses its own conclusions, and remembers your environment, and it can reach production-grade coverage. Left to run raw, it cannot.
Q: Will ChatGPT replace cybersecurity analysts? No. The benchmark shows the opposite of a drop-in replacement, with the strongest model still missing more than half of the attack. What AI does is move analysts off manual triage and onto supervising and improving the systems that grind through it, so the work climbs rather than disappears.
Q: Is ChatGPT a cybersecurity risk? It can be, for a different reason. Pasting sensitive logs or source code into a consumer chatbot invites data exposure and prompt-injection attacks, which is why enterprises run private, governed deployments instead. Simbian, for one, runs on-prem inside the customer's own environment in production, so sensitive telemetry never leaves their network. That risk is about how the model is used, not how well it defends, which is the question the benchmark answers.
Want to see how every frontier model scores against real attacks? The Cyber Defense Benchmark sample and methodology are public. And if you are weighing an AI SOC vendor, the AI SOC Buyer's Scorecard lays out the eight dimensions that matter and the 30-plus questions to ask before you sign anything.
