Security Operations Metrics and Maturity

Most security operations programs report on some subset of the same handful of measures. The list is short, and the disagreement is almost entirely about definitions rather than about which metrics matter.

• MTTD (mean time to detect): how long between something happening and the team knowing about it.
• MTTR: how long between knowing and some later milestone. Which milestone is the contested part, and is covered below.
• Dwell time: how long an adversary was present in the environment before being found.
• Alert volume and alert-to-incident ratio: how much arrives, and what proportion turns out to be real.
• False positive rate: the proportion of alerts that were not what the rule claimed.
• Escalation rate and escalation accuracy: how much of the queue moves up a tier, and how much of that was correct to escalate.
• Coverage: which log sources, which assets, and which adversary techniques are actually observable. Frequently the most useful number and the least often reported.

Every one of these measures can be improved without improving security, and several can be improved by making security worse, which is why a metrics conversation that does not also cover how each number is instrumented and who is accountable for the definition tends to produce a dashboard that looks better every quarter while nothing underneath it changes. Coverage is the hardest of them to game. That's part of why it deserves more attention than it gets.

MTTD (mean time to detect) measures the average elapsed time between when a security event occurred and when the security team became aware of it. It is the metric that most directly reflects whether your monitoring is working.

The measurement is harder than the definition because the start point is only knowable in retrospect. You do not know when the compromise began until you have investigated it, so MTTD for any given incident is a number established during the post-incident review, not something a dashboard computes in real time.

That leads to a widespread substitution. Many tools report a much narrower interval, usually the time from alert generation to alert acknowledgement, and label it detection time. That number is useful for measuring analyst responsiveness, and it is not MTTD. If your reported MTTD is measured in minutes while your dwell time is measured in weeks, this substitution is almost certainly what is happening.

MTTR is four different metrics sharing one acronym, and almost nobody discloses which one they are reporting, and mean time to respond, mean time to repair, mean time to recover, and mean time to resolve each measure a different span of time with a different endpoint.

These can differ by an order of magnitude on the same incident. Isolating an infected laptop takes minutes. Restoring the affected business process might take days. Closing the case with root cause understood might take a week. Reporting one of these and comparing it against someone else's other one is a routine and mostly invisible error.

Two consequences follow. First, when you compare your MTTR to a benchmark, an SLA, or a competitor's marketing number, the comparison is meaningless unless both sides use the same expansion and the same clock. Second, when a tool reports MTTR, find out in the documentation which expansion it implements before you put the number in a board deck. Atlassian is one of very few vendors that states plainly which expansion it means; most of the rest leave it to be inferred.

There are two common answers and they are not reconcilable, so the only safe practice is to state which one you use.

• Clock starts at occurrence: the timer begins when the adverse event actually happened, regardless of when anyone noticed. This produces a number that includes your detection failure, which makes it honest and unflattering.
• Clock starts at detection: the timer begins when the alert fired or when the team acknowledged it. This measures the response process in isolation, which makes it actionable for the responders and hides everything upstream.

Most tooling defaults to the second, because the tool only knows about events it saw. Most executives assume the first, because it matches the intuitive question of "how long were we exposed."

The gap between the two is exactly your detection gap, which suggests a more useful practice than picking a side. Report both, and the difference between them becomes a metric in its own right, and a team whose response clock is tight and whose occurrence clock is long has a detection problem, not a response problem, and no amount of response optimization will move the number that matters.

Dwell time is the length of time an adversary is present in an environment before being discovered, it is the metric most closely tied to how much damage an intrusion is able to do, since nearly everything an attacker accomplishes requires time inside.

The clock start is not controversial: it begins at initial compromise, established retrospectively during investigation. The clock stop is genuinely contested, and the disagreement is between named sources rather than between careless ones. Corelight and Optiv stop the clock at eradication, meaning the moment the adversary's access was actually removed. ConnectWise and Barracuda stop it at detection, meaning the moment the organization became aware. Fortinet defines dwell time as mean time to detect plus mean time to respond combined, which is a third measurement again.

The difference isn't academic. An organization that detects an intrusion on day nine and completes eradication on day twenty-three has a dwell time of nine days or twenty-three days depending on whose definition it uses, and both numbers get published as dwell time. Industry dwell-time benchmarks are drawn from vendor incident response datasets that do not all use the same convention, which is one reason published figures vary and why year-over-year comparisons across different sources should be treated cautiously.

If you report dwell time internally, define the stop point once and keep it. Pick one and keep it. Trending your own number against itself is far more useful than comparing it to a published benchmark whose definition you cannot see.

Because they are not measuring the same population of events, and the two numbers come from completely different sources.

MTTD as reported by tooling is generally computed over alerts, meaning things the detection stack found. Its denominator is the set of events that were detected, by definition, and the interval is usually from alert generation to acknowledgement.

Dwell time as reported in industry research is generally computed over confirmed intrusions investigated after the fact, most often by incident response firms who were called in because something went badly. Its population is skewed toward the cases where detection failed, and its interval starts at initial compromise, which is only knowable in hindsight.

So the gap is not a contradiction. One number describes how fast you react to the things you catch. The other describes how long the things you did not catch stayed hidden. A team can genuinely be excellent at the first and poor at the second, and that combination is common rather than unusual.

Stop treating a fast MTTD as evidence against a long dwell time, they answer different questions, and only coverage measurement connects them, because what determines dwell time is mostly whether the telemetry that would have shown the activity was being collected and evaluated at all.

A false positive rate is the proportion of alerts that turn out not to be what the detection claimed. The honest answer to "what is normal" is that the number is close to meaningless without knowing what is in the denominator and how tuned the environment is.

The definitional problem comes first. Practitioners routinely use "false positive" for at least three different things:

• A genuine false positive: the activity the rule claimed to see didn't actually occur. The detection was wrong.
• A benign true positive: the activity did occur exactly as detected, and it was authorized. An administrator really did run that tool, with approval. The detection was correct and the alert was still not actionable.
• An irrelevant true positive: the activity occurred, was unauthorized, and does not matter, such as a blocked attempt that never had a chance of succeeding.

Only the first is a detection defect. The second is a tuning and context problem, and it is usually the largest of the three by volume, and teams that lump all three together produce a false positive rate that mostly measures how much environmental context their alerts lack.

Reported rates vary enormously across organizations, and the variation is driven by tuning maturity, alert-rule breadth, environment size, and which of the three definitions above is being counted. A rate that is very low may mean detections are well tuned, or it may mean coverage is narrow and things are being missed. Because of that, the direction of your own number over time, alongside a measure of what you're still catching, is far more informative than any cross-organization comparison.

Nearly all of them, and the how matters, because the people who control the definitions are usually the people the numbers evaluate.

• MTTR: change the clock start, or change what counts as the ending action. Redefining "response" as "first analyst touch" can move the number by an order of magnitude without anything improving.
• Alert volume: disable or narrow noisy rules. The queue gets shorter, the dashboard improves, and coverage quietly falls.
• False positive rate: reclassify benign true positives as true positives. Same alerts, better ratio.
• Time to close: close aggressively and reopen if something turns up. Throughput looks excellent right up until an incorrectly closed alert becomes an incident.
• Escalation rate: the target can be gamed in either direction depending on which way the pressure runs, and both directions are harmful.
• Incidents handled: count alerts as incidents. This is the single most common inflation in outsourced reporting.
• Coverage percentage: define the denominator as the assets you monitor rather than the assets you have.

None of this requires bad faith. It happens by default whenever a metric becomes a target and the same team owns the definition, the instrumentation, and the outcome, and two mitigations help more than the rest: publish the definition alongside the number every time, and pair every efficiency metric with a quality metric that moves the other way, such as pairing time-to-close with the rate of reopened cases.