Expert Briefing: Making Detection Engineering Measurable

A briefing on replacing rule counts and heatmap percentages with measures of detection coverage, quality and reliability.

Briefing at a glance	Details
Focus	Replacing rule counts and heatmap percentages with measures of detection coverage, quality and reliability
Audience	CISOs and security leaders, SOC and detection engineering leads, threat hunters, security operations managers
Scope	Cross-industry / Enterprise

About this briefing

Ask a security operations team how good its detection is and the answer is usually a number: rules in production, techniques covered on a heatmap, alerts handled per day. All of them describe activity. None says whether the next intrusion will be caught.

The external picture is mixed. Mandiant’s latest M-Trends report found that organizations detected 52% of the intrusions it investigated internally, up from 43% the year before. In 34% of cases an outside party notified them, and in 14% the attacker did. Ransomware cases look worse: the adversary announced itself to the victim in 44% of them, while the victim’s own team found the intrusion in 41%. These figures come from one firm’s incident response work, not from a sample of all organizations, so they are a reference point rather than a benchmark. They still show how often the detection program is not the first to know.

This briefing is about making detection engineering something a team can measure, improve and defend to leadership. It covers what to measure, how to measure it without gaming it, and what to say upward.

What you will leave with

Why rule counts and ATT&CK heatmaps overstate capability, and what to use instead

Four families of measures: coverage, quality, validation and health

A lifecycle view that shows where detections are stuck

A short leadership view that does not reward the wrong behavior

Briefing flow

Segment	Focus	Question it answers
1	What most programs count	What do the usual numbers actually say?
2	Coverage, quality, validation, health	What should we measure instead?
3	The detection lifecycle	Where do detections stall or decay?
4	Two detections, one green cell	Why can identical coverage hide very different protection?
5	Reporting to leadership	What belongs on the page, and what does not?
6	How measurement goes wrong	Which targets produce the wrong behavior?

What most programs count

Common metric	What it measures	What it misses
Number of detection rules	Authoring activity	Whether the rules fire, fire correctly, or have the data they depend on
ATT&CK heatmap coverage	Rules mapped to techniques	That one mapped rule may catch a single way of performing a technique. MITRE’s Center for Threat-Informed Defense built Summiting the Pyramid to measure beyond a heatmap.
Alert volume	Analyst workload	Signal quality. A high volume can mean broad coverage or poor tuning.
Mean time to detect	An average over a few incidents	Most of the picture. Real incidents are few, uneven and rarely comparable.

None of this is wrong to track. It is wrong to present as capability.

Coverage, quality, validation, health

The Center for Threat-Informed Defense frames detection depth with two ideas. Implementation coverage asks how many behaviorally distinct ways of performing a technique an analytic can observe. Detection quality asks how robust and precise the signals behind it are. Both aim at detections that are harder for adversaries to evade. We add two operational families that decide whether those properties hold in practice.

Family	Question	How to measure it	Common failure
Coverage	For the behaviors that matter to us, how many distinct ways can we observe them, on which assets?	Pick a priority list from your threat model and incidents, not all of ATT&CK. For each behavior, record the observable variants, the data sources, and the in-scope systems.	Counting a technique as covered when only one variant is detected, or only on part of the estate
Quality	When it fires, is it right, and how hard is it to evade?	Precision of alerts (share that are true or actionable), analyst triage time, and a robustness rating of what the rule keys on	Tuning for precision until the rule no longer catches what it was written for
Validation	Do we know it works?	Share of priority detections tested by emulation or replay, pass rate, and days since last test	Writing rules and never testing them against the behavior
Health	Is the telemetry still arriving and parsed?	Log source freshness, parsing error rates, and rules silent for longer than expected	Reading a quiet rule as a quiet environment, when the data feed broke weeks ago

Where to start. Health and validation cost the least and change the most. A rule on a dead log source and a rule that has never been tested are both counted as coverage today.

The detection lifecycle

Counting finished rules hides where work is stuck. Treat each detection as something that moves through stages, and measure the movement: threat input, hypothesis, build, test, deploy, tune, retire. A hunt that finds a gap should land in the backlog as a detection request. An incident review should do the same.

• Cycle time from a validated threat input to a tested detection in production. This shows how quickly the team can respond to what it learns.

• Backlog age. A backlog full of year-old requests tells you more about prioritization than any rule count.

• Share of detections with a test case and a documented triage path. A detection an analyst cannot act on is incomplete.

• Tuning debt. Rules generating most of the noise, and how long they have waited for work.

• Retirement. Detections that no longer match your environment or the threats you face should leave the set, and the removal should be recorded.

Many teams now manage detections as code, with version control, peer review and automated tests. This is common industry practice rather than a standard, but it is what makes cycle time and test coverage measurable without extra effort, because the pipeline records them.

Detection also depends on logging that exists. A joint guide on event logging and threat detection, produced by Australia’s ACSC with CISA, the FBI, the NSA and agencies from a number of other countries, sets a baseline for what to collect. If the data is not there, a rule that references it is decoration.

Two detections, one green cell

The following is an illustration. Take OS credential dumping from LSASS memory, a technique in ATT&CK. Two rules both map to it.

	Rule A	Rule B
What it keys on	A specific tool’s command-line string	A non-system process opening a handle to the LSASS process with memory-read access
Evasion	Rename the binary or change the arguments	Harder. The attacker needs a different access method or a way to avoid the telemetry.
Dependency	Command-line logging	Process-access telemetry from the endpoint sensor on every in-scope host
Heatmap	Green	Green

The heatmap cannot tell them apart. Rule A is easy to produce and easy to defeat. Rule B is stronger, but it protects only the hosts that send the telemetry, which is exactly what a health measure would show. Quality and health are two sides of the same question.

Reporting to leadership

Leaders want to know whether the organization would catch the thing that worries them. A short page is enough:

• Priority behaviors with a tested detection, as a fraction of the list agreed with the business (for example, a set tied to the current threats and the incidents you have actually had).

• Detection quality trend, using precision and triage time on the same rules over time.

• Validation freshness, meaning how recently the priority detections were tested.

• Cycle time from threat input to production detection.

• Share of incidents found internally, with the caveat that the sample is small and the trend is more useful than any single figure.

Comparing the last measure with external figures like M-Trends is fine if the limits are stated. Your incident mix is not the same as any investigator’s caseload.

How measurement goes wrong

Any measure that becomes a target changes behavior. A rule-count target produces rules. A heatmap target produces shallow mappings, one rule per cell. A precision target alone invites suppression: tune out the noise, and the true positives go with it. A mean-time target rewards closing alerts quickly over closing them correctly.

The defense is to measure in pairs. Pair coverage with quality, precision with validation, speed with accuracy. If one improves while its partner falls, that is information.

Questions on the table

How do we choose the priority list, and how often do we revisit it as threats change?

How do we test detections safely in production, and who signs off?

How do we weigh a precise but easily evaded rule against a noisy but robust one?

Where do SOC, MDR and threat hunting each feed the backlog?

What do we tell the board when the honest number looks worse than the heatmap?

What to do next

1. Choose 15 to 25 behaviors from your threat model and recent incidents. For each, record how many distinct ways you can observe it today.

2. Check log source health for every data feed those detections depend on. Fix the dead ones first.

3. Test the top detections with an emulation or replay, and record pass or fail with the date.

4. Add cycle time and backlog age to the SOC report next to the rule count, then watch which one people discuss.

5. Replace the heatmap percentage on the leadership page with the priority-behavior fraction.

Putting this to work

Cybrilliant’s Threat Detection & Detection Engineering service works on this problem, with SIEM Professional Services, SOC / MDR and Threat Hunting as adjacent capabilities for teams that need help on the platform, the operating model or the hunting program.

Sources

1. Google Cloud, M-Trends report (latest edition). Mandiant / Google Threat Intelligence Group.

2. Summiting the Pyramid. Center for Threat-Informed Defense, operated by MITRE.

3. MITRE, ATT&CK knowledge base. The MITRE Corporation.

4. Best Practices for Event Logging and Threat Detection. Australian Signals Directorate’s ACSC with CISA, FBI, NSA and international partners.