Field Note: Building a Detection Use-Case Backlog

Build the backlog from threats and data, not from a vendor rule pack or an ATT&CK heatmap. Every item should answer three questions: which attacker behavior does this catch, can we see it today, and how will we know it works?

Most backlogs start as a list of requests. A hunt turns something up, an incident exposes a gap, an auditor asks about logging, a vendor ships a new content pack. Each request is reasonable. Together they make a pile with no order, no definition of done, and no way to retire anything.

Define what an item is

Give every item the same shape. A detection use case is one attacker behavior, the data needed to see it, the logic that finds it, a test that proves it fires, and guidance for the analyst who receives the alert.

Palantir’s Alerting and Detection Strategy template is a well-known model. It has sections for goal, technical context, blind spots and assumptions, false positives, validation and response. You do not have to adopt it as written. Borrow the discipline: an item with no validation method and no stated blind spot is not ready to build.

Feed the backlog from a short list of sources

• Threat relevance. Intelligence on the actors and techniques that target your sector and technology stack.

• Incidents and near misses. What the SOC missed, or found late.

• Offensive testing. Red team and penetration test results, and attack path analysis. Each technique that worked without detection is a candidate.

• Hunts. A hunt hypothesis that finds activity becomes a detection once the logic is stable.

• Change. New cloud services, SaaS applications and identity providers bring new behaviors and new log sources.

• Obligations. Regulatory or contractual logging and monitoring requirements.

Treat ATT&CK as a vocabulary. It is a knowledge base of adversary behavior drawn from real-world observation, so it is a good way to name and group items. It does not say which techniques matter to you. Counting techniques that have at least one rule produces a heatmap, not a measure of detection.

MITRE’s Center for Threat-Informed Defense makes the same point through its Summiting the Pyramid project. It adds two measures: implementation coverage, which counts how many behaviorally distinct ways of performing a technique an analytic can observe, and detection quality, which assesses how robust and precise the signal is.

Check data readiness before you rank

A behavior you cannot see is not a detection yet. It is a telemetry gap. Before ranking, check three things. Is the log source collected? Are the fields you need parsed and searchable? Is the data kept long enough to help an investigation? If any answer is no, move the item to a blocked state and open a task with the owner of the log source.

This keeps two lists honest. The detection backlog holds what engineers can build now. The telemetry list holds what is blocking them. The joint guidance on event logging and threat detection from ASD’s ACSC, CISA, the FBI, NSA and international partners is a practical reference for the logging baseline, including why logging matters for spotting living-off-the-land activity.

Rank on value first, effort second

Score each ready item on a few factors using a 1 to 3 scale, where a higher number is always better for you. Avoid false precision. The aim is a defensible order, not a formula.

Rank on relevance and impact first. Use the other factors to sort within a tier. A cheap, noisy rule on a low-impact behavior should not outrank a harder rule on credential theft.

Run the backlog as a pipeline

Cap the number of items in build and in test. Reserve fixed capacity for tuning and retirement. Without that reserve, new rules crowd out maintenance and noisy rules stay forever.

Define done

• A test event triggers the detection, and the test is stored with the rule.

• False positive causes are documented and, where safe, excluded.

• Triage steps exist and an analyst has read them.

• Blind spots and assumptions are recorded.

• An owner and a review date are set.

Tie re-tests to change. A parser update, an agent upgrade or a log schema change can break a detection without anyone noticing.

Retire as deliberately as you build

Review rules on a schedule. Rewrite the ones an attacker can evade with trivial changes. Retire rules that never fire and cannot be validated, and rules that fire constantly without anyone acting. A smaller set of rules that fire rarely and mean something beats a large catalog the SOC has learned to ignore.

Where backlogs go wrong

• Treating an imported rule pack as coverage.

• Ranking by what is easy to build.

• Building detections for data that is not collected.

• Marking a technique green after a single signature.

• Leaving rules without an owner, so nobody tunes them.

• Forcing everything into the SIEM when the endpoint or identity platform sees the behavior more directly.

Use this note to improve detection coverage and quality: rank items by threat and impact, confirm the data exists before building, and prove every rule fires before it goes live.

Sources

• MITRE — ATT&CK

• MITRE Center for Threat-Informed Defense — Summiting the Pyramid

• Palantir — Alerting and Detection Strategy Framework

• ASD’s ACSC, CISA, FBI, NSA and partners — Best Practices for Event Logging and Threat Detection