When the SOC Has Too Many Alerts

The situation

The queue grows. Analysts triage faster and close more, and the queue still grows. Leadership hears three options: hire more people, buy a platform that promises to cut noise, or raise thresholds. Each can help a little. None of them tells you why the queue is full.

The cause usually sits upstream of the analyst. It is in how detections were written, what the log sources send, what context an alert carries, and what happens to the learning when an alert is closed. This note walks through how to diagnose that, in the order that tends to pay off.

Count the right thing

Total alert count says very little. Count by rule. For each rule, record how many alerts it raised, what share of the queue that is, and how each one ended. For that last part you need a disposition standard that analysts actually use. Five categories are enough to start:

The split matters because the fixes differ. Many teams close all of the middle three as “false positive.” That erases the difference between a broken rule and a good rule with no context, and it sends tuning effort in the wrong direction. If analysts cannot record a structured disposition and a short reason in the case tool, fix that before you tune anything.

Read the pattern

Once dispositions exist, the queue usually shows one or more of these patterns.

Fix the producers before the consumers

Work in this order. Each step reduces the load on the next.

1. Log sources. Repair misconfigured sources, duplicate feeds and verbose logging that adds volume without detection value. Confirm that sources are healthy and still sending. A quiet source can mean a broken source.

2. Rules. Review the highest-volume and lowest-actionability rules first. Rewrite those with weak logic, merge those that overlap, and retire those no one has acted on after a defined review period.

3. Context. Give each alert the information an analyst would otherwise look up: asset criticality, user privilege, ownership, recent related alerts, and whether the activity matches a known change.

4. Triage. Write short playbooks for the rules that produce the most work. State what to check, what closes the alert, and what escalates it. Set a time limit for the first decision.

5. Feedback. Review dispositions every week with the people who own detection content. A rule that analysts keep closing as benign should return to its owner with the evidence.

One caution on retiring rules. A rule with no true positives in its history is not automatically useless. Rare events, such as someone disabling a security agent or clearing logs, produce rare alerts. Judge a rule by what it would catch and what you would lose, not by its history alone. If you remove a rule that covered a high-impact technique, replace the coverage or accept the gap in writing.

Coverage maps built on MITRE ATT&CK, the public knowledge base of adversary tactics and techniques, show which techniques have a rule. They do not show whether that rule works, fires on the right data, or gets read. Treat coverage and quality as two measurements.

Suppression is a security decision

Allowlists and exceptions reduce noise. They also create blind spots that attackers can use. Adversaries often work through tools that already exist in the environment, including legitimate remote access software (ATT&CK technique T1219). A blanket suppression of an administration or remote access tool hides the abuse of that tool along with the normal use.

A safe exception has four properties:

• It is narrow: one rule, one entity, one parameter, not a whole category of activity.

• It has a named owner who can say why it exists.

• It expires, and someone reviews it before renewal.

• It is logged, so you can see what has been silenced and for how long.

Measure whether it is working

Be careful with speed targets alone. If analysts are judged on alerts closed per hour, quality drops and the data you rely on becomes unreliable. Pair speed measures with a regular sample of closed alerts, reviewed by a second person, especially those closed as benign.

If you use an MDR provider

A provider can take first-line triage off your team. It does not repair your detection content or log sources, and it can move the volume problem out of sight rather than remove it. Settle a few things in writing:

• Which telemetry and which detections the provider triages, and which stay with your team.

• What qualifies for escalation to you, and how fast.

• Which response actions the provider can take without asking, and which need your approval.

• Whether you can see the provider’s closure reasons, so that your own tuning can use them.

A two-week starting plan

First week. Export a recent block of alerts, long enough to include routine cycles such as month-end jobs or patch windows. Add dispositions where they are missing. Rank by rule. Group the top producers by pattern, using the table above. Check log source health.

Second week. Fix the top producers: repair sources, rewrite or merge rules, add the enrichment analysts repeat by hand, and create scoped exceptions with owners and expiry dates. Record the measures above as a baseline, so that the next review compares against something.

What not to do

• Raise thresholds across the board. It removes weak alerts and strong ones together.

• Auto-close by severity label. The label is only as good as the rule author’s guess.

• Hire first. More analysts working an unchanged queue will close more noise.

• Disable a noisy rule with no replacement and no recorded decision.

• Treat a shrinking queue as proof of better detection. Check what stopped being detected.

What to do next

Run the rule-level review on your own queue. If the same few rules account for most of the volume and no one owns their tuning, that is where the work starts. If you want an outside view of monitoring and response effectiveness, covering detection quality, triage process and log source health, Cybrilliant can assess it with your team.

Sources

• MITRE ATT&CK. MITRE. https://attack.mitre.org/

• ATT&CK technique T1219, Remote Access Tools. MITRE ATT&CK. https://attack.mitre.org/techniques/T1219/

• Best Practices for Event Logging and Threat Detection. Australian Signals Directorate’s Australian Cyber Security Centre, with CISA and international partners. https://www.cisa.gov/resources-tools/resources/best-practices-event-logging-and-threat-detection