What Should a CISO Actually Measure?

The short answer

Measure change in exposure and the organization’s ability to absorb an attack. Do not measure activity.

Keep the set small. Tie every measure to a decision someone in the room can actually make. If a number can’t change an investment, a risk acceptance or a priority, it belongs in an operational report, not in front of executives.

Why the usual dashboard fails

Take a hypothetical company. Its board pack reports 97% patch compliance, 4,000 phishing emails blocked, and an average ticket closure time that improved this quarter. Every number is accurate and every one is green.

Meanwhile the one internet-facing remote-access appliance that sits on the path to the finance platform has been awaiting a maintenance window for six weeks. The dashboard can’t show that, because it counts the volume of work, not where the risk sits.

That is the pattern. Activity measures are easy to collect and easy to improve. They say little about whether an attacker has a harder or easier path to what the business cannot afford to lose.

Four questions a measure must answer

Each measure should answer one of four questions. Together they cover the risk and the capability behind it.

A practical test for any candidate measure:

1. What decision does it inform?

2. Who owns the number, and who can act when it moves?

3. What threshold triggers action?

4. What is it compared against: a target, a tolerance or a trend?

If a measure fails the first question, drop it from the executive set. If it fails the third, you don’t have a measure yet. You have a statistic.

Four design rules

Anchor every measure to a tolerance. A number with no threshold is trivia. The business has to say how much exposure it will accept, or the CISO ends up setting risk appetite by default. NIST CSF 2.0 places risk appetite and tolerance inside its Govern function for this reason.

Prefer tested over asserted. “Controls deployed” is an assertion. “Control worked when we tested it” is evidence. Where you can only assert, say so on the page. Leaders can handle uncertainty. They handle surprises badly.

Show the denominator and the trend. “Forty critical findings remain open” means nothing until you say out of how many, and whether that is better or worse than last quarter. Note data quality and known gaps next to the number. NIST’s measurement guidance treats data quality and uncertainty as part of reporting, not as footnotes.

Run two tiers. Give the board and executive committee a short set, in our view no more than six or seven, each tied to a decision. Keep a larger operational set inside the security function for management. Most reporting problems come from showing the second tier to the first audience.

What the standards give you, and what they don’t

NIST published SP 800-55 in two volumes in December 2024, replacing the 2008 revision. Volume 1 covers identifying, selecting and prioritizing measures, including quantitative and qualitative approaches. Volume 2 covers building a measurement program. CSF 2.0’s Oversight category asks organizations to evaluate how well cyber risk management is performing and to adjust accordingly (GV.OV-03).

Neither document tells you which measures to choose. That selection depends on your business, your tolerance and your decisions. It is leadership judgment, and it is the CISO’s job to make it.

For US-listed companies, the SEC’s cybersecurity disclosure rule requires a description of board oversight and management’s role in managing material cyber risk. It does not prescribe metrics. A reporting set built on the four questions gives you something real to say under that requirement.

Two common objections

“We can’t quantify our risk.” You don’t have to start with money. NIST’s guidance accepts qualitative measures alongside quantitative ones. A consistent, defensible scale beats false precision.

“The board wants one score.” A composite score hides the trade-offs the board needs to see. A single “72” can improve while your most important exposure gets worse. Show the four questions and let the gaps be visible.

What to decide in the next 30 days

5. List the decisions. Write down the five or six decisions leadership will make in the next twelve months: investments, risk acceptances, cloud or M&A moves, recovery commitments.

6. Rebuild the set backwards. For each decision, pick the one or two measures that would change the answer. Drop the rest from the executive pack.

7. Agree tolerances with the business. Get an executive owner to sign off on the threshold for each measure.

8. Test the data. Pick the three numbers you trust least and validate them by testing, not by asking the team that produces them.

Next step. Take your current executive dashboard and run every measure through the decision test above. Rebuild the reporting set around what survives. If you want a second opinion on that set, Cybrilliant’s vCISO & CISO Advisory team can review it with you.

Sources

• NIST SP 800-55 Vol. 1, Identifying and Selecting Measures (Dec 2024)

• NIST announcement of SP 800-55 Vols. 1 and 2 (4 Dec 2024)

• NIST Cybersecurity Framework 2.0 (CSWP 29)

• SEC final rule on cybersecurity risk management, strategy, governance and incident disclosure (33-11216)