Expert Briefing: Cyber Resilience Testing That Produces Evidence

Briefing overview

Many resilience exercises end with a good discussion and a slide. Six months later, nobody can say what was proved, what was assumed, or whether the fixes worked.

This briefing treats a resilience test like any other control test. State what you expect to happen. Create a realistic disruption. Record what actually happens. Fix what failed, then retest. The result is evidence that an auditor, a supervisor or your own board can inspect.

What you will take away

• A test design pattern: hypotheses, injects and measures defined before the exercise starts.

• A list of evidence artifacts to capture, and the weak substitutes to stop accepting.

• A way to move from discussion exercises to technical restoration tests.

• A close-out routine that turns findings into retested fixes.

Agenda

A story is not evidence

Evidence is recorded, comparable and repeatable. A facilitator’s recollection that “the team handled it well” is none of these. The table sets out what a test should answer, what to capture, and what to stop accepting in its place.

For a floor, the joint advisory on Scattered Spider recommends offline, encrypted, immutable backups kept separate from the source systems, with restoration tested at least yearly. Treat that as a minimum. A critical service with a tight recovery objective needs more frequent and more realistic restoration tests than once a year.

Designing the test

1. State the hypotheses and targets first. Write down the recovery objective, the restore order you expect and who you expect to make each call. Without these, you cannot score the result.

2. Pick a disruption that matches a credible threat. Examples: the identity provider is unavailable; attacker credentials can reach the backup repository; the hypervisor layer is encrypted (the advisory reports this, citing trusted third parties, in connection with the group); a key SaaS dependency is down.

3. Use observers who record and do not participate. Participants are busy making decisions. Someone else should write down the times and the choices.

4. Score against the targets. Report the gap between the objective and the measured result, service by service.

5. Log each finding with an owner, a due date and a retest. A finding without these is an opinion.

6. Retest. The retest is the evidence that the fix worked.

The ladder of tests

Different tests prove different things. This framing is ours. It is not taken from a standard.

NIST SP 800-84 gives guidance on planning, conducting and assessing test, training and exercise programs for IT plans and capabilities. For readers in regulated financial services in the European Union, the Digital Operational Resilience Act (DORA) requires an operational resilience testing program that includes a range of tests, with advanced testing among them. Supervisors will ask what the tests produced, so the evidence format matters.

Closing the loop

Measure the program, not just the exercise. Useful measures include the share of findings closed by their due date, the results of retests, and the trend in time to restore for each critical service. The trend tells you more than any single run.

Next step

Use this briefing to test resilience under realistic disruption. Pick one critical service, write its recovery objective down, and plan a restoration test that records timings, so the next exercise produces numbers and not just notes.

Sources

NIST SP 800-84: Guide to Test, Training, and Exercise Programs for IT Plans and Capabilities — National Institute of Standards and Technology

Scattered Spider (joint cybersecurity advisory) — CISA and partner agencies

Digital Operational Resilience Act (DORA) — European Securities and Markets Authority