Recovery Testing: Proving That Backups Are Actually Recoverable

The claim you are testing. A system is recoverable only if you have restored it to a working state, inside its recovery time, using only what would survive the incident. Anything short of that is an assumption.

Backup success is not recovery success

Backup tooling reports on the write. It tells you the job finished and the catalog was updated. It tells you very little about whether the data is consistent, whether the application will start on it, whether the keys and credentials needed to read it will exist during an incident, or how long the restore takes when dozens of systems are queued behind one network link. Those are recovery properties, and they only appear when someone actually restores.

Define “recoverable” before you test

Two terms anchor the test. The recovery time objective (RTO) is how long the business can tolerate a service being down. The recovery point objective (RPO) is how much data loss, measured in time, it can tolerate. Regulation has started to make this explicit: DORA Article 12 expects financial entities to set both for each function and to test backup and restoration procedures periodically. A restore only counts if it meets four conditions.

The test ladder

Each level proves something the one below cannot. Record the highest level evidenced for each system and report that, not the level you intended to reach.

What to record every time

Where recovery tests usually fail

These are common failure modes worth testing for directly.

• The restore depends on something the incident removes. A domain-joined backup server, admin credentials held in a vault that is itself down, or encryption keys stored in a key service inside the affected environment. Test by restoring with those unavailable.

• The backup is application-inconsistent. A crash-consistent copy of a busy database may need log replay, or may not start. Test the database, not just the volume.

• Dependency order is unwritten. The application comes back before identity or DNS, and nothing authenticates.

• Retention is shorter than detection time. If an intrusion went unnoticed longer than your oldest clean backup, every available copy may be compromised. CISA’s ransomware guide tells responders to look for precursor dropper malware before rebuilding from backups, because ransomware is often used to hide an earlier compromise.

• Immutability is mistaken for cleanliness. Write-once storage preserves whatever was written, including a compromised state. CISA also notes that misconfigured immutable cloud storage can be costly.

• SaaS and cloud-native data is assumed covered. Check what the provider restores and what remains your responsibility, and test the part that is yours.

• The build is missing, not just the data. Infrastructure-as-code templates, images, licenses and configurations need protected copies too, which CISA’s guide also recommends keeping offline.

• Hardware and bandwidth were never sized for a full restore. Throughput measured in a single-system test rarely survives a many-system recovery.

Reporting a rating that means something

Cadence and triggers

Frequency should follow criticality, not the calendar alone. Tier-one services justify scenario-level tests; lower tiers can stay at sample or full-system restores. Retest after anything that can change the answer: a new backup platform or architecture, a major application upgrade, an identity or network redesign, and any fix made after a failed test. A failed restore is a successful test. Log it, assign an owner and set a retest date.

Checklist before you call a system recoverable

• Restored from a stored backup set, not a snapshot taken for the test

• Restored without production admin credentials or the production identity platform

• Application-level validation passed

• Elapsed time recorded against the RTO

• Recovery point recorded against the RPO

• Dependencies restored in a documented order

• Runbook updated with every manual step discovered

• Failures logged with owner and retest date

Sources

#StopRansomware Guide — CISA

Article 12: Backup policies and procedures, restoration and recovery procedures and methods (DORA) — Advisera

Data Integrity: Recovering from Ransomware and Other Destructive Events (SP 1800-11) — NIST National Cybersecurity Center of Excellence