The gap between present and effective
A board paper can say that multifactor authentication is enabled, endpoint protection is deployed and backups run nightly, and every statement can be true. None of them tells the reader whether an attacker holding a stolen session is stopped, whether a malicious process is blocked or whether a restore succeeds when it is needed. Those are separate questions, and they need separate evidence.
Audit practice draws the same line. A SOC 2 Type 1 report describes the design of controls at a point in time. A Type 2 report tests whether they operated over a period. Neither says whether the design stops a determined adversary. Leadership needs to know which kind of evidence sits behind each claim it hears.
An evidence ladder
The ladder below is our framing, not a published standard. It orders the evidence behind a control by how much it can show.
| Level | Evidence | What it shows | What it cannot show |
|---|---|---|---|
| 1. Attestation | The control owner states the control is in place | Ownership and intent | Whether the control operates |
| 2. Documented design | Policy, procedure or configuration standard | The control has been defined | Whether reality matches the document |
| 3. Coverage data | Automated checks; percentage of assets covered | Where the control is deployed | Blind spots in scope; whether it blocks anything |
| 4. Operating effectiveness testing | Independent testing over a period | The control operated as designed | Whether the design stops a real attack |
| 5. Adversarial validation | Penetration testing, red teaming, attack simulation, detection tests | The control holds against realistic attacker behavior in the scope tested | Anything outside the scope; whether it still holds tomorrow |
Not every control needs level 5. A password policy can sit at level 4. The controls that protect critical services, privileged access and recovery should not stop at level 2 or 3.
Where assurance breaks
-
The wrong scope. The assured area is easy to test and is not the area that protects the crown jewels. A clean result then says little about real exposure.
-
A single point in time. A control tested once says nothing about the next configuration change or the next acquisition.
-
Self-assessment. Control owners and the risk functions that support them cannot give independent assurance about their own work. The IIA’s Three Lines Model reserves that role for an independent line, typically internal audit.
-
Sampling that hides systemic failure. A sample can pass while the exceptions cluster in the systems that matter most.
-
Presence mistaken for effect. Typical illustrations are authentication that is enabled but with exempted accounts, or protection software that is installed but set to log without blocking.
-
Findings that are never verified closed. A remediation ticket marked done is a claim. Validation that the gap is actually closed is the evidence.
What regulators and standards expect
The sources below point the same way: management is expected to oversee cybersecurity risk measures and to be able to show how their effectiveness is assessed. Applicability depends on the entity, its sector and its jurisdiction, so confirm with legal counsel before relying on any row.
| Source | What it says | What it means for leadership |
|---|---|---|
| EU NIS 2 Directive, Articles 20 and 21 | Management bodies must approve and oversee the cybersecurity risk-management measures, can be held liable for breaches of Article 21, and must complete cybersecurity training. Article 21 includes policies and procedures to assess the effectiveness of those measures. | For entities in scope, leadership must be able to show how effectiveness is assessed, not only that measures exist. |
| EU Digital Operational Resilience Act, Article 24 | Financial entities other than microenterprises must run a risk-based resilience testing program, use independent testers, prioritize and remediate findings with internal validation that gaps are closed, and test systems that support critical or important functions at least once a year. | Independence, remediation validation and a regular testing cycle are written into the obligation. |
| US SEC cybersecurity disclosure rules | Public companies must describe annually their processes for assessing, identifying and managing material cybersecurity risks, and the board’s oversight and management’s role. Material incidents are reported on Form 8-K Item 1.05, generally within four business days of the materiality determination. | Statements about oversight and risk processes are public statements. The evidence behind them needs to hold up. |
| UK Cyber Governance Code of Practice | A voluntary code published to help boards and directors govern cyber risk across the organization. | A useful lens for board oversight, whether or not an entity is formally bound. |
| ISO/IEC 27001 | An information security management system built to the standard includes monitoring, measurement, analysis and evaluation, internal audit and management review. Certification by an accredited body adds independent confidence. | Management review is an assurance mechanism, and it is only as good as the data fed into it. |
| NIST Cybersecurity Framework 2.0 | The Govern function places strategy, oversight and policy for cyber risk in leadership’s remit, including using results to adjust the risk management approach. | Results of control testing should change decisions, not just fill a report. |
What a useful assurance pack contains
Leadership gets better assurance from a short pack on the controls that matter than from a long dashboard on all of them. For each priority control, the pack should show:
-
The risk the control addresses and the service or asset it protects
-
The highest level of evidence reached on the ladder above, and who produced it
-
The date and result of the last test run by someone who does not operate the control
-
Known exceptions, each with an owner and an expiry date
-
Failed tests and the status of fixes, including whether the fix has been verified
Ten to fifteen controls is a workable size for a first pack. Choose them by asking which failures would most raise the likelihood or impact of a serious incident.
Questions leadership should ask
-
Which ten controls would most increase the chance or impact of a serious incident if they failed, and how do we know each one works?
-
For each, what is the highest level of evidence we hold, and who produced it?
-
When was it last tested by someone who does not operate it?
-
What did that test find, and has the fix been verified?
-
What is excluded from scope, and who accepted that exclusion?
-
Which public or regulatory statements about our controls rely on these answers?
-
What would a test look like if we wanted the control to fail?
-
How quickly would we learn that a control had stopped working?
Decisions for the leadership team
-
Set the standard of evidence. Decide that critical controls need independent operating effectiveness testing or adversarial validation, and that attestation alone does not meet the bar.
-
Fund independent testing as a standing activity. Set scope by risk, not by what is easiest to test, and keep it independent of the teams that run the controls.
-
Decide how failures are reported. Failed tests should reach leadership as findings with an owner, a date and verification before closure, not be absorbed into a status color.
Use this briefing to strengthen assurance over security controls. A practical first step is to name the ten controls in question one and record the highest evidence level for each. Leadership teams that want help structuring the assurance pack and the test plan can start a conversation with Cybrilliant’s governance, risk and compliance specialists.
Related Reading
UK Cybersecurity Compliance: Turning Requirements Into Controls
Article on translating UK obligations into controls that can be evidenced.
US Cybersecurity Compliance: Building a Defensible Control Model
Article on structuring controls so they stand up to US regulatory scrutiny.
Control Design: Why Policies Alone Do Not Reduce Risk
Article on the distance between a written policy and a working control.
Field Note: Designing Security Controls Around Business Processes
A practitioner field note on fitting controls to how work is actually done.
Regulatory Change Management for CISOs
Article on keeping the control set aligned as obligations change.
Sources
-
European Union. Directive on measures for a high common level of cybersecurity across the Union (NIS 2 Directive), Articles 20 and 21. eur-lex.europa.eu
-
European Union. Regulation on digital operational resilience for the financial sector (DORA), Article 24. eur-lex.europa.eu
-
US Securities and Exchange Commission. SEC Adopts Rules on Cybersecurity Risk Management, Strategy, Governance, and Incident Disclosure by Public Companies. sec.gov
-
UK Government. Cyber Governance Code of Practice. gov.uk
-
International Organization for Standardization. ISO/IEC 27001, Information security management systems. iso.org
-
NIST. The NIST Cybersecurity Framework (CSF) 2.0. nist.gov/cyberframework
-
AICPA. System and Organization Controls (SOC) 2 reporting. aicpa-cima.com
-
The Institute of Internal Auditors. The IIA’s Three Lines Model. theiia.org
How we label evidence
Facts carry a source. Analysis and recommendations are marked as ours. Examples that are hypothetical are labeled as hypothetical.
Part of the solutionSecure & ComplyTurn regulatory duties into controls that hold up.See how we work →Related service
Regulatory Assurance
Use this to strengthen assurance over security controls.