Would Your Security Evidence Survive an Assessment?

TL;DR: An assessment does not grade your security. It grades your evidence. A control that works but cannot be proven is scored the same as one that does not exist. Assessors examine, interview, and test; they weigh evidence by its quality, not its quantity; and they distinguish a control that is well designed from one that actually operated over time. This article explains how assessors really judge evidence, what separates proof that holds up from a screenshot that does not, and why AI now sits on both sides of the problem.

The gap between working and provable

A security team can deploy a control, watch it run for a year, and still fail the assessment on that control. The tool was on. The configuration was correct. The protection was real. And the finding still reads "not met," because when the assessor asked for proof, there was none to give.

This is the uncomfortable truth of every framework. As the N-able analysis of CMMC assessments puts it, Level 2 "certifies what you can prove, not what you intended." The same logic runs through SOC 2, ISO 27001, FedRAMP, and PCI. The assessment is not a measurement of your security posture. It is a measurement of your ability to demonstrate it to someone who starts from doubt.

An assessor does not score the control you run. They score the evidence you can produce for it.

How assessors actually look

Assessors do not simply ask whether a control exists. The NIST methodology that underpins FedRAMP and CMMC, SP 800-53A, defines three assessment methods, and each targets a different kind of object.

Two attributes then govern how hard they look. Depth is the rigor applied, from a basic review of a document up to a detailed analysis of how a mechanism behaves. Coverage is the breadth, from a basic sample of systems up to a comprehensive one. The practical consequence is sampling. An assessor rarely checks one server; they check a representative set, and, as that same analysis notes, "if implementation is inconsistent across in-scope systems, sampling will surface those gaps." Evidence that exists for your showcase host and nowhere else does not survive coverage.

Two questions, not one

Every control is judged against two separate questions, and most failed evidence answers only the first. Is the control designed to meet the objective, and did it operate effectively over time?

The distinction is formal. A SOC 2 Type 1 report covers a single date and attests that controls "exist and are suitably designed." A Type 2 report tests operation across a period of months, pulling samples that span the window. The blueprint proves capability; only the artifacts prove behavior. A policy document proves design. It proves nothing about operation. For that you need a trail that stretches across the whole period under review.

What makes evidence hold up

Auditors have a precise vocabulary for evidence quality, and it is worth borrowing because it predicts exactly what will be challenged. The U.S. Public Company Accounting Oversight Board's standard on audit evidence, AS 1105, separates two properties.

Sufficiency is quantity. Appropriateness is quality, and quality is made of relevance and reliability. The sentence every evidence program should memorize is the standard's warning that "obtaining more of the same type of audit evidence cannot compensate for the poor quality." You cannot bury a weak artifact under a hundred copies of itself.

Reliability is a hierarchy, and the same standard ranks it. Evidence from an independent source outranks internal assertions. Evidence the assessor obtains directly outranks what you hand them. Original records outrank copies. And evidence produced by a system with effective controls over it outranks a loose export. A screenshot pasted into a document sits near the bottom of that hierarchy. A signed, timestamped log pulled directly from a system of record sits near the top. The auditing standards converge on this. ISO 19011, the international standard for management system audits, defines audit evidence as records, statements of fact, or other information that are relevant to the audit criteria and verifiable. Relevant and verifiable is the whole test, and it is the one every artifact in this article is built to pass.

The three kinds of evidence

Mature assessments expect evidence in three layers, and the gap that sinks most programs is the third. A CMMC analysis frames them cleanly.

Documentary evidence answers "what is your process?" A policy, a standard, a plan. Configuration evidence answers "how is the system set up?" A ruleset, an enforcement setting, an export of the current state. Operational evidence answers "can you prove this actually happened, over time?" Tickets, review records, logs with a documented review, investigation notes.

Most organizations are strong on the first two and thin on the third. The analysis is blunt: "configuration screenshots alone are rarely sufficient for controls that require demonstration of ongoing process execution." An endpoint tool that runs all year but produces no detection history, no signature-update record, and no incident notes is scored "not met," not because the protection was absent, but because the evidence was never generated.

Anatomy of evidence that survives

A defensible artifact is not just a file. It carries the context an assessor needs to trust it without trusting you. Strip any of these away and the artifact invites a follow-up question, which is the thing a smooth assessment is trying to avoid.

The test is simple. Hand the artifact to someone who was not in the room when it was collected. Can they tell what control it supports, when it was captured, which systems it covers, where it came from, and who produced it? If any answer requires your narration, the evidence is not yet self-sufficient.

Generate provenance, do not promise it

The most reliable way to make evidence trustworthy is to stamp it with its own provenance at the moment of collection, rather than reconstructing the story later. The script below wraps any artifact in a manifest that records the control it supports, when and by whom it was collected, the scope it covers, the source system, and a cryptographic hash that proves the file has not changed since.

#!/usr/bin/env python3
# Wrap an evidence artifact in a manifest that carries its own provenance.
# The hash lets anyone, including the assessor, prove the file is unaltered.
import hashlib
import json
import sys
from datetime import datetime, timezone

def sha256(path: str) -> str:
    h = hashlib.sha256()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(65536), b""):
            h.update(chunk)
    return h.hexdigest()

def manifest(path, control_id, scope, source_system, collector):
    return {
        "control_id":     control_id,          # e.g. "AU.L2-3.3.1"
        "artifact":       path,
        "sha256":         sha256(path),         # integrity: proves no change
        "collected_at":   datetime.now(timezone.utc).isoformat(),
        "scope":          scope,                # the systems this covers
        "source_system":  source_system,       # where it came from
        "collector":      collector,            # the human responsible
    }

if __name__ == "__main__":
    # usage: evidence_manifest.py <file> <control_id> <scope> <source> <collector>
    m = manifest(sys.argv[1], sys.argv[2], sys.argv[3], sys.argv[4], sys.argv[5])
    out = sys.argv[1] + ".manifest.json"
    with open(out, "w") as f:
        json.dump(m, f, indent=2)
    print(f"Wrote {out}")
    print(json.dumps(m, indent=2))

#!/usr/bin/env python3
# Wrap an evidence artifact in a manifest that carries its own provenance.
# The hash lets anyone, including the assessor, prove the file is unaltered.
import hashlib
import json
import sys
from datetime import datetime, timezone

def sha256(path: str) -> str:
    h = hashlib.sha256()
    with open(path, "rb") as f:
        for chunk in iter(lambda: f.read(65536), b""):
            h.update(chunk)
    return h.hexdigest()

def manifest(path, control_id, scope, source_system, collector):
    return {
        "control_id":     control_id,          # e.g. "AU.L2-3.3.1"
        "artifact":       path,
        "sha256":         sha256(path),         # integrity: proves no change
        "collected_at":   datetime.now(timezone.utc).isoformat(),
        "scope":          scope,                # the systems this covers
        "source_system":  source_system,       # where it came from
        "collector":      collector,            # the human responsible
    }

if __name__ == "__main__":
    # usage: evidence_manifest.py <file> <control_id> <scope> <source> <collector>
    m = manifest(sys.argv[1], sys.argv[2], sys.argv[3], sys.argv[4], sys.argv[5])
    out = sys.argv[1] + ".manifest.json"
    with open(out, "w") as f:
        json.dump(m, f, indent=2)
    print(f"Wrote {out}")
    print(json.dumps(m, indent=2))

The hash is the part that matters most. It lets anyone, including the assessor, confirm that the file they are looking at is byte-for-byte the one that was collected, on the date claimed. Evidence that can prove it has not been altered is evidence that starts from a position of strength. Integrity is not independence, though. A hash proves the file is the one collected on the date claimed; it cannot prove the person who collected it was impartial. That is why evidence an assessor gathers or reproduces directly still outranks even a well-stamped artifact you hand over.

Prove operation, not just a moment

Operating effectiveness is where point-in-time evidence fails hardest. A screenshot shows that a control was enabled when you looked. It says nothing about any other day in the period. The query below turns a log table into evidence of continuous operation: it reports, for every day in the assessment period, whether the control produced the activity it should, and it lists the days it did not.

-- Operating-effectiveness evidence: did the control run every day of the period?
-- Generates one row per calendar day and flags days with no activity (the gaps).
-- PostgreSQL. Point it at the log table that records the control's output.
WITH period AS (
    SELECT generate_series(
        DATE '2026-07-01',            -- assessment period start
        DATE '2026-09-30',            -- assessment period end
        INTERVAL '1 day'
    )::date AS day
),
daily AS (
    SELECT date_trunc('day', event_time)::date AS day,
           COUNT(*) AS events
    FROM control_activity
    WHERE control_id = 'AU.L2-3.3.1'  -- audit logging, as one example
    GROUP BY 1
)
SELECT p.day,
       COALESCE(d.events, 0) AS events,
       CASE WHEN COALESCE(d.events, 0) = 0 THEN 'GAP' ELSE 'ok' END AS status
FROM period AS p
LEFT JOIN daily AS d ON d.day = p.day
ORDER BY p.day;

-- Operating-effectiveness evidence: did the control run every day of the period?
-- Generates one row per calendar day and flags days with no activity (the gaps).
-- PostgreSQL. Point it at the log table that records the control's output.
WITH period AS (
    SELECT generate_series(
        DATE '2026-07-01',            -- assessment period start
        DATE '2026-09-30',            -- assessment period end
        INTERVAL '1 day'
    )::date AS day
),
daily AS (
    SELECT date_trunc('day', event_time)::date AS day,
           COUNT(*) AS events
    FROM control_activity
    WHERE control_id = 'AU.L2-3.3.1'  -- audit logging, as one example
    GROUP BY 1
)
SELECT p.day,
       COALESCE(d.events, 0) AS events,
       CASE WHEN COALESCE(d.events, 0) = 0 THEN 'GAP' ELSE 'ok' END AS status
FROM period AS p
LEFT JOIN daily AS d ON d.day = p.day
ORDER BY p.day;

Output like this answers the operating-effectiveness question directly, because it covers the entire period and surfaces its own gaps. An assessor who sees a clean daily series across the window has little left to sample. An assessor who sees you hiding the gaps has found their finding for you, so show them. One caveat the query cannot cover on its own: a daily series proves the control produced activity, which is necessary but not sufficient. For any control that turns on human judgment, the second half of operating effectiveness is the review trail, the evidence that someone actually read the output and acted on it. That has to be collected too.

Where evidence falls apart

Most assessment findings trace to a small set of evidence failures. Each one is a specific piece of missing context or a mismatch an assessor is trained to catch.

The most expensive of these is the mismatch between what you documented and what you run. Lazarus Alliance, cataloguing the most misunderstood FedRAMP requirements, reports "idealized architecture drawings that assessors cannot reconcile with production evidence," and advise teams to "walk the diagrams against production before assessment kickoff." An assessment is not the time to discover that your network diagram describes a system you decommissioned two quarters ago.

The AI angle: both a threat and a tool

AI has arrived on both sides of the evidence table, and treating only one side is a mistake.

AI is eroding the default trust in evidence. In June 2026, ISACA published a warning titled "The Audit Evidence Crisis," arguing that generative models can now produce documents "completed with synthetic signatures and fabricated metadata" that "appear authentic." Its conclusion is a shift in posture: "auditors can no longer afford to start with trust," and must "validate the authenticity of evidence before relying on it." The practical effect for the assessed is that a loose screenshot or an exported document, always weak evidence, is now actively suspect. The premium on evidence an assessor can obtain directly from a system, and verify independently, has gone up.

AI is also a powerful evidence engine, used carefully. The same models are genuinely good at the preparation work: mapping a pile of policies and configurations to control objectives, flagging the controls with no operational evidence behind them, and drafting the narrative that ties an artifact to the requirement it satisfies. That is real leverage. The guardrail is provenance. Let AI find the gaps and organize the package, but never let it generate the artifact. Evidence should be collected from the system of record, hashed, and traceable to a human and a source, so that nothing in the package is something a model produced from a prompt.

Why evidence decays

Even good evidence does not stay good. An artifact that was defensible when it was collected can quietly become a finding months later, and the decay is usually invisible until an assessor points at it.

The common thread is drift. The system changes but the diagram does not. The policy is updated but the screenshots still show the old setting. A reviewer leaves and the quarterly sign-off stops happening, but no one notices until the sample lands on the missing quarter. Treat evidence as perishable: give each artifact a useful life, re-collect it on a schedule that matches the control's cadence, and reconcile your documentation against production often enough that the two never drift far apart. Evidence you refresh on purpose is evidence that is always close to assessment-ready.

Run the readiness test before they do

The cheapest assessment finding is the one you catch yourself. Before an assessor arrives, run your own evidence through the same questions they will, control by control.

Pick a sample of your controls the way an assessor would, including the unglamorous ones, and ask for the evidence cold. Do not let the person who built the control narrate it. If the artifact cannot stand on its own, if it has no date, no scope, no source, or only proves a single moment, you have found a finding while it is still free to fix.

Build the evidence, not just the control

The work does not end when a control is deployed. A control that generates no durable, contextual, period-spanning proof of its own operation is a control you will have to defend with a story, and stories do not survive a skeptical assessor.

Design the evidence alongside the control. Make it carry its own context, generate it continuously, stamp it with provenance, and test it against the two questions an assessor will ask: was this designed to work, and did it actually operate? Do that, and the assessment stops being an interrogation and becomes a formality, which is exactly what a well-run assessment should be.

Key takeaways

• An assessment scores your evidence, not your security. A control you cannot prove is scored as absent.
• Assessors examine, interview, and test, and they sample across systems, so evidence must exist everywhere the control does.
• Answer both questions: design effectiveness at a point in time, and operating effectiveness across the whole period.
• Quality beats quantity. More copies of a weak artifact cannot replace one reliable, independently verifiable record.
• Expect three layers: documentary, configuration, and operational. The operational layer is the one most programs lack.
• AI makes loose evidence suspect and strong provenance essential. Let AI find gaps and organize, never generate the artifact.

References

• NIST: SP 800-53A assessment cases, methods, objects, depth and coverage
• PCAOB: AS 1105, Audit Evidence
• Agency: Design vs. operating effectiveness (SOC 2)
• N-able: The CMMC evidence problem, why deployed controls fail assessment
• Lazarus Alliance: The most misunderstood FedRAMP requirements observed in assessments
• ISACA: The audit evidence crisis, how AI deepfakes are rewriting assurance standards (June 2026)

Figures are drawn from government, standards-body, and research publications and may be revised. Code examples are illustrative; adapt them to your environment and test them before relying on them.

If you found this useful, follow me for more on security assurance and AI-era risk, and share it with the person who owns your next audit.