Building a Self-Assembling Compliance Evidence Library

Industry estimates suggest most organizations spend roughly twelve working weeks per year gathering evidence for compliance audits (figure commonly attributed to the Vanta State of Trust report; the underlying full report is gated, so treat as a credible benchmark rather than a precisely sourced number). Security teams scramble in the months before an audit, digging through GitHub, Jira, spreadsheets, and Slack to reconstruct what happened.

This is insane. Compliance evidence should be collected automatically at the moment it's created — at build time, at deployment time, at code review time. By the time an auditor arrives, you should have a machine-generated evidence archive spanning 12+ months.

Organizations using autonomous coding agents have an additional compliance burden: proving that AI-generated code was reviewed before deployment. Building an evidence library becomes critical because auditors will ask for proof, and "we probably reviewed it" doesn't satisfy SOC 2, FedRAMP, or EU CRA.

This article covers the architecture and tooling for a self-assembling compliance evidence library.

The Problem: Compliance at Audit Time

Here's the current workflow at 95% of organizations:

Month 1–11: Engineering ships code, deploys, fixes bugs. Nobody thinks about compliance.

Month 12: Auditor sends questionnaire: "Show us your code review records, test coverage, vulnerability response times, deployment approvals, etc."

Week 1 of Audit: Compliance team starts digging:

"Let me pull all PRs from the past 12 months..."
"Who can run the test coverage report?"
"How do we query deployment logs?"
"Are SBOMs... something we should have?"

Week 2–4: Evidence collection scramble. Chaos. Overtime.

Result: Auditor finds gaps ("You don't have SBOM for version 2.13?" "Who approved this deployment?"). Organization pays for audit delays or fails controls.

Cost: ~12 weeks/year × 40 hours/week × $150/hr fully-loaded staff cost ≈ $72,000 per engineer per year spent in audit mode.

The Solution: Continuous Compliance (Evidence-as-Code)

Instead of gathering evidence at audit time, collect it continuously:

At build time: Generate SBOM, sign with Sigstore, publish to evidence repository.

At code review: Capture reviewer approval record, mark AI-generated code, store in evidence database.

At deployment: Log who deployed what, when, which approvals were satisfied, commit hash, artifact digest.

At vulnerability discovery: Record CVE, assessed severity, remediation timeline, patches deployed.

At release: Generate compliance attestation: "This release contains X components, Y were reviewed by humans, Z were generated by AI and reviewed by [names], all tests passed, all vulnerabilities remediated."

By the time the auditor arrives, you have an evidence archive that's complete, timestamped, and cryptographically signed.

What to Automate (Evidence Categories)

1. SBOM Metadata

Every build generates:

CycloneDX SBOM with component hashes
Timestamp (when SBOM was generated)
Build provenance (who triggered the build, which commit)
AI-attribution tags (which components are agent-generated)
Cryptographic signature (Sigstore attestation)

Storage: .well-known/sbom/ + centralized compliance repository

Auditor use: "Show me the SBOM for v2.14.1" → Served instantly with cryptographic proof

2. Code Review Records

Every code review generates:

PR identifier (GitHub, GitLab, Gitea)
Author (human or agent)
Reviewer name and approval timestamp
For AI-generated code: Explicit reviewer checkbox: "I reviewed this AI-generated code"
Diff (what changed)
Comments (why reviewer approved or requested changes)

Storage: Structured log format (JSON, queryable database)

Auditor use: "Show me code review for auth service changes in Q1" → Query returns all matching reviews

3. Test Results

Every build runs tests and stores:

Test coverage percentage (per module, per file)
SCA (Software Composition Analysis) results: found vulnerabilities, severity, remediations
SAST (Static Application Security Testing) results: security issues found and addressed
DAST (Dynamic testing) results for critical paths
Test execution time, pass/fail status

Storage: Structured reports, timestamped

Auditor use: "Show test coverage trend for auth module over 12 months" → Dashboard visualization

4. Deployment Records

Every deployment logs:

Artifact identifier (container image hash, version)
Deployment timestamp
Deployed-by (human who triggered, or automation account)
Approvals satisfied (change advisory board, security, ops)
Environment (staging, prod)
Rollback triggers (if any)

Storage: Immutable audit log

Auditor use: "Who approved the v2.13.5 deployment to production?" → Instant answer

5. Vulnerability Response

Every vulnerability discovery and fix records:

CVE identifier
Date discovered
Severity (CVSS score)
Component affected
Date patched
Patch version
Approvals for deployment
Deployment to production

Storage: Searchable index

Auditor use: "Show response time for critical vulnerabilities from 2025" → Metrics dashboard

6. Compliance Attestations

Periodically (every release, monthly, quarterly):

Generate compliance checklist attestation
"All components have SBOM: Yes"
"All code reviewed before merge: Yes (100%)"
"All AI-generated code reviewed explicitly: Yes"
"All vulnerabilities remediated within SLA: Yes"
"No unreviewed code in production: Yes"
Sign with organizational key

Storage: Timestamped and signed

Auditor use: "Show quarterly compliance attestations for 2025" → 4 signed documents

Architecture: Evidence Collection Pipeline

┌─────────────────────────────────────────────────────────┐
│                     CI/CD Pipeline                       │
│                                                          │
│  ┌───────────────────────────────────────────────────┐  │
│  │ Build Stage                                       │  │
│  │  ├─ Compile / Package                            │  │
│  │  ├─ Generate SBOM (cdxgen)                        │  │
│  │  ├─ Sign SBOM (cosign)                            │  │
│  │  └─ Emit: SBOM artifact event                     │  │
│  └────────────┬────────────────────────────────────┘  │
│               │                                         │
│  ┌────────────▼────────────────────────────────────┐  │
│  │ Test Stage                                       │  │
│  │  ├─ Run unit tests                               │  │
│  │  ├─ Generate coverage report                     │  │
│  │  ├─ SCA scan (detect vulns in deps)              │  │
│  │  ├─ SAST scan (detect security bugs)             │  │
│  │  └─ Emit: Test results event                     │  │
│  └────────────┬────────────────────────────────────┘  │
│               │                                         │
│  ┌────────────▼────────────────────────────────────┐  │
│  │ Compliance Stage                                │  │
│  │  ├─ Validate all evidence collected              │  │
│  │  ├─ Emit to compliance database                  │  │
│  │  ├─ Generate build attestation                   │  │
│  │  └─ Sign attestation (org key)                   │  │
│  └────────────┬────────────────────────────────────┘  │
│               │                                         │
└───────────────┼─────────────────────────────────────┘
                │
        ┌───────▼────────┐
        │ Evidence Store │
        ├─────────────────┤
        │ SBOM archive    │
        │ Test results    │
        │ Coverage trends │
        │ Vuln database   │
        │ Deployment log  │
        │ Attestations    │
        └────────────────┘

Example: GitHub Actions Integration

name: Build with Compliance Evidence

on:
  push:
    branches: [main]
    tags: ["v*"]

jobs:
  build:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
      id-token: write
    
    steps:
      - uses: actions/checkout@v4
      
      # Build
      - name: Build
        run: npm run build
      
      # Test
      - name: Run tests
        run: npm run test -- --coverage
        
      - name: Upload coverage
        uses: actions/upload-artifact@v3
        with:
          name: coverage
          path: coverage/coverage-final.json
      
      # SBOM
      - name: Generate SBOM
        run: |
          npx @cyclonedx/npm > sbom.cdx.json
          cosign attest --predicate sbom.cdx.json --type cyclonedx dist/
      
      # Compliance Evidence
      - name: Emit compliance evidence
        run: |
          node tools/emit-evidence.js \
            --build-id "${{ github.run_id }}" \
            --commit "${{ github.sha }}" \
            --sbom sbom.cdx.json \
            --coverage coverage/coverage-final.json \
            --target-url "${{ secrets.COMPLIANCE_API_URL }}" \
            --api-key "${{ secrets.COMPLIANCE_API_KEY }}"
      
      # Attestation
      - name: Generate compliance attestation
        if: startsWith(github.ref, 'refs/tags/')
        run: |
          node tools/generate-attestation.js \
            --version "${{ github.ref_name }}" \
            --evidence-store "${{ secrets.COMPLIANCE_API_URL }}" > attestation.json
          
          # Sign attestation
          gpg --sign --detach-sign --armor attestation.json
      
      - name: Upload attestation
        if: startsWith(github.ref, 'refs/tags/')
        uses: actions/upload-artifact@v3
        with:
          name: compliance-attestation
          path: |
            attestation.json
            attestation.json.asc

Code Review Integration

Add to your PR template:

## Compliance Checklist

- [ ] Code review completed and approved
- [ ] **AI-Generated Code:** This PR contains AI-generated code
  - Reviewer confirms: I have reviewed this AI-generated code specifically
  - Model: (e.g., Copilot 2026-q1-v4)
  - Prompt: (summarize what was asked of the agent)
- [ ] Test coverage >= 80% (see coverage report above)
- [ ] No SAST issues (green check below)
- [ ] No critical vulnerabilities in dependencies (SBOM scan clean)

<!-- Compliance robot will validate these checkboxes and emit evidence -->

A bot then:

Parses this checklist when PR is merged
Records reviewer name, timestamp, AI-generated flag
Emits to compliance database
Signs the record

Tools to Implement This

For SBOM Collection

cdxgen or syft — Generate from artifacts
cyclonedx-cli — Validate and manipulate SBOMs
Sigstore cosign — Sign SBOMs

For Test Results

npm test -- --coverage --json — Structured coverage output
npx snyk test --json — SCA results
npx eslint --format json or checkmarx, sonarqube — SAST results

For Compliance Database

Open source: PostgreSQL + JSON queries, Elasticsearch, InfluxDB
Commercial: Vanta, Laika, Drata (continuous compliance platforms)
DIY: Simple approach is JSON files in Git with GitHub as the database

For Code Review Records

GitHub API: GET /repos/{owner}/{repo}/pulls/{pull_number/reviews — Query all reviews
GitLab API: GET /projects/{id}/merge_requests/{mr_iid}/approvals
Gitea API: similar

For Deployment Records

Kubernetes audit logs — All deployments logged by default
Cloud provider audit logs (AWS CloudTrail, GCP Cloud Audit Logs, Azure Activity Log)
ArgoCD or Flux — Git-based deployments already logged

Orchestration

Simple: GitHub Actions / GitLab CI emit to an HTTP API (see above)
Complex: Custom controller that watches CI/CD events and Kubernetes audit logs

Evidence Retention and Compliance

The retention windows that actually appear in the source standards:

SOC 2 Type 2: retain evidence covering the audit period (commonly three to twelve months for an initial Type 2 report, then twelve months for renewals). The AICPA TSC do not impose a flat fixed-year retention rule.

FedRAMP: continuous-monitoring evidence retained for the duration of the authorization. Audit logs commonly retained for at least three years per NIST SP 800-53 AU-11 default; the FedRAMP overlay can extend this for specific control families.

EU CRA: technical documentation (which includes the SBOM and vulnerability handling evidence) retained per Article 31 for 10 years from when the product is placed on the market or for the support period, whichever is longer. Routine vulnerability handling evidence under Annex I, Part II also runs for the support period.

Pick the longest window that any of your obligations imposes, and design retention around that.

Implementation:

# Archive evidence monthly
tar czf evidence-2026-04.tar.gz evidence/
gpg --symmetric --armor evidence-2026-04.tar.gz
aws s3 cp evidence-2026-04.tar.gz.asc s3://compliance-archive/

# Retention policy
# Keep recent evidence (< 2 years) hot in database
# Archive older evidence to S3 Glacier / GCP Archive
# Audit trail: log every access to archived evidence

Compliance Automation Checklist

SBOM generation at every build (artifact-based, not lockfile)
Code review records captured to evidence database
Test coverage metrics published at every build
SCA and SAST results stored and queryable
Deployment logs captured from CI/CD or infrastructure
Vulnerability discovery and response tracked
AI-generated code explicitly flagged and reviewer tracked
Monthly/quarterly attestations generated and signed
Evidence retention policy defined against the longest applicable obligation (FedRAMP / EU CRA Article 31 typically dominate)
Audit trail on evidence access (compliance db logs all queries)

What Auditors Will Ask

SOC 2 Auditor:

"Show me 10 random PRs from the past 6 months with code review records"
- Response: Query dashboard, 10 records generated in seconds
"Did any code merge without review?"
- Response: No, dashboard shows 100% review rate, with filters for AI code

FedRAMP Auditor:

"Show me the SBOM for version 2.14.1"
- Response: Signed SBOM served from .well-known/sbom/, with Sigstore verification

EU CRA Inspector:

"How did you respond to CVE-2024-1234 in your dependencies?"
- Response: Evidence record shows: discovered 2024-02-15, patch released 2024-02-18 (3 days), deployed to prod 2024-02-20

References

Vanta State of Trust report — industry estimates suggest a typical compliance burden of roughly twelve working weeks per engineer per year (the full report is gated; treat the figure as a credible benchmark rather than a precisely-sourced number)
NIST SP 800-218 — SSDF v1.1 — Baseline for evidence requirements
NIST SP 800-218A — SSDF Generative AI / Dual-Use Foundation Model Profile — AI-relevant SSDF extension (July 2024)
CISA 2025 SBOM Minimum Elements (Draft for Comment, comment period closed Oct 2025) — What to collect (still a draft, not a finalised standard)
OpenPolicy Agent (OPA) / Rego — Policy-as-code for compliance rules
Kubernetes Audit Logging — Deployment evidence source