Gold Images for Incident Response: How to Verify That What's Running Is What You Built

An incident is happening. Your Kubernetes dashboard shows high latency in production. Your first instinct: "Did we deploy something risky?"

But before you can diagnose the root cause, you need to answer a more fundamental question: "Is what's running right now actually what we built?"

Image drift is the enemy. When production pods run modified versions of your images — whether by accident, by operator tweaking, or by active attack — incident diagnosis becomes impossible. You can't compare the running container against your source code because the running container isn't your code anymore.

Gold images (hardened, verified base images) are the foundation of answering that question quickly. Paired with cryptographic signing and policy enforcement, they let you answer "was this container tampered with?" in seconds.

The Image Drift Problem

Container images can drift in three ways:

1. Post-Build Modification

An operator SSH'd into a container and modified a file. The running container is now different from the built image, but deployment logs show the original image pushed to the registry.

# What the deployment manifest says
image: gcr.io/myorg/api:v1.2.3

# What's actually in that container
$ docker exec <container> cat /app/config.json
# Different from source repo!

2. Runtime Injection

An attacker (or misconfigured init container) injects code into a running container. The image hash hasn't changed, but the running process is executing different code.

3. Layer Mutation

The container registry was compromised and layers were overwritten. The image tag points to different content than it did yesterday.

All three break the assumption: "If I can prove the image digest, I can prove the code."

Gold images mitigate this by:

• Starting from known-good, hardened base images
• Removing unnecessary components (no shell, no package manager)
• Cryptographically signing every build
• Enforcing policies that only allow signed, approved images

What Are Gold Images?

A gold image is a pre-approved, hardened base image that serves as the foundation for application containers.

Characteristics:

• Minimal — Only runtime dependencies, nothing extra
• Hardened — No shell, no package manager, no unnecessary binaries
• Signed — Cryptographically signed by your build system
• Scanned — Regular vulnerability scanning, zero-drift baseline
• Approved — Explicitly blessed by your security team

Image Types and Their Incident Response Implications

For incident response: Distroless and Chainguard are superior because:

1. No shell = attacker can't modify files post-build
2. Fewer CVEs = less triage surface during CVE incidents
3. Signed artifacts = proof of provenance

Hardened Base Image Providers

Google Distroless

Google maintains distroless images with zero CVEs for common runtimes:

# VULNERABLE - Attacker can SSH in and modify files
FROM ubuntu:22.04
COPY myapp /app
CMD ["/app"]

# HARDENED - No shell, no tools to modify running container
FROM gcr.io/distroless/static-debian12
COPY --chown=nonroot:nonroot myapp /app
USER nonroot:nonroot
ENTRYPOINT ["/app"]

Available distroless images:

• gcr.io/distroless/static — Static binaries (Go, Rust)
• gcr.io/distroless/base — glibc + openssl
• gcr.io/distroless/java — OpenJDK
• gcr.io/distroless/python3 — Python 3
• gcr.io/distroless/nodejs — Node.js

During incident: If your service runs distroless, you know immediately: "No attacker can shell into this container. What we built is what's running."

Chainguard Images

Chainguard provides hardened, SBOM-included, signed images with daily CVE patching. Every image includes:

• Complete SBOM (queryable via cosign download attestation --predicate-type https://cyclonedx.org/bom)
• Sigstore signature (verifiable via cosign verify)
• Non-root user enforcement
• Near-zero CVEs (rapid patch cadence — Chainguard rebuilds and re-publishes on every upstream advisory)

FROM cgr.dev/chainguard/python:latest
COPY --chown=nonroot:nonroot app/ /app/
WORKDIR /app
CMD ["python", "main.py"]

During incident: cosign verify proves the image came from Chainguard's build system and hasn't been modified. cosign download attestation --predicate-type https://cyclonedx.org/bom retrieves the signed SBOM (cosign download sbom was deprecated in cosign 2.0 — see the cosign 2.0 release notes) so you know exactly what's inside.

Verifying Image Signatures During an Incident

When an incident fires, your first defensive action is to verify the running image:

# Find the running image digest
POD_IMAGE=$(kubectl get pod mypod -n prod \
  -o jsonpath='{.spec.containers[0].image}')
# Output: gcr.io/myorg/api@sha256:a1b2c3d4e5f6...

# Retrieve image digest from running pod
POD_DIGEST=$(kubectl get pod mypod -n prod \
  -o jsonpath='{.status.containerStatuses[0].imageID}' | \
  grep -o 'sha256:[^"]*')

# Verify the image signature using cosign
cosign verify gcr.io/myorg/api@$POD_DIGEST \
  --certificate-identity-regexp=".*" \
  --certificate-oidc-issuer-regexp=".*" \
  2>&1 | tee image_verification.log

# If verification passes:
# ✓ Image was built by your CI/CD system
# ✓ Image has not been modified since it was signed
# ✓ Image is the exact one you intended to deploy

# If verification fails:
# ✗ Image is tampered with, or
# ✗ Image was not signed by your build system, or
# ✗ Image signature is forged
# → IMMEDIATE CONTAINMENT: isolate the pod

Detecting Image Drift with Policy Enforcement

Use Kyverno or OPA Gatekeeper to enforce that only signed, approved images can run:

Kyverno Policy: Allow Only Chainguard Images

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: only-approved-images
spec:
  validationFailureAction: Enforce  # Prevent pod creation
  rules:
    - name: verify-chainguard-signature
      match:
        any:
          - resources:
              kinds:
                - Pod
      verifyImages:
        - imageReferences:
            - "cgr.dev/chainguard/*"
          required: true
          attestors:
            - entries:
                - keys:
                    publicKeys: |-
                      -----BEGIN PUBLIC KEY-----
                      MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE...replace-with-chainguard-public-key...
                      -----END PUBLIC KEY-----
                    signatureAlgorithm: sha256

Schema notes (per the Kyverno verifyImages docs):

• attestors[].entries[].keys.publicKeys accepts either an inline PEM block (shown above) or a Kubernetes Secret reference via keys.secret: { name: <secret>, namespace: <ns> }.
• signatureAlgorithm lives inside the keys block, not at the rule root.
• validationFailureAction accepts only Enforce or Audit (capitalised); block, enforce, and other values cause the policy to be rejected at admission.
• For keyless (Sigstore OIDC) verification of public images such as Chainguard's, replace the keys: block with keyless: { subject: "...", issuer: "..." }.

Effect: Only Chainguard images can be deployed. Any attempt to run a modified image fails at pod creation time.

OPA Gatekeeper: Block Unsigned Images

OPA Gatekeeper requires Rego to be packaged inside a ConstraintTemplate, then instantiated by a Constraint that scopes it to the kinds you want enforced. The pair below is deployable as-is.

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8srequiredimages
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredImages
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8srequiredimages

        violation[{"msg": msg}] {
          container := input.review.object.spec.containers[_]
          image := container.image
          not startswith(image, "gcr.io/myorg/")
          not startswith(image, "cgr.dev/chainguard/")
          msg := sprintf("Image not from approved registry: %v", [image])
        }

        violation[{"msg": msg}] {
          container := input.review.object.spec.containers[_]
          image := container.image
          startswith(image, "gcr.io/myorg/")
          not contains(image, "@sha256:")  # Must use image digest, not tag
          msg := sprintf("Image must use digest pinning: %v", [image])
        }
---
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredImages
metadata:
  name: require-approved-pinned-images
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
  enforcementAction: deny

Effect: Only approved registries allowed. All images must use digest pinning (sha256:...) to prevent tag tampering.

Multi-Stage Builds for Minimal Images

Go

# Build stage
FROM golang:1.21-alpine AS builder
WORKDIR /build
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -o app .

# Runtime stage: distroless (no shell, no attack surface)
FROM gcr.io/distroless/static-debian12
COPY --from=builder /build/app /app
USER nonroot:nonroot
ENTRYPOINT ["/app"]

Incident response advantage: The distroless image has zero CVEs. No shell. No tools to modify files. If the app is misbehaving, it's the application logic, not environmental modification.

Node.js

FROM node:20-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production

FROM gcr.io/distroless/nodejs20-debian12
COPY --from=builder /app /app
WORKDIR /app
CMD ["server.js"]

Python

FROM python:3.12-slim AS builder
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt --target=/deps

FROM gcr.io/distroless/python3-debian12
COPY --from=builder /deps /deps
COPY app.py /app/
ENV PYTHONPATH=/deps
WORKDIR /app
CMD ["app.py"]

Gold Image Selection Matrix

Incident Response Runbook: Image Verification

Add this to your incident playbook:

PRODUCTION ISSUE - IMAGE VERIFICATION
======================================

1. Identify the service and pod
   kubectl get pods -n production | grep <service>
   POD=<pod_name>

2. Extract the running image digest
   DIGEST=$(kubectl get pod $POD -n production \
     -o jsonpath='{.status.containerStatuses[0].imageID}' | \
     grep -o 'sha256:[^"]*')
   
3. Verify image signature
   cosign verify gcr.io/myorg/<service>@$DIGEST
   
   If PASS → Image is trusted, proceed to code review
   If FAIL → Image tampered with
       → CONTAINMENT: kubectl delete pod $POD
       → ESCALATION: Security + Engineering
       → FORENSICS: Pull logs, collect evidence

4. Compare image to source
   Download SBOM from running image:
   cosign download attestation \
     --predicate-type https://cyclonedx.org/bom \
     gcr.io/myorg/<service>@$DIGEST \
     | jq -r '.payload' | base64 -d | jq '.predicate' > running.sbom.json
   
   Compare to built SBOM:
   diff running.sbom.json artifacts/sbom-v1.2.3.json
   
   If SAME → Image matches source
   If DIFFERENT → Investigate drift
       → Where did extra packages come from?
       → Who modified the image?

5. Proceed with incident diagnosis based on findings

Gold Image Migration Checklist

• Audit current base images across organisation
• Identify runtime requirements per language and service
• Create multi-stage Dockerfiles for each service
• Test with distroless or hardened images locally
• Verify signature and SBOM availability
• Update CI/CD pipelines with image signing step
• Deploy Kyverno or OPA policies to enforce approved registries
• Configure image digest pinning (no :latest or :v1 tags)
• Document debugging procedures for engineers
• Set up continuous monitoring for drift and new CVEs
• Schedule quarterly reviews of gold image updates

References

• Google Distroless Images
• Chainguard Images
• Sigstore: Secure Supply Chain Authentication
• Kyverno Image Verification Policies
• OPA Gatekeeper
• SLSA Framework: Supply Chain Levels