How do you build a container image signing and verification pipeline with Trivy and Cosign?

Think of this pipeline as the pharmaceutical supply chain for prescription medications. Before a drug reaches a pharmacy shelf, it passes through manufacturing quality control, is sealed with a tamper-evident closure, receives a serialized barcode proving its origin, and each handler in the distribution chain verifies the seal and records the handoff. At the pharmacy, the pharmacist checks the seal, verifies the barcode against the manufacturer's database, and confirms the batch passed FDA testing before dispensing it to patients. If any seal is broken, any barcode fails verification, or testing records are missing, the drug is quarantined. The Trivy-plus-Cosign pipeline creates the same chain of trust for container images: scan for quality, sign for provenance, attest for compliance, and verify before deployment.

The pipeline begins with Trivy scanning the freshly built container image in CI/CD. For a service like payments-api, the build stage compiles the application and produces a container image. Before anything else happens, Trivy scans the image for vulnerabilities, misconfigurations, and exposed secrets. The scan results are evaluated against the organization's policy: no critical CVEs, no high CVEs without a documented VEX exception, no embedded credentials. If the image fails the scan, the pipeline stops and the image is never signed, ensuring that the signature acts as a quality seal. The scan results themselves are captured in a structured format (SARIF, CycloneDX, or a custom JSON schema) because they will be signed as an attestation in the next step, creating a cryptographic proof that this specific image passed this specific set of security checks.

Cosign keyless signing provides the cryptographic foundation without the operational burden of managing long-lived private keys. In a GitHub Actions workflow, the OIDC token that identifies the workflow run is exchanged with Sigstore's Fulcio certificate authority for a short-lived signing certificate. Cosign uses this certificate to sign the image digest, and the signing event is recorded in Sigstore's Rekor transparency log, creating a tamper-evident audit trail. The signature is stored as an OCI referrer artifact in the same container registry, linked to the image by its digest. For enterprise services like order-processing-service, checkout-service, and inventory-sync, each image built by the CI/CD pipeline receives a unique signature tied to the specific GitHub repository, workflow, and commit that produced it. This binding between identity and artifact is stronger than traditional key-based signing because there is no private key that can be stolen or shared.

Attestation adds a second layer beyond image signing. While the image signature proves who built the image, attestations prove what checks the image passed. After Trivy scanning completes and the image is signed, Cosign attest creates a signed in-toto attestation containing the scan results as the predicate. This means the Trivy scan report is cryptographically bound to the specific image digest and signed by the same CI/CD identity. Kubernetes admission controllers can then verify not only that the image was signed by an authorized pipeline but also that the attestation contains scan results meeting the organization's security policy. A Kyverno ClusterPolicy can extract fields from the attestation predicate, check that the vulnerability count at critical severity equals zero, verify that the scan was performed within the last 24 hours, and reject the pod if any condition fails.

The production complexity lies in the verification chain and failure modes. When Kyverno or OPA Gatekeeper verifies an image, it must contact the container registry to retrieve the signature and attestation, contact Rekor to verify the transparency log entry, and evaluate the attestation predicate against the policy. Each of these steps can fail due to network issues, registry rate limiting, or Rekor availability. Enterprise teams configure timeout handling, signature caching, and fallback policies that define behavior during verification outages. A common pattern is a break-glass mechanism where a designated security officer can temporarily annotate a namespace to bypass verification during incident response, with all bypass events logged and audited. Teams also implement pre-admission caching where a controller periodically verifies all images referenced by running workloads and caches the results, so that pod restarts during a verification service outage can proceed based on cached verification. For services like settlement-processor and notification-gateway, this ensures that legitimate scaling events and pod evictions do not cause outages because the signature verification infrastructure is temporarily unreachable.