Supply Chain Security in GitLab CI: SBOM, cosign, and the Policy That Blocks

TL;DR A pipeline that scans images and writes an SBOM, with nothing checking either at admission, had been green for a year and had stopped nothing. Four steps below: an SBOM Trivy can read back, a cosign signature on the digest with the key in Vault, the SBOM attached as an attestation, and a Kyverno policy that refuses the rest. Then a table of which SLSA Build L3 requirements a self-hosted GitLab runner meets, and the two it doesn’t.

I inherited a pipeline on a client’s GitLab, self-managed on their own hardware, and on paper it did everything right.

Trivy scanned every image. A second job wrote an SBOM , added the quarter a customer’s security questionnaire asked whether they produced one. Both jobs were green, and had been for about a year. Both dropped a JSON file into GitLab’s artifact storage, where it sat for thirty days and expired. I couldn’t find anyone who’d opened one.

Meanwhile the cluster would start anything. Any image from any registry, as long as whoever pushed it held a token. The scan reported on something that had already shipped, and the SBOM was a file with no reader.

Nobody had done anything wrong, and I want to say that before the rest of this. Every job in that file was added by somebody solving the problem their own ticket described, and each one worked. The scan job found vulnerabilities. The SBOM job produced an SBOM. What was missing is the step that turns either of them into a decision, and that step was in nobody’s ticket, so nobody wrote it. By the time I was reading the file, both people whose names were on those commits had moved on.

That’s the part that itches, and I’ll admit it’s an occupational habit. I do shadow-CISO work for clients often enough, and the holes I find there are rarely inside a job somebody wrote. They sit in the space between two tickets, where there’s no owner and both sides assume the other one covers it. An auditor sees two green jobs and ticks the box. The cluster never sees either job.

The gap

Here’s what that adds up to. The supply chain was described in a document, and nothing made the cluster act on it.

Three holes, and I’d argue they’re one:

1. The SBOM had no consumer. It answers “which of my images contain this library” only if something can find it later, and a job artifact that expires after thirty days isn’t findable.
2. Nothing tied the image to the pipeline. The registry held a tag. Anyone with push rights can move a tag. Nothing in the image said “this came out of that pipeline, from that commit”.
3. The cluster verified nothing. Kyverno was running with a stack of governance policies about resource limits and namespaces, and not one of them cared where an image came from.

There’s a compliance version of this too, and it’s worse because it sounds fine. A page in the internal wiki said the platform targets SLSA level 3. It had a date and no author. Nobody still there could tell me which of that specification’s requirements a GitLab runner actually meets, and neither could I until I sat down with it. I’ll get to that further down. The short version is that it doesn’t meet all of them, and one it structurally can’t.

So what would actually stop an unsigned image from starting?

Four moving parts. The first three are preparation, and the fourth is a gate that says no. What follows assumes you already have a GitLab pipeline that builds an image and deploys it; every job below slots into that.

Step one: an SBOM worth consuming

Start with Trivy, which is probably already in your pipeline. The command is short and it has one trap in it:

trivy image --format cyclonedx --output sbom.cdx.json "$IMAGE"

Run that and Trivy prints a line most people scroll past:

"--format cyclonedx" disables security scanning. Specify "--scanners vuln"
explicitly if you want to include vulnerabilities in the CycloneDX report.

So the default gives you an inventory with no findings in it. I think that’s the right default. An SBOM is a parts list, a scan is a judgement about that list on a given day, and they go stale at different speeds. It’s also not what most people think they’re getting. If you want both in one document:

trivy image --scanners vuln --format cyclonedx --output sbom.cdx.json "$IMAGE"

I generate the inventory without findings and scan separately. The parts list is a fact about the image and stays true for as long as the image exists. The vulnerability list is true on Tuesday and wrong on Thursday, so it belongs in a scheduled job that rescans what’s already deployed, not welded into the artefact.

The other half of “worth consuming” is that Trivy reads an SBOM back as a scan target:

trivy sbom sbom.cdx.json

That’s the loop that makes the file worth producing. Six months from now, when a CVE lands in a library you’ve never heard of, you scan the SBOMs of everything currently running instead of rebuilding and rescanning every image.

Step two: sign the digest, not the tag

cosign attaches a signature to an image in the registry. The subtlety is what you point it at.

IMAGE_DIGEST=$(docker inspect --format='{{index .RepoDigests 0}}' "$IMAGE_URI")
cosign sign "$IMAGE_DIGEST"

Signing myapp:1.4.2 signs a name. Names move, even when the version number falls out of your commits and nobody picks it by hand, because a tag is still a pointer and a pointer is still writable. Signing myapp@sha256:9f86d0... signs one specific pile of bytes, and that’s the only version of this that means anything. GitLab’s own tutorial resolves the digest out of docker inspect for exactly this reason, and their documentation adds one more rule I’d follow: build and sign in the same job, so nothing gets a window to touch the artefact in between.

Where the key lives, and why there

This is the decision the post exists for, so I’ll lay out all three and say which one we landed on.

Option one: a private key in a masked CI/CD variable. It works, it takes ten minutes, and it’s the one almost everybody starts with. The problem is that the key is now an environment variable inside a build job, which means every script in that pipeline can read it, including a dependency’s install hook. Rotating it means re-signing history or accepting a break in the chain.

Option two: the key stays in Vault and never leaves. cosign speaks KMS, and HashiCorp Vault’s transit engine is one of the backends:

export VAULT_ADDR=https://vault.example.com
vault secrets enable transit
cosign generate-key-pair --kms hashivault://cosign
cosign sign --key hashivault://cosign "$IMAGE_DIGEST"

The runner never holds the private key. It holds a Vault token that’s allowed to ask for a signature, which is a much smaller thing to steal. Sigstore’s documented policy for this grants read on transit/keys/cosign, update on transit/hmac/cosign/* and transit/sign/cosign/*, and create on transit/verify/cosign, and nothing else. openbao:// is the sibling scheme and behaves the same, so this option survives the move off Vault if you ever make it. Mine is on the list. If you already run Vault or OpenBao for your cluster secrets, this is a config change and no new component.

Option three: keyless, with Sigstore. No long-lived key at all. GitLab hands cosign an OIDC token, Fulcio issues a certificate that lives for ten minutes, cosign signs with it and throws the key away, and the whole event is recorded in Rekor . The identity in that certificate is your project and your CI config file, which is a far better claim than “someone who had the key”.

On GitLab.com the whole of it is the id_tokens block, and it goes in the job that already builds:

build:
  stage: build
  image: docker:cli
  services:
    - docker:dind
  variables:
    COSIGN_YES: "true"
    IMAGE_URI: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
  id_tokens:
    SIGSTORE_ID_TOKEN:
      aud: sigstore
  before_script:
    - apk add --no-cache cosign
    - docker login -u gitlab-ci-token -p "$CI_JOB_TOKEN" "$CI_REGISTRY"
  script:
    - docker build -t "$IMAGE_URI" .
    - docker push "$IMAGE_URI"
    - IMAGE_DIGEST=$(docker inspect --format='{{index .RepoDigests 0}}' "$IMAGE_URI")
    - cosign sign "$IMAGE_DIGEST"

One job, because IMAGE_DIGEST is a shell variable and a second job gets a different shell. Split the signing out and you’re passing the digest through a dotenv report, and the image sits in the registry unsigned for however long the pipeline takes to reach the next stage.

cosign picks SIGSTORE_ID_TOKEN out of the environment on its own. COSIGN_YES replaces the interactive confirmation, since a pipeline has nobody to confirm anything.

Now the part the tutorials skip. On a self-hosted GitLab, keyless costs you three services. The public Sigstore deployment only trusts GitLab.com pipelines, so GitLab’s own documentation for self-managed instances tells you to run your own Fulcio, your own Rekor, and your own Certificate Transparency log. The CT log isn’t optional, because a Fulcio without one issues certificates cosign refuses to verify. Rekor needs a persistent signing key, because an in-memory signer regenerates on restart and invalidates everything logged before it.

We costed this on the client’s self-managed instance and took the key in Vault instead. Three more stateful services, one of them an append-only log where losing the key is unrecoverable, is a real bill even for a team with an operations budget, and what it buys is a better identity story on top of signing that already works. Sovereignty means making that call with the costs visible, and here the simpler option was the honest one. If you’re on GitLab.com, take the keyless path. You get the strictly better identity for four lines of YAML.

Two gotchas from that page if you do build the stack. Container image signing needs cosign v3 and a registry on a resolvable HTTPS hostname, so an instance whose registry_external_url is a bare IP can sign files and can’t sign images. And Fulcio’s /healthz reports healthy even when its OIDC provider failed to initialise, which is a fun afternoon.

Verifying, and the double slash

Whichever option you pick, the verify command is where the identity shows up:

cosign verify "$IMAGE_DIGEST" \
  --certificate-identity "https://gitlab.com/my-group/my-project//.gitlab-ci.yml@refs/heads/main" \
  --certificate-oidc-issuer "https://gitlab.com"

That double slash between the project path and the CI config path is deliberate and required. Write one slash and you get:

Error: none of the expected identities matched what was in the certificate, got subjects

which tells you nothing about the actual mistake. GitLab documents it. I still lost twenty minutes to it.

One more constraint worth knowing before you plan around it. The id_tokens block has to live in the project being built. Auto DevOps, CI files included from another repository, and child pipelines don’t work, which GitLab tracks as Support keyless signing when CI config is located outside of the project. If your organisation centralises pipeline templates, that’s a design constraint and you want to know it before the templates exist.

Step three: attach the SBOM to the image

Signing proves an image came from your pipeline. Attesting attaches a signed statement about that image, and the SBOM is exactly such a statement. Trivy documents the pairing:

trivy image --format cyclonedx -o sbom.cdx.json "$IMAGE_DIGEST"
cosign attest --key hashivault://cosign --type cyclonedx --predicate sbom.cdx.json "$IMAGE_DIGEST"

The SBOM now travels with the image, in the registry, signed by the same key. Pull the image anywhere and the parts list comes with it. Compare that with a JSON file in artifact storage and thirty days to live.

Read it back with:

cosign verify-attestation --key cosign.pub --type cyclonedx "$IMAGE_DIGEST" > sbom.intoto.jsonl
trivy sbom ./sbom.intoto.jsonl

Two practical notes. cosign attest writes to the OCI registry, so the job needs push permission on that repository, and --no-upload exists if you only want to look at the statement. And only CycloneDX-type attestations can be fed back to trivy sbom, so if you were going to pick SPDX for some other reason, pick it knowing that.

Step four: make the cluster refuse

Everything so far produces evidence. This is the part that acts on it.

A note on which Kyverno API to write against, because this changed recently and it changes what you should type. The ClusterPolicy kind with its verifyImages rule is deprecated as of Kyverno v1.19 and slated for removal in v1.20. The replacement is ImageValidatingPolicy, which reached stable in the same release and expresses its checks in CEL. If you write a new policy today, write this one.

Start in audit mode. I mean it, every time.

apiVersion: policies.kyverno.io/v1
kind: ImageValidatingPolicy
metadata:
  name: require-signed-images
spec:
  validationActions: [Audit]
  failurePolicy: Fail
  webhookConfiguration:
    timeoutSeconds: 15
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods"]
  matchImageReferences:
    - glob: "registry.example.com/platform/*"
  attestors:
    - name: pipeline
      cosign:
        key:
          data: |
            -----BEGIN PUBLIC KEY-----
            MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE...
            -----END PUBLIC KEY-----
  validations:
    - expression: >-
        images.containers.map(image,
          verifyImageSignatures(image, [attestors.pipeline])).all(e, e > 0)
      message: "Image is not signed by the platform pipeline"

The public key comes straight out of Vault, and it’s public, so pasting it into a policy is fine:

cosign public-key --key hashivault://cosign

Note matchImageReferences is a glob on your own registry and no *. Scope it deliberately. Everything you didn’t build is going to fail this check, starting with the ingress controller.

Leave it on Audit and go read the policy reports for a week. You’ll find images you forgot you were running. That week is the whole point of audit mode, and skipping it is how a policy rollout becomes an outage.

Then verify the SBOM too, which is the step that closes the loop I opened at the top. Add an attestations: block, and grow the validations: list you already have from one entry to three. There’s exactly one validations: key in a policy, so these replace it instead of sitting beside it:

  attestations:
    - name: sbom
      intoto:
        type: https://cyclonedx.org/bom
  validations:
    - expression: >-
        images.containers.map(image,
          verifyImageSignatures(image, [attestors.pipeline])).all(e, e > 0)
      message: "Image is not signed by the platform pipeline"
    - expression: >-
        images.containers.map(image,
          verifyAttestationSignatures(image, attestations.sbom, [attestors.pipeline])).all(e, e > 0)
      message: "Image has no signed SBOM attestation"
    - expression: >-
        images.containers.map(image,
          extractPayload(image, attestations.sbom).predicate.bomFormat == 'CycloneDX').all(e, e)
      message: "SBOM attestation is not CycloneDX"

https://cyclonedx.org/bom is the predicate type cosign attest --type cyclonedx writes, and intoto: is how you name a predicate type. Kyverno also has a referrer: form that matches an OCI 1.1 artifact type, and it’s the wrong one here, because cosign attest writes a tag-based attestation at sha256-<digest>.att by default and no referrer. Pick referrer: and the policy reports the attestation missing while step three reports success, which is a miserable hour.

The .predicate in the third expression is there for the same reason. cosign wraps your BOM in an in-toto statement, so bomFormat sits one level down and no longer at the top.

extractPayload() errors unless verifyAttestationSignatures() ran first on the same attestation, which is the correct order anyway. Reading a payload you haven’t verified means trusting an attacker’s parts list.

If you run a self-hosted Sigstore after all, the keyless attestor takes a trustedRoot.expression that reads your own trust root JSON out of a ConfigMap, so the policy doesn’t have to trust the public Sigstore instance.

When the reports are quiet, flip one word:

  validationActions: [Deny]

Prove that it blocks

Don’t take the policy’s word for it. Push an unsigned image and watch the API server refuse it:

docker pull alpine:3.22
docker tag alpine:3.22 registry.example.com/platform/unsigned-test:1
docker push registry.example.com/platform/unsigned-test:1

kubectl run blocked --image=registry.example.com/platform/unsigned-test:1

Error from server: admission webhook "ivpol.validate.kyverno.svc-fail" denied the request:
ImageValidatingPolicy require-signed-images: Image is not signed by the platform pipeline

Then run the same test with an image your pipeline built and signed, and watch it start. Both halves matter. A policy that blocks everything is broken too, and that failure looks like success on a compliance report.

Put both commands in a scheduled pipeline. Policies drift, webhooks get disabled during an incident and nobody turns them back on, and a cached result can hide a problem for an hour. Which brings me to a caching detail worth knowing. Kyverno caches image verification results, with imageVerifyCacheTTLDuration defaulting to 60m. A signature you revoke in the registry is still accepted until that entry expires.

SLSA level 3, and what a GitLab runner actually reaches

Now the compliance claim from the top, taken seriously.

First, a citation correction that catches a lot of writing on this subject, including things I had bookmarked. SLSA v1.0 is retired. The current specification is v1.2, and it reorganised the URLs, so the level descriptions now live at build track basics and the detail at build requirements. The Build track text itself barely changed; v1.2’s addition is a Source track. If you’re citing spec/v1.0, you’re citing a retired document.

The Build track goes like this. L1 is provenance that exists. L2 is signed provenance from a hosted build platform, aimed at “tampering after the build”. L3 is a hardened build platform, aimed at “tampering during the build”, and it adds two requirements: Isolated and Provenance is Unforgeable.

GitLab’s own claim is precise and easy to miss. The documentation states that “GitLab offers a SLSA level 2 compliant provenance statement”, produced by the runner itself when you set RUNNER_GENERATE_ARTIFACTS_METADATA: "true". Level two. There’s a separate SLSA level 3 feature, and reading its header is the fastest way to know whether it applies to you. Tier Ultimate, Offering GitLab.com, Status Experiment, behind a feature flag called slsa_provenance_statement that’s off by default, and the project has to be public. A self-managed instance can’t use it at all.

So where does that leave a self-hosted runner? GitLab publishes no gap analysis, so what follows is my reading of the spec text against how a runner works, and nobody’s official position.

The last two are the ones that matter, and they’re structural, so no configuration you forgot fixes them. The spec says signing material “MUST NOT be accessible to the environment running the user-defined build steps”. Whether you hand the job a cosign key from a CI variable or a SIGSTORE_ID_TOKEN from id_tokens, the credential arrives as an environment variable in the same shell that runs your build script. And the provenance the runner generates is produced on the same machine that runs the job, so a tenant who can run arbitrary commands there is in a position to influence it. The spec’s word for the property you need is “strongly resistant to forgery by tenants”, and a control plane you operate as the same person who writes the pipeline isn’t that.

The Vault option narrows this. The private key genuinely never reaches the runner. The job can still ask Vault to sign something of its choosing, so it moves the problem from key theft to signing-oracle abuse. Smaller problem. Still a problem.

One line from the spec keeps the whole exercise honest: “Build L3 is limited to ensuring that a well-intentioned build runs securely.” It doesn’t stop you from writing a risky build. Nothing here does.

My conclusion, and I’d rather write it plainly than hedge. A self-hosted GitLab reaches Build L2 and most of L3’s isolation properties, and doesn’t reach L3. I’ve stopped putting a level number in documents. What I document instead is the verification command, because that’s a claim a reader can check.

Hardening the runner itself is a real subject and a large one, and it gets its own post.

What this does not buy you

Worth being blunt about, because the diagram makes it look like more than it is.

A signature proves an image came from your pipeline. It says nothing about whether the code in it is any good. Sign a backdoor and you get a correctly signed backdoor, and the policy will happily admit it.

Verification is a dependency too. cosign v3.1.3, released 6 August 2026, fixes GHSA-fx35-mq7g-6g98, rated 7.4. When a legacy JSON bundle’s certificate failed to parse as X.509, cosign fell back to reading it as a raw public key, which skipped chain validation and made --certificate-identity and --certificate-oidc-issuer do nothing. Image verification isn’t affected, so none of the commands in this post are, but verify-blob is. The shape of the bug is what I’d take away. The thing checking your signatures is software with bugs in it.

And the versions in this post move fast. On the day I wrote it: cosign v3.1.3, Kyverno v1.19.1, Trivy v0.74.0. Kyverno’s removal of ClusterPolicy in v1.20 is an announced plan, and no release has shipped it yet. Check your own versions instead of trusting mine.

The bit that mattered

Three of the four steps in this post produce evidence. One acts on it, and that pipeline had the three and not the one for about a year. Most supply chain work I see has that shape. Enormous effort goes into generating artefacts, and almost none into the thing that refuses.

What I got out of it has nothing to do with the level number or the questionnaire. I can answer “what’s running in this cluster and where did it come from” with a command now, where before it was a belief. Everything on this site comes back to that preference. I’d rather understand what I run than trust it.

Start with Audit and one glob, read the reports for a week, and flip it to Deny when they go quiet. Keep pushing the unsigned test image on a schedule after that.

• GitLab
• Containers
• DevOps