GitLab CI for Kubernetes: From Commit to Deployment

For a long time my CI lived on someone else’s servers. Push code, wait for a hosted runner, watch the minutes tick down against a monthly quota, hope the provider doesn’t change the rules next quarter. It worked, right up until I started asking where my source code actually sat while it was being built, and who else could see it.

So I moved GitLab in-house. I run it self-hosted now, on my own hardware, next to the clusters it deploys to. That’s sovereignty applied to CI/CD: my code, my builds, my artifacts, no vendor that can change pricing, deprecate a feature I depend on, or read my repositories without me knowing.

The reasons stack up once you start counting:

Data sovereignty: your code, your builds, your artifacts stay on your infrastructure
No usage limits: unlimited CI minutes, unlimited storage, unlimited users
Network locality: builds run close to your clusters, so artifact transfers are fast
Customisation: configure runners exactly how you need them
Air-gap capable: works in offline environments

None of this is free. You maintain GitLab now. You patch it, back it up, and you own the 2am page when a runner pod gets OOM-killed. For me that overhead buys something I can’t get any other way, so I pay it gladly.

That settles where the pipeline runs. The harder question is what it should actually do, and the part most people get wrong is the last step.

The complication: how does code get from a commit into a running pod, safely?

Building an image is easy. Anyone can write a docker build. The mess starts after the image exists. How do you stop a vulnerable image reaching production? Where do the cluster credentials live, and how many places get to hold them? When a deploy goes sideways, can you tell what changed and roll it back without spelunking through job logs?

The naive answer is to give the CI pipeline a kubeconfig and let it run kubectl apply at the end. It works on day one. By day ninety you’ve got cluster admin credentials sitting in a dozen project pipelines, no audit trail of what’s actually running, and a deploy process that only exists as imperative shell commands inside a YAML file.

I’d rather draw a hard line between what CI is good at and what it has no business touching.

The Pipeline Architecture

A Kubernetes deployment pipeline has distinct stages:

flowchart LR
    subgraph pipeline["GitLab CI Pipeline"]
        Build["Build<br/>Compile, Lint,<br/>Build image"] --> Test["Test<br/>Unit, Integr.,<br/>E2E"]
        Test --> Scan["Scan<br/>Trivy, SAST,<br/>Secrets"]
        Scan --> Publish["Publish<br/>Push to Registry,<br/>Sign"]
        Publish --> Deploy["Deploy<br/>Update GitOps<br/>Repo"]
    end

Each stage has a clear purpose, and the last one stops at the GitOps repo rather than the cluster. More on why that matters below. Let’s build it.

The Complete `.gitlab-ci.yml`

Here’s a production-ready pipeline:

stages:
  - build
  - test
  - scan
  - publish
  - deploy

variables:
  # Container settings
  DOCKER_HOST: tcp://docker:2376
  DOCKER_TLS_CERTDIR: "/certs"
  DOCKER_TLS_VERIFY: 1
  DOCKER_CERT_PATH: "$DOCKER_TLS_CERTDIR/client"

  # Image naming
  IMAGE_NAME: $CI_REGISTRY_IMAGE
  IMAGE_TAG: $CI_COMMIT_SHORT_SHA

  # Kubernetes
  KUBE_NAMESPACE: $CI_PROJECT_NAME

# Reusable configurations
.docker-base:
  image: docker:24
  services:
    - docker:24-dind
  before_script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY

# ============================================
# BUILD STAGE
# ============================================

build:
  extends: .docker-base
  stage: build
  script:
    - docker build
        --cache-from $IMAGE_NAME:latest
        --tag $IMAGE_NAME:$IMAGE_TAG
        --tag $IMAGE_NAME:latest
        --build-arg BUILDKIT_INLINE_CACHE=1
        .
    - docker push $IMAGE_NAME:$IMAGE_TAG
    - docker push $IMAGE_NAME:latest
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    - if: $CI_MERGE_REQUEST_ID

lint:
  stage: build
  image: golangci/golangci-lint:latest  # Adjust for your language
  script:
    - golangci-lint run ./...
  rules:
    - if: $CI_MERGE_REQUEST_ID

# ============================================
# TEST STAGE
# ============================================

unit-tests:
  stage: test
  image: golang:1.22  # Adjust for your language
  script:
    - go test -v -race -coverprofile=coverage.out ./...
    - go tool cover -func=coverage.out
  coverage: '/total:\s+\(statements\)\s+(\d+\.\d+)%/'
  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage.xml
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    - if: $CI_MERGE_REQUEST_ID

integration-tests:
  stage: test
  extends: .docker-base
  script:
    - docker compose -f docker-compose.test.yml up -d
    - docker compose -f docker-compose.test.yml run tests
    - docker compose -f docker-compose.test.yml down
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

# ============================================
# SCAN STAGE
# ============================================

container-scan:
  stage: scan
  image:
    name: aquasec/trivy:latest
    entrypoint: [""]
  script:
    - trivy image
        --exit-code 1
        --severity HIGH,CRITICAL
        --ignore-unfixed
        $IMAGE_NAME:$IMAGE_TAG
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    - if: $CI_MERGE_REQUEST_ID
  allow_failure: false

sast:
  stage: scan
  image: returntocorp/semgrep
  script:
    - semgrep --config auto --error .
  rules:
    - if: $CI_MERGE_REQUEST_ID

secret-detection:
  stage: scan
  image: trufflesecurity/trufflehog:latest
  script:
    - trufflehog git file://. --only-verified --fail
  rules:
    - if: $CI_MERGE_REQUEST_ID

# ============================================
# PUBLISH STAGE
# ============================================

publish:
  extends: .docker-base
  stage: publish
  script:
    # Tag with semantic version if tagged
    - |
      if [ -n "$CI_COMMIT_TAG" ]; then
        docker pull $IMAGE_NAME:$IMAGE_TAG
        docker tag $IMAGE_NAME:$IMAGE_TAG $IMAGE_NAME:$CI_COMMIT_TAG
        docker push $IMAGE_NAME:$CI_COMMIT_TAG
      fi
  rules:
    - if: $CI_COMMIT_TAG

# ============================================
# DEPLOY STAGE
# ============================================

deploy-staging:
  stage: deploy
  image: alpine:latest
  before_script:
    - apk add --no-cache git openssh-client
    - eval $(ssh-agent -s)
    - echo "$GITOPS_SSH_KEY" | tr -d '\r' | ssh-add -
    - mkdir -p ~/.ssh && chmod 700 ~/.ssh
    - ssh-keyscan gitlab.com >> ~/.ssh/known_hosts
  script:
    - git clone git@gitlab.com:$GITOPS_REPO.git gitops
    - cd gitops
    - |
      # Update image tag in Kustomize
      cd apps/$CI_PROJECT_NAME/overlays/staging
      kustomize edit set image $IMAGE_NAME:$IMAGE_TAG
    - git config user.email "ci@gitlab.local"
    - git config user.name "GitLab CI"
    - git add -A
    - git commit -m "Deploy $CI_PROJECT_NAME:$IMAGE_TAG to staging" || exit 0
    - git push
  environment:
    name: staging
    url: https://staging.example.com
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH

deploy-production:
  stage: deploy
  image: alpine:latest
  before_script:
    - apk add --no-cache git openssh-client
    - eval $(ssh-agent -s)
    - echo "$GITOPS_SSH_KEY" | tr -d '\r' | ssh-add -
    - mkdir -p ~/.ssh && chmod 700 ~/.ssh
    - ssh-keyscan gitlab.com >> ~/.ssh/known_hosts
  script:
    - git clone git@gitlab.com:$GITOPS_REPO.git gitops
    - cd gitops
    - |
      cd apps/$CI_PROJECT_NAME/overlays/production
      kustomize edit set image $IMAGE_NAME:$IMAGE_TAG
    - git config user.email "ci@gitlab.local"
    - git config user.name "GitLab CI"
    - git add -A
    - git commit -m "Deploy $CI_PROJECT_NAME:$IMAGE_TAG to production" || exit 0
    - git push
  environment:
    name: production
    url: https://example.com
  rules:
    - if: $CI_COMMIT_TAG
  when: manual  # Require manual approval for production

That’s a lot of YAML at once. Let me walk through the stages that earn their keep.

Breaking Down the Pipeline

Build Stage: Creating the Container

build:
  extends: .docker-base
  stage: build
  script:
    - docker build
        --cache-from $IMAGE_NAME:latest
        --tag $IMAGE_NAME:$IMAGE_TAG
        --tag $IMAGE_NAME:latest
        .

Key points:

Cache from previous builds: --cache-from speeds up builds dramatically
Commit SHA as tag: immutable, traceable image versions. The tag tells you exactly which commit produced the image, which you’ll want the first time a deploy misbehaves
Also tag latest: for the cache source in the next build

Test Stage: Verify Before Deploy

Tests run in parallel where possible:

unit-tests:
  # ...
  coverage: '/total:\s+\(statements\)\s+(\d+\.\d+)%/'

The coverage regex extracts the coverage percentage so GitLab can show it on the merge request. Small thing, but it puts the number in front of you at review time instead of buried in a log nobody opens.

Scan Stage: Security Gates

This is where the pipeline starts saying no on your behalf. Three scans, each catching a different class of mistake:

Container scanning (Trivy): finds vulnerabilities in container images
SAST: static analysis of source code
Secret detection: catches credentials someone accidentally committed

container-scan:
  script:
    - trivy image
        --exit-code 1          # Fail pipeline on findings
        --severity HIGH,CRITICAL  # Only block on serious issues
        --ignore-unfixed       # Skip vulnerabilities without fixes

Deploy Stage: GitOps Integration

Here’s the answer to the question I posed up top. The pipeline never touches the cluster. It updates a GitOps repository, and ArgoCD does the actual deployment by reconciling the cluster against what’s now in Git.

deploy-staging:
  script:
    - git clone git@gitlab.com:$GITOPS_REPO.git gitops
    - cd gitops/apps/$CI_PROJECT_NAME/overlays/staging
    - kustomize edit set image $IMAGE_NAME:$IMAGE_TAG
    - git commit -m "Deploy $CI_PROJECT_NAME:$IMAGE_TAG to staging"
    - git push

This pattern:

Clones the GitOps repository
Updates the image tag using Kustomize
Commits and pushes
ArgoCD detects the change and deploys

The win is in what’s missing. The application repo never holds cluster credentials. It only needs write access to one GitOps repo. Compromise a project pipeline and the blast radius is a Git commit someone will notice on review, not a kubeconfig with the keys to the cluster. Every deploy is now a commit with a diff, an author, and a timestamp, so the question “what’s running right now and who put it there” has a one-line answer. And when production catches fire, rollback is git revert, which is muscle memory rather than a fresh round of imperative kubectl under pressure.

That separation is the whole point. CI builds and proves the artifact; Git records intent; ArgoCD makes reality match. Each part does the thing it’s actually good at.

GitLab Runners for Kubernetes

None of this runs without GitLab Runners to execute the jobs. I run them in Kubernetes too, so the same cluster that hosts my workloads also builds them:

# values.yaml for gitlab-runner Helm chart
gitlabUrl: https://gitlab.example.com
runnerRegistrationToken: "your-token"

runners:
  config: |
    [[runners]]
      [runners.kubernetes]
        namespace = "gitlab-runners"
        image = "alpine:latest"
        privileged = true  # Required for Docker-in-Docker

        [[runners.kubernetes.volumes.empty_dir]]
          name = "docker-certs"
          mount_path = "/certs/client"
          medium = "Memory"

rbac:
  create: true

resources:
  limits:
    memory: 256Mi
    cpu: 250m

Install with Helm:

helm repo add gitlab https://charts.gitlab.io
helm install gitlab-runner gitlab/gitlab-runner -f values.yaml -n gitlab-runners

Environment Variables and Secrets

Store sensitive values in GitLab CI/CD variables:

Project Settings → CI/CD → Variables

Variable	Protected	Masked	Description
`GITOPS_SSH_KEY`	✓	✓	SSH key for GitOps repo
`KUBECONFIG`	✓	✓	Kubernetes config (if direct deploy)
`SONAR_TOKEN`	✓	✓	SonarQube token

Protected: only available on protected branches Masked: hidden in job logs

Mark the GitOps SSH key both protected and masked. A feature branch should never be able to push a deploy, and the key should never show up in a log a teammate can scroll back through.

Merge Request Pipelines

You don’t need the full pipeline on every push. Merge requests run a subset so feedback comes back fast:

lint:
  rules:
    - if: $CI_MERGE_REQUEST_ID  # Only on MRs

unit-tests:
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
    - if: $CI_MERGE_REQUEST_ID  # Also on MRs

Fast feedback on the MR, the heavier jobs reserved for the default branch. A pipeline you wait ten minutes for on every push is a pipeline people start skipping.

Caching for Speed

The other thing that keeps a pipeline from getting ignored is speed, and caching does most of the work:

variables:
  GOPATH: $CI_PROJECT_DIR/.go

cache:
  key: ${CI_COMMIT_REF_SLUG}
  paths:
    - .go/pkg/mod/
    - node_modules/
    - .npm/

For Docker layer caching, use BuildKit:

build:
  variables:
    DOCKER_BUILDKIT: 1
  script:
    - docker build --build-arg BUILDKIT_INLINE_CACHE=1 ...

Monitoring Pipeline Health

A pipeline is infrastructure, and infrastructure you don’t watch is infrastructure you don’t understand. Scrape GitLab’s own metrics:

# In your monitoring stack
- job_name: 'gitlab'
  static_configs:
    - targets: ['gitlab.example.com']
  metrics_path: '/-/metrics'

Key metrics to watch:

Pipeline duration (creeping up means caching is failing somewhere)
Success/failure rate
Queue time (climbing queue time means runners are saturated)
Cache hit rate

The Complete Flow

flowchart TD
    Dev["Developer<br/>pushes code"] --> GitLab["GitLab<br/>triggers CI"]
    GitLab --> Pipeline["Pipeline<br/>Build → Test → Scan → Publish → Update GitOps"]
    Pipeline --> GitOpsRepo["GitOps Repo<br/>(updated)"]
    GitOpsRepo --> ArgoCD["ArgoCD<br/>detects & deploys"]
    ArgoCD --> K8s["Kubernetes<br/>(running)"]

Clean separation of concerns:

GitLab CI: build, test, scan, publish
GitOps repo: desired state
ArgoCD: reconciliation
Kubernetes: runtime

Each box knows nothing about the others’ internals. CI doesn’t know how the cluster applies manifests. The cluster doesn’t know how the image got built. They meet at a Git commit, which is the most boring and most inspectable interface I could ask for.

My Recommendations

Self-host GitLab if you care about sovereignty. The operational overhead is real, and it’s worth it.
Never deploy directly from CI to Kubernetes. Update GitOps repos and let a reconciler do the applying.
Fail fast on security: container scans with --exit-code 1 block vulnerable images before they ship.
Use protected variables: secrets should only be reachable from protected branches.
Manual production deploys: keep a human in the loop for production.

The pipeline I trust most is the one I’d be happy to explain to a stranger line by line. Build, prove, record, reconcile. Nothing magic, nothing hidden, and a Git history that tells you exactly what happened when something breaks.

The complication: how does code get from a commit into a running pod, safely?#

The Pipeline Architecture#

The Complete .gitlab-ci.yml#

Breaking Down the Pipeline#

Build Stage: Creating the Container#

Test Stage: Verify Before Deploy#

Scan Stage: Security Gates#

Deploy Stage: GitOps Integration#

GitLab Runners for Kubernetes#

Environment Variables and Secrets#

Merge Request Pipelines#

Caching for Speed#

Monitoring Pipeline Health#

The Complete Flow#

My Recommendations#