I used to keep a wiki page titled “Cluster conventions”. Resource limits on everything. No :latest tags. No deploys in the default namespace. It was a good page. Nobody read it. Six months in, half the cluster broke those rules and I only found out when something fell over.

A rule that lives in a doc is a suggestion. A rule the API server refuses to accept is governance. That gap is the whole reason this post exists.

Kubernetes hands every team enormous freedom. Deploy anything, configure it however you like. Without guardrails that freedom turns into a slow accumulation of mess you discover at the worst possible moment. I want the cluster to enforce my standards so I don’t have to remember them, and so I can actually understand what’s running instead of hoping people followed the wiki.

Kyverno is a policy engine for Kubernetes. It validates, mutates, and generates resources based on policies you write as plain Kubernetes-native YAML. The rest of this post builds up from the smallest possible policy to the full set I run on every cluster, so you can stop wherever you have enough.

Why Kyverno and not OPA

You have options here: Open Policy Agent (OPA) with Gatekeeper, Kyverno, and Kubewarden. I landed on Kyverno for a few honest reasons:

  1. Native YAML. No Rego, no second language to keep in my head.
  2. Validation, mutation, and generation in one tool. I don’t want three different things for three different jobs.
  3. Declarative. Policies are just Kubernetes resources, so they live in Git next to everything else.
  4. GitOps-friendly. ArgoCD syncs them like any other manifest.

OPA/Gatekeeper is genuinely powerful, and Rego buys you expressiveness Kyverno can’t match for the gnarliest rules. The trade-off is real: you learn a new language before you write your first useful policy. For my homelab and most clusters I touch, that cost isn’t worth it. If you already know YAML, Kyverno lets you write something useful in the next five minutes.

The three things a policy can do

Before any YAML, hold onto these three verbs. Everything Kyverno does is one of them.

Validate. Block resources that don’t meet your criteria.

"You can't deploy without resource limits"

Mutate. Modify resources on the way in.

"All pods get this label automatically"

Generate. Create new resources when others appear.

"Every namespace gets a default NetworkPolicy"

That’s the whole mental model. Validate is where almost everyone starts, so that’s where we’ll start too.

Installing Kyverno

helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update

helm install kyverno kyverno/kyverno \
  --namespace kyverno \
  --create-namespace

For GitOps with ArgoCD:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: kyverno
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://kyverno.github.io/kyverno/
    chart: kyverno
    targetRevision: 3.1.4
    helm:
      values: |
        admissionController:
          replicas: 3
        backgroundController:
          replicas: 2
        cleanupController:
          replicas: 2
        reportsController:
          replicas: 2
  destination:
    server: https://kubernetes.default.svc
    namespace: kyverno
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

The simplest possible policy: require a label

Here’s the smallest useful thing you can write. It requires every pod to carry a team label, and nothing more:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-labels
spec:
  validationFailureAction: Enforce
  rules:
    - name: require-team-label
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        message: "The label 'team' is required."
        pattern:
          metadata:
            labels:
              team: "?*"

Try to create a pod without the label and the API server slams the door:

kubectl run test --image=nginx
# Error: The label 'team' is required.

That’s it. One CRD, and the convention is now load-bearing instead of decorative. Notice nobody had to remember anything.

Enforce versus Audit: don’t break prod on day one

Before you go further, the one knob that saves you from yourself. validationFailureAction has two settings.

Enforce blocks non-compliant resources outright. This is what you want in production, eventually.

validationFailureAction: Enforce

Audit lets the resource through but records the violation in a report.

validationFailureAction: Audit

I learned this one the hard way. Roll out a strict policy in Enforce mode against a cluster that’s been ignoring your wiki for months, and you’ll wedge half your deploys at once. Start every new policy in Audit, look at what would have failed, fix the real offenders, then flip to Enforce. Graceful degradation applies to your own rollout too.

Layer 1: the validation policies that earn their keep

A label requirement is a warm-up. These are the rules that actually keep a cluster sane. Each one is still a single ClusterPolicy, just pointed at a more useful thing.

Require resource limits

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-resource-limits
spec:
  validationFailureAction: Enforce
  rules:
    - name: require-limits
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        message: "CPU and memory limits are required."
        pattern:
          spec:
            containers:
              - resources:
                  limits:
                    memory: "?*"
                    cpu: "?*"

Disallow privileged containers

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-privileged
spec:
  validationFailureAction: Enforce
  rules:
    - name: deny-privileged
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        message: "Privileged containers are not allowed."
        pattern:
          spec:
            containers:
              - securityContext:
                  privileged: false

Disallow the latest tag

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-latest-tag
spec:
  validationFailureAction: Enforce
  rules:
    - name: require-image-tag
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        message: "Using ':latest' tag is not allowed. Specify a version tag."
        pattern:
          spec:
            containers:
              - image: "!*:latest"

Restrict image registries

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-registries
spec:
  validationFailureAction: Enforce
  rules:
    - name: allowed-registries
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        message: "Images must come from approved registries."
        pattern:
          spec:
            containers:
              - image: "registry.example.com/* | gcr.io/my-project/*"

Layer 2: mutation, or fixing instead of nagging

Validation is the bouncer who turns people away. Mutation is the host who quietly fixes their outfit at the door. Instead of rejecting a resource for missing a label or a security setting, Kyverno can just add it. Less friction for whoever’s deploying, same end state for the cluster.

Add default labels

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-default-labels
spec:
  rules:
    - name: add-managed-by
      match:
        any:
          - resources:
              kinds:
                - Pod
                - Deployment
                - Service
      mutate:
        patchStrategicMerge:
          metadata:
            labels:
              managed-by: kyverno
              environment: "{{request.namespace}}"

Add a default security context

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-security-context
spec:
  rules:
    - name: add-run-as-nonroot
      match:
        any:
          - resources:
              kinds:
                - Pod
      mutate:
        patchStrategicMerge:
          spec:
            securityContext:
              runAsNonRoot: true
              seccompProfile:
                type: RuntimeDefault

Add image pull secrets

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-imagepullsecret
spec:
  rules:
    - name: add-registry-secret
      match:
        any:
          - resources:
              kinds:
                - Pod
      mutate:
        patchStrategicMerge:
          spec:
            imagePullSecrets:
              - name: registry-credentials

Layer 3: generation, the one that surprised me

This is the verb I underused for the longest time, and it’s the one that changed how I think about cluster defaults. Generation creates brand-new resources in response to other resources appearing. Make a namespace, and Kyverno can drop a NetworkPolicy and a ResourceQuota into it before anyone deploys a single pod.

A default-deny NetworkPolicy in every namespace

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: generate-default-networkpolicy
spec:
  rules:
    - name: generate-deny-all
      match:
        any:
          - resources:
              kinds:
                - Namespace
      exclude:
        any:
          - resources:
              namespaces:
                - kube-system
                - kyverno
      generate:
        apiVersion: networking.k8s.io/v1
        kind: NetworkPolicy
        name: default-deny
        namespace: "{{request.object.metadata.name}}"
        data:
          spec:
            podSelector: {}
            policyTypes:
              - Ingress
              - Egress

Every new namespace starts locked down. Teams have to explicitly allow the traffic they need. That’s zero trust as a default rather than a checklist item someone forgets, and it shrinks the blast radius when something does get popped.

A default ResourceQuota per namespace

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: generate-resourcequota
spec:
  rules:
    - name: generate-quota
      match:
        any:
          - resources:
              kinds:
                - Namespace
      exclude:
        any:
          - resources:
              namespaces:
                - kube-system
                - kyverno
      generate:
        apiVersion: v1
        kind: ResourceQuota
        name: default-quota
        namespace: "{{request.object.metadata.name}}"
        data:
          spec:
            hard:
              requests.cpu: "4"
              requests.memory: 8Gi
              limits.cpu: "8"
              limits.memory: 16Gi
              persistentvolumeclaims: "10"

Variables: making policies context-aware

Hardcoded values get you far, but the good stuff comes from reading the request. Kyverno supports JMESPath expressions so a policy can react to who’s deploying, into which namespace, at what time:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-namespace-label
spec:
  rules:
    - name: add-ns-to-pods
      match:
        any:
          - resources:
              kinds:
                - Pod
      mutate:
        patchStrategicMerge:
          metadata:
            labels:
              namespace: "{{request.namespace}}"
              created-at: "{{time_now_utc()}}"

Useful variables:

  • {{request.namespace}} is the namespace of the resource
  • {{request.object.metadata.name}} is the name of the resource
  • {{request.userInfo.username}} is the user who made the request
  • {{time_now_utc()}} is the current timestamp

Excluding the resources you can’t touch

One gotcha that bites everyone: apply a strict policy cluster-wide and you’ll break kube-system or Kyverno itself. System components don’t follow your rules, and they shouldn’t have to. Exclude them explicitly:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-labels
spec:
  rules:
    - name: require-team-label
      match:
        any:
          - resources:
              kinds:
                - Pod
      exclude:
        any:
          - resources:
              namespaces:
                - kube-system
                - kube-public
                - kyverno
          - subjects:
              - kind: ServiceAccount
                name: system:*

Policy reports: seeing the state of the cluster

Enforcement is half the value. The other half is visibility. Kyverno writes policy reports that show exactly what’s passing and what isn’t, which means I can answer “is this cluster compliant right now?” without trusting anyone’s memory:

kubectl get policyreport -A
kubectl get clusterpolicyreport

Example report:

apiVersion: wgpolicyk8s.io/v1alpha2
kind: PolicyReport
metadata:
  name: polr-ns-default
  namespace: default
results:
  - message: "validation rule 'require-team-label' passed."
    policy: require-labels
    result: pass
    source: kyverno
  - message: "validation rule 'require-limits' failed."
    policy: require-resource-limits
    result: fail
    source: kyverno

Integrate with Prometheus for alerting on policy violations:

# ServiceMonitor for Kyverno metrics
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: kyverno
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: kyverno
  endpoints:
    - port: metrics

Don’t write everything yourself: curated policy sets

You don’t have to reinvent the security baseline. Kyverno maintains curated policy collections, including ready-made implementations of the Kubernetes Pod Security Standards:

# Install Pod Security Standards (Baseline)
kubectl apply -f https://raw.githubusercontent.com/kyverno/policies/main/pod-security/baseline/

# Install Pod Security Standards (Restricted)
kubectl apply -f https://raw.githubusercontent.com/kyverno/policies/main/pod-security/restricted/

These give you the Baseline and Restricted Pod Security Standards as Kyverno policies, no Rego, no hand-rolling.

The full picture: what I actually run

Remember that single label policy at the top? Here’s where it ends up. This is the core set I drop on every cluster I build, all in Enforce mode after an Audit shakedown: readiness probes required, no deploys in default, resource requests mandatory.

# require-probes.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-probes
spec:
  validationFailureAction: Enforce
  rules:
    - name: require-readiness-probe
      match:
        any:
          - resources:
              kinds:
                - Deployment
      exclude:
        any:
          - resources:
              namespaces:
                - kube-system
                - kyverno
      validate:
        message: "Readiness probe is required for all deployments."
        pattern:
          spec:
            template:
              spec:
                containers:
                  - readinessProbe:
                      "?*": "?*"
---
# disallow-default-namespace.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-default-namespace
spec:
  validationFailureAction: Enforce
  rules:
    - name: deny-default
      match:
        any:
          - resources:
              kinds:
                - Pod
                - Deployment
                - Service
              namespaces:
                - default
      validate:
        message: "Using the 'default' namespace is not allowed."
        deny: {}
---
# require-requests.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-requests
spec:
  validationFailureAction: Enforce
  rules:
    - name: require-cpu-memory-requests
      match:
        any:
          - resources:
              kinds:
                - Pod
      exclude:
        any:
          - resources:
              namespaces:
                - kube-system
      validate:
        message: "CPU and memory requests are required."
        pattern:
          spec:
            containers:
              - resources:
                  requests:
                    memory: "?*"
                    cpu: "?*"

Test policies before they touch the cluster

Policies are code, so I treat them like code. The Kyverno CLI lets me check a policy against a sample resource on my laptop before anything reaches a real API server:

# Install CLI
brew install kyverno

# Test policy against a resource
kyverno apply policy.yaml --resource pod.yaml

# Test all policies in a directory
kyverno apply ./policies/ --resource ./manifests/

Create a test suite:

# test/values.yaml
policies:
  - policy.yaml
resources:
  - resources/valid-pod.yaml
  - resources/invalid-pod.yaml
results:
  - policy: require-labels
    resource: valid-pod.yaml
    result: pass
  - policy: require-labels
    resource: invalid-pod.yaml
    result: fail

Run tests:

kyverno test ./test/

Background scanning: catching what’s already there

Admission control only sees new requests. But your cluster is full of pods that were created before you wrote any of this. Turn on background scanning and Kyverno audits the resources already running:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: audit-existing
spec:
  validationFailureAction: Audit
  background: true  # Enable background scanning
  rules:
    - name: check-labels
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        message: "Label 'team' is missing."
        pattern:
          metadata:
            labels:
              team: "?*"

Now you get reports for every pod in the cluster, not just the ones created after you showed up. It’s the difference between guarding the door and actually knowing what’s in the building.

Exceptions, and why they should sting a little

Sometimes a workload genuinely needs to break a rule. node-exporter wants a privileged container, and that’s fine. Kyverno has a first-class object for it so you don’t water down the policy itself:

apiVersion: kyverno.io/v1
kind: PolicyException
metadata:
  name: allow-privileged-monitoring
  namespace: monitoring
spec:
  exceptions:
    - policyName: disallow-privileged
      ruleNames:
        - deny-privileged
  match:
    any:
      - resources:
          kinds:
            - Pod
          namespaces:
            - monitoring
          names:
            - node-exporter*

Write down why each exception exists, ideally in the manifest itself. A PolicyException is a named, version-controlled hole in your defenses, which is exactly how I want my holes: visible. If you find yourself adding a third one this month, the policy is probably wrong, not the workloads.

Catch violations at merge, not at deploy

The best place to reject a bad manifest is in a merge request, long before it reaches a cluster. Run the same CLI in your pipeline:

# .gitlab-ci.yml
validate-policies:
  stage: test
  image: ghcr.io/kyverno/kyverno-cli:latest
  script:
    - kyverno apply ./policies/ --resource ./k8s/
  rules:
    - if: $CI_MERGE_REQUEST_ID

Now a developer gets the feedback in their MR pipeline, with the rest of their tests, while the change is still cheap to fix.

What this buys me

Go back to that wiki page I started with. Here’s what changed when the rules moved out of the doc and into Kyverno.

Without a policy engine, standards live in wikis nobody reads, reviews catch problems inconsistently depending on who’s looking, and non-compliance piles up quietly until something breaks. With Kyverno, the standards are code in Git, enforcement is automatic and immediate, and every violation shows up in a report I can actually query.

This is low-friction governance. The point isn’t that it’s stricter, it’s that the strictness costs nobody any attention. I don’t ask developers to memorise a convention page, and I don’t have to police it myself. The cluster remembers the rules so my brain doesn’t have to, which is the whole reason I build systems this way.

The best policies are the ones you stop thinking about. Mine have been running quietly for months, and the only time I notice them is when one stops a mistake I would otherwise have shipped.