For a long time my certificates renewed the way most people’s do: a calendar reminder, a manual certbot run, and a quiet hope that I’d remember before the thing actually expired. It worked. It worked right up until the morning a service threw cert errors at me and I had no idea why, because the renewal cron had been silently failing for weeks.

That’s the part nobody tells you about manual TLS. The failure doesn’t announce itself. The cert just expires, usually at the worst possible moment, and you find out because a browser is yelling at someone. Renewal knowledge ends up living in one person’s head. Teams skip HTTPS on internal services because wiring it up by hand is annoying enough to put off.

I wanted certificates to be something the cluster owned, not something I had to remember. cert-manager gives you exactly that. You declare which certificates you need, and it handles issuance, renewal, and the Kubernetes Secrets they live in. You stop thinking about expiry dates entirely.

It’s one of the first things I install in any new cluster, right after the CNI.

How cert-manager Works

flowchart TD
    subgraph cluster["Kubernetes Cluster"]
        CM["cert-manager"]
        CERT["Certificate<br/>Resource"]
        SECRET["TLS Secret"]
        INGRESS["Ingress"]
    end

    subgraph external["External"]
        LE["Let's Encrypt<br/>ACME Server"]
        DNS["DNS Provider"]
    end

    CERT -->|"watches"| CM
    CM -->|"creates"| SECRET
    CM <-->|"ACME protocol"| LE
    CM <-->|"DNS challenge"| DNS
    SECRET -->|"mounts"| INGRESS

The loop is simple once you see it:

  1. You create a Certificate resource
  2. cert-manager requests a certificate from the issuer (Let’s Encrypt, Vault, etc.)
  3. cert-manager completes the challenge (HTTP-01 or DNS-01) to prove you control the domain
  4. cert-manager stores the certificate in a Kubernetes Secret
  5. Your Ingress or Gateway mounts that Secret for TLS

Renewal happens on its own, 30 days before expiry, with no calendar reminder involved. That last sentence is the whole point of this post.

Installation

helm repo add jetstack https://charts.jetstack.io
helm repo update

helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --set installCRDs=true

The --set installCRDs=true matters. Skip it and cert-manager’s custom resources never get registered, which produces some confusing errors later. If you run everything through GitOps with ArgoCD like I do, the equivalent Application looks like this:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: cert-manager
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://charts.jetstack.io
    chart: cert-manager
    targetRevision: v1.14.0
    helm:
      values: |
        installCRDs: true
        prometheus:
          servicemonitor:
            enabled: true
  destination:
    server: https://kubernetes.default.svc
    namespace: cert-manager
  syncPolicy:
    syncOptions:
      - CreateNamespace=true

Issuers

An Issuer tells cert-manager where to get certificates from. There are two scopes, and the difference trips people up:

  • Issuer: works in one namespace
  • ClusterIssuer: works across all namespaces

I default to ClusterIssuer unless I have a real reason to scope a certificate authority to a single namespace. Less duplication, fewer surprises.

Let’s Encrypt (ACME)

This is the setup most people reach for first, free certificates from Let’s Encrypt:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: admin@example.com
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
      - http01:
          ingress:
            class: nginx

Important: start with the staging issuer. Let’s Encrypt has aggressive rate limits, and if you’re iterating on config you will hit them. I learned this the slow way, locked out for a week after fat-fingering a solver config a few too many times. Staging gives you untrusted certs that exercise the exact same flow:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: admin@example.com
    privateKeySecretRef:
      name: letsencrypt-staging-account-key
    solvers:
      - http01:
          ingress:
            class: nginx

HTTP-01 vs DNS-01 Challenges

The challenge is how Let’s Encrypt confirms you actually control the domain before it hands you a cert. There are two ways to prove it, and which one you pick shapes everything downstream.

HTTP-01: Let’s Encrypt verifies you control the domain by placing a file at /.well-known/acme-challenge/:

solvers:
  - http01:
      ingress:
        class: nginx

It works for anything publicly reachable on port 80. Quick to set up, nothing extra to configure. The catch is in the name: the service has to be reachable from the public internet for Let’s Encrypt to fetch that file.

DNS-01: Let’s Encrypt verifies via a DNS TXT record instead:

solvers:
  - dns01:
      cloudflare:
        email: admin@example.com
        apiTokenSecretRef:
          name: cloudflare-api-token
          key: api-token

DNS-01 is the one I end up using most, because it covers the cases HTTP-01 can’t:

  • Wildcard certificates (*.example.com), which HTTP-01 flat out cannot issue
  • Internal services that never touch the public internet
  • Split-horizon DNS setups

Cloudflare DNS-01 Setup

# API Token Secret
apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-token
  namespace: cert-manager
type: Opaque
stringData:
  api-token: "your-cloudflare-api-token"
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-cloudflare
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: admin@example.com
    privateKeySecretRef:
      name: letsencrypt-cloudflare-account
    solvers:
      - dns01:
          cloudflare:
            apiTokenSecretRef:
              name: cloudflare-api-token
              key: api-token

The Cloudflare API token needs Zone:DNS:Edit permission, scoped to the zones you actually use. Don’t reach for the global API key here. Scoped tokens are the whole point, and a leaked global key is a bad afternoon.

Private CA

Public certificates are great until you have services that should never be exposed to the public internet at all. For those, you run your own CA, which keeps the whole trust chain inside infrastructure you control:

# Create CA key pair
apiVersion: v1
kind: Secret
metadata:
  name: ca-key-pair
  namespace: cert-manager
type: kubernetes.io/tls
data:
  tls.crt: <base64-encoded-ca-cert>
  tls.key: <base64-encoded-ca-key>
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: internal-ca
spec:
  ca:
    secretName: ca-key-pair

Or let cert-manager create a self-signed CA:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned-issuer
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: my-ca
  namespace: cert-manager
spec:
  isCA: true
  commonName: my-ca
  secretName: my-ca-secret
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: selfsigned-issuer
    kind: ClusterIssuer
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: my-ca-issuer
spec:
  ca:
    secretName: my-ca-secret

HashiCorp Vault

If you already run Vault, its PKI engine makes a solid issuer. You get short-lived certs, central audit logging, and one place to revoke from:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: vault-issuer
spec:
  vault:
    server: https://vault.vault:8200
    path: pki/sign/my-role
    auth:
      kubernetes:
        role: cert-manager
        mountPath: /v1/auth/kubernetes
        secretRef:
          name: cert-manager-vault-token
          key: token

Requesting Certificates

Once an issuer exists, there are two ways to actually get a certificate. You can ask for one explicitly, or you can let cert-manager infer it from an Ingress. I use both, depending on how much control I want over the details.

Certificate Resource

The explicit route. You spell out exactly what you want:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: api-example-com
  namespace: production
spec:
  secretName: api-example-com-tls
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
    - api.example.com
    - api-v2.example.com
  duration: 2160h  # 90 days
  renewBefore: 720h  # Renew 30 days before expiry

Ingress Annotation (Automatic)

The lazy route, in the good sense. Add one annotation to an Ingress and cert-manager does the rest:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: api
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  tls:
    - hosts:
        - api.example.com
      secretName: api-example-com-tls
  rules:
    - host: api.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: api
                port:
                  number: 80

cert-manager spots the annotation, generates the Certificate resource for you, and populates the Secret. For most app deployments this is all you need, and it keeps the TLS config next to the routing config where it belongs.

Gateway API

Moving to the Gateway API instead of Ingress? Same idea, different resource:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: main
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  gatewayClassName: cilium
  listeners:
    - name: https
      port: 443
      protocol: HTTPS
      hostname: "*.example.com"
      tls:
        mode: Terminate
        certificateRefs:
          - name: wildcard-example-com-tls

Wildcard Certificates

A wildcard cert covers every subdomain under one name, which means one certificate instead of a fresh one per service. That alone keeps you well clear of Let’s Encrypt’s rate limits. The price is that you have to use DNS-01, since HTTP-01 can’t validate a wildcard:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: wildcard-example-com
  namespace: production
spec:
  secretName: wildcard-example-com-tls
  issuerRef:
    name: letsencrypt-cloudflare
    kind: ClusterIssuer
  dnsNames:
    - "*.example.com"
    - "example.com"

Note: list both *.example.com and example.com. The wildcard does not cover the apex domain, and forgetting that line is a classic way to serve a broken cert on your root domain.

Cross-Namespace Secrets

Here’s a wrinkle that catches people. A certificate Secret lives in exactly one namespace, but you often need the same cert in several. A Secret in cert-manager can’t be mounted by a pod in production. So how do you share one cert across namespaces without minting a separate one each time? Two approaches.

Reflector (External Controller)

kubernetes-reflector copies a Secret into the namespaces you list:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: shared-cert
  namespace: cert-manager
  annotations:
    reflector.v1.k8s.emberstack.com/reflection-allowed: "true"
    reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces: "production,staging"
spec:
  secretName: shared-cert-tls
  # ...

trust-manager

For distributing CA bundles specifically, cert-manager ships its own answer. trust-manager pushes a trust bundle into ConfigMaps across namespaces, which is what you want when the thing you’re sharing is the CA cert rather than a leaf certificate:

helm install trust-manager jetstack/trust-manager \
  --namespace cert-manager

apiVersion: trust.cert-manager.io/v1alpha1
kind: Bundle
metadata:
  name: my-ca-bundle
spec:
  sources:
    - secret:
        name: my-ca-secret
        key: ca.crt
  target:
    configMap:
      key: ca-bundle.crt
    namespaceSelector:
      matchLabels:
        trust-bundle: enabled

Monitoring and Troubleshooting

Automation is great until it fails silently, which is exactly the trap I started this post complaining about. The fix is to make cert-manager observable, so a stuck renewal becomes an alert instead of an outage. Here’s how I poke at it when something looks off.

Check Certificate Status

kubectl get certificates -A
kubectl describe certificate api-example-com -n production

Check Certificate Requests

kubectl get certificaterequests -A
kubectl describe certificaterequest api-example-com-xyz -n production

Check Orders and Challenges

kubectl get orders -A
kubectl get challenges -A

# Debug a stuck challenge
kubectl describe challenge api-example-com-xyz-123 -n production

When a cert is misbehaving, you walk down this chain: Certificate, then CertificateRequest, then Order, then Challenge. The error is almost always sitting in one of those describe outputs, and the failure usually surfaces at the Challenge level.

Prometheus Metrics

Manual checks are for debugging. For the safety net, you want metrics. cert-manager exports them, and these two alerts are the ones I actually rely on:

# Alert on expiring certificates
- alert: CertificateExpiringSoon
  expr: certmanager_certificate_expiration_timestamp_seconds - time() < 604800
  for: 1h
  labels:
    severity: warning
  annotations:
    summary: "Certificate {{ $labels.name }} expires in less than 7 days"

# Alert on failed renewals
- alert: CertificateRenewalFailed
  expr: certmanager_certificate_ready_status{condition="False"} == 1
  for: 1h
  labels:
    severity: critical
  annotations:
    summary: "Certificate {{ $labels.name }} is not ready"

That first alert is the one that would have saved me back in my certbot-cron days. An expiry warning seven days out gives you plenty of room to fix the actual problem before anything user-facing breaks.

Common Issues

Challenge stuck pending:

  • HTTP-01: Ingress not routing to cert-manager solver
  • DNS-01: API credentials wrong or missing permissions

Rate limited:

  • Too many certificates requested from Let’s Encrypt
  • Use staging issuer for testing
  • Consolidate into wildcard certificates

Secret not updating:

  • Check Certificate resource status
  • Check CertificateRequest and Order resources
  • Look at cert-manager logs: kubectl logs -n cert-manager deploy/cert-manager

Best Practices

A handful of habits that came out of running this for a while:

1. Use ClusterIssuers

Unless you genuinely need namespace isolation, ClusterIssuers cut down duplication:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
# Available to all namespaces

2. Separate Staging and Production

Always have both issuers:

# letsencrypt-staging for testing
# letsencrypt-prod for production

3. Monitor Expiration

Even with automation, monitor as a safety net:

certmanager_certificate_expiration_timestamp_seconds - time() < 604800

4. Use DNS-01 for Internal Services

If your services aren’t publicly accessible, HTTP-01 won’t work.

5. Standardize Secret Names

Convention like ${service}-tls makes it predictable:

secretName: api-example-com-tls

My Production Setup

Here’s what I actually run. One DNS-01 ClusterIssuer and one wildcard certificate, and almost everything flows from those two resources:

# ClusterIssuer for Let's Encrypt with Cloudflare DNS
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: certs@example.com
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
      - dns01:
          cloudflare:
            apiTokenSecretRef:
              name: cloudflare-api-token
              key: api-token
        selector:
          dnsZones:
            - "example.com"

---
# Wildcard certificate for all services
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: wildcard-example-com
  namespace: cert-manager
spec:
  secretName: wildcard-example-com-tls
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
    - "*.example.com"
    - "example.com"

Why these four:

  • DNS-01 via Cloudflare handles everything, wildcards included, so I never have to think about whether a service is publicly reachable
  • A wildcard certificate means one cert for every subdomain and effectively zero rate limit worry
  • A central namespace keeps the certificate in cert-manager and reflects it out to where it’s needed
  • Prometheus alerts so I find out about a problem days before a user does

Why This Matters

TLS stopped being optional a while ago. Browsers warn on plain HTTP, APIs reject it, and internal traffic needs encryption if you’re taking zero trust at all seriously.

The deeper reason I care about this comes back to friction. Anything that depends on me remembering to do a thing is a system that will eventually fail, because I will eventually forget. Manual certificate renewal is exactly that kind of system, dressed up as a calendar reminder. cert-manager turns TLS into something the cluster maintains on its own: define a certificate once, and it renews forever, the same way for every service and every team.

That’s the kind of automation worth having. Not the flashy kind, the kind that quietly removes a whole category of 3 AM pages and lets you forget the problem exists. The best security control here is the one you stopped having to think about.