[{"content":" TL;DR I ran the five questions from my licence post on the OS under my own cluster. The licence holds. The Talos Linux vendor lock-in sits in two defaults that point at Sidero’s servers, the BSL code Sidero keeps adding around the MPL core, and a company Yardi has owned since 14 September.\nMy licence post ends with five questions to ask of every component you run. I didn’t ask them of Talos, which is the operating system every node in my homelab boots. Nine days before that post went out, Yardi had bought Sidero Labs, the company that makes it.\nMy note for this post already had an answer. Talos is MPL-2.0, so the licence isn’t the risk. The risk is bus factor and a small ecosystem, and the fix is an own build pipeline and an own registry mirror.\nWas the note right? Partly. Here’s where it wasn’t.\nThe licence can’t move without everyone Question one is who owns the copyright, and whether there’s a CLA .\nThe LICENSE file has exactly one commit in its history, the initial one on 3 November 2017, and it said Mozilla Public License 2.0 then. CONTRIBUTING.md asks for a DCO sign-off on every commit and there’s no CLA anywhere in the repository. Every contributor keeps their copyright, so Sidero can’t relicense Talos on its own.\nOmni, Sidero’s management platform, shows what the other setup looks like. Since January its contributing guide takes contributions under 0BSD , and says why, the “flexibility to re-license the project in the future”.\nSo the obvious worry was the wrong one.\nWhat a node talks to out of the box If the licence can’t move, the next question is what a running node needs from Sidero. My note said nothing. The docs say two things.\nDiscovery first. “By default, Talos uses the service registry”, and the discovery page names Sidero’s public instance at discovery.talos.dev. The server is under BSL 1.1 with no additional use grant, and self-hosting it takes “a commercial license”. You can switch it off by removing the DiscoveryServiceConfig documents, so that one’s a config change.\nThen the installer. Since Talos 1.14, released 3 September, “the ghcr.io/siderolabs/installer image is no longer published”, according to the upgrade guide. Installers come from factory.talos.dev. Both of my Talos posts on this blog, the introduction and the NixOS comparison, show an upgrade command pointing at ghcr.io/siderolabs/installer. They were right for v1.6 and v1.7, the versions they show.\nCould I build my own? Yes, and the build docs say the result “can be pushed to any registry you have access to”. But the default build pulls prebuilt layers from ghcr.io/siderolabs, and rebuilding every layer means working through five repositories, not one make. A registry mirror covers container pulls. Installer and boot images need your own imager runs or a self-hosted Image Factory , which is MPL-2.0 apart from an enterprise/ directory.\nQuestion two is the trademark, and it settles what a fork may call itself. Sidero’s design system README says forking is welcome and that the licence “grants no trademark rights”. A redistributed build needs its own name.\nWho pays, and what they did before Here the note was right, and short. Over the last twelve months one maintainer wrote 47% of the commits to Talos, the top three wrote 76%, and at least 90% came from Sidero people.\nQuestion three is who pays them. Yardi does now. Sidero says Talos “remains fully open source under the MPL-2.0 license” and that it runs as a standalone company. The pricing page sells Talos Enterprise Linux at $1,000 per node per year, ten nodes minimum, “the same codebase with a commercial layer on top”.\nQuestion five is whether they’ve done it before. Not to Talos. Around it, yes.\nDate What happened 2021-10-28 Discovery service relicensed from MPL to BSL 2024-02-29 Omni published under BSL from the first commit 2025-12-08 Image Factory gets its BSL enterprise/ directory 2026-05-28 go-vex’s change date set to N/A, so it never converts 2026-06-17 Omni’s grant narrowed to count anything the business “depends” on as production So Sidero keeps the core open and puts the new commercial code in BSL right next to it. I don’t hold that against them, somebody has to pay for that 90%. It does tell me where to look when the next change comes.\nWhere I’d go, and what my note got wrong Question four is whether an exit can start from where I am. The machine config is YAML, and talosctl etcd snapshot gives a plain etcd snapshot. Moving a node onto my own build should be talosctl upgrade --image with my installer, but I haven’t tested that. Leaving Talos entirely means reinstalling each node, because there’s no in-place path to another OS.\nSo I checked my note’s list of Talos properties against Flatcar and NixOS as well. Two of my Talos claims didn’t survive. There’s no dm-verity and no composefs in Talos, and “no drift” is too strong, because talosctl edit machineconfig changes a live node without touching Git.\nTalos Flatcar NixOS Read-only OS squashfs, verified only with Secure Boot /usr with dm-verity, / writable store mounted ro, not verified Atomic update, rollback A/B A/B generations No shell, no sshd yes sshd on by default shell present, sshd off Declarative yes, API edits drift first boot only yes, nix-env drifts Secure Boot, TPM both, opt-in TPM only TPM in-tree, Secure Boot via Lanzaboote Ephemeral, reset yes persistent, flatcar-reset opt-in, no reset command Nothing else on that table does what Talos does on a Kubernetes node, which is why my nodes run it.\nMy note priced the exit at a registry mirror. The real list is discovery switched off, an imager build per release or an Image Factory to run, and a new name if I ever hand the build to someone else.","permalink":"https://tommeurs.nl/en/posts/talos-linux-vendor-lock-in/","title":"Talos Linux Vendor Lock-In: Five Questions, Run on My Own Nodes"},{"content":" TL;DR A pipeline that scans images and writes an SBOM, with nothing checking either at admission, had been green for a year and had stopped nothing. Four steps below: an SBOM Trivy can read back, a cosign signature on the digest with the key in Vault, the SBOM attached as an attestation, and a Kyverno policy that refuses the rest. Then a table of which SLSA Build L3 requirements a self-hosted GitLab runner meets, and the two it doesn’t.\nI inherited a pipeline on a client’s GitLab, self-managed on their own hardware, and on paper it did everything right.\nTrivy scanned every image. A second job wrote an SBOM , added the quarter a customer’s security questionnaire asked whether they produced one. Both jobs were green, and had been for about a year. Both dropped a JSON file into GitLab’s artifact storage, where it sat for thirty days and expired. I couldn’t find anyone who’d opened one.\nMeanwhile the cluster would start anything. Any image from any registry, as long as whoever pushed it held a token. The scan reported on something that had already shipped, and the SBOM was a file with no reader.\nNobody had done anything wrong, and I want to say that before the rest of this. Every job in that file was added by somebody solving the problem their own ticket described, and each one worked. The scan job found vulnerabilities. The SBOM job produced an SBOM. What was missing is the step that turns either of them into a decision, and that step was in nobody’s ticket, so nobody wrote it. By the time I was reading the file, both people whose names were on those commits had moved on.\nThat’s the part that itches, and I’ll admit it’s an occupational habit. I do shadow-CISO work for clients often enough, and the holes I find there are rarely inside a job somebody wrote. They sit in the space between two tickets, where there’s no owner and both sides assume the other one covers it. An auditor sees two green jobs and ticks the box. The cluster never sees either job.\nThe gap Here’s what that adds up to. The supply chain was described in a document, and nothing made the cluster act on it.\nThree holes, and I’d argue they’re one:\nThe SBOM had no consumer. It answers “which of my images contain this library” only if something can find it later, and a job artifact that expires after thirty days isn’t findable. Nothing tied the image to the pipeline. The registry held a tag. Anyone with push rights can move a tag. Nothing in the image said “this came out of that pipeline, from that commit”. The cluster verified nothing. Kyverno was running with a stack of governance policies about resource limits and namespaces, and not one of them cared where an image came from. There’s a compliance version of this too, and it’s worse because it sounds fine. A page in the internal wiki said the platform targets SLSA level 3. It had a date and no author. Nobody still there could tell me which of that specification’s requirements a GitLab runner actually meets, and neither could I until I sat down with it. I’ll get to that further down. The short version is that it doesn’t meet all of them, and one it structurally can’t.\nSo what would actually stop an unsigned image from starting?\nFour moving parts. The first three are preparation, and the fourth is a gate that says no. What follows assumes you already have a GitLab pipeline that builds an image and deploys it; every job below slots into that.\nStep one: an SBOM worth consuming Start with Trivy, which is probably already in your pipeline. The command is short and it has one trap in it:\ntrivy image --format cyclonedx --output sbom.cdx.json \"$IMAGE\" Run that and Trivy prints a line most people scroll past:\n\"--format cyclonedx\" disables security scanning. Specify \"--scanners vuln\" explicitly if you want to include vulnerabilities in the CycloneDX report. So the default gives you an inventory with no findings in it. I think that’s the right default. An SBOM is a parts list, a scan is a judgement about that list on a given day, and they go stale at different speeds. It’s also not what most people think they’re getting. If you want both in one document:\ntrivy image --scanners vuln --format cyclonedx --output sbom.cdx.json \"$IMAGE\" I generate the inventory without findings and scan separately. The parts list is a fact about the image and stays true for as long as the image exists. The vulnerability list is true on Tuesday and wrong on Thursday, so it belongs in a scheduled job that rescans what’s already deployed, not welded into the artefact.\nThe other half of “worth consuming” is that Trivy reads an SBOM back as a scan target:\ntrivy sbom sbom.cdx.json That’s the loop that makes the file worth producing. Six months from now, when a CVE lands in a library you’ve never heard of, you scan the SBOMs of everything currently running instead of rebuilding and rescanning every image.\nStep two: sign the digest, not the tag cosign attaches a signature to an image in the registry. The subtlety is what you point it at.\nIMAGE_DIGEST=$(docker inspect --format='{{index .RepoDigests 0}}' \"$IMAGE_URI\") cosign sign \"$IMAGE_DIGEST\" Signing myapp:1.4.2 signs a name. Names move, even when the version number falls out of your commits and nobody picks it by hand, because a tag is still a pointer and a pointer is still writable. Signing myapp@sha256:9f86d0... signs one specific pile of bytes, and that’s the only version of this that means anything. GitLab’s own tutorial resolves the digest out of docker inspect for exactly this reason, and their documentation adds one more rule I’d follow: build and sign in the same job, so nothing gets a window to touch the artefact in between.\nWhere the key lives, and why there This is the decision the post exists for, so I’ll lay out all three and say which one we landed on.\nOption one: a private key in a masked CI/CD variable. It works, it takes ten minutes, and it’s the one almost everybody starts with. The problem is that the key is now an environment variable inside a build job, which means every script in that pipeline can read it, including a dependency’s install hook. Rotating it means re-signing history or accepting a break in the chain.\nOption two: the key stays in Vault and never leaves. cosign speaks KMS, and HashiCorp Vault’s transit engine is one of the backends:\nexport VAULT_ADDR=https://vault.example.com vault secrets enable transit cosign generate-key-pair --kms hashivault://cosign cosign sign --key hashivault://cosign \"$IMAGE_DIGEST\" The runner never holds the private key. It holds a Vault token that’s allowed to ask for a signature, which is a much smaller thing to steal. Sigstore’s documented policy for this grants read on transit/keys/cosign, update on transit/hmac/cosign/* and transit/sign/cosign/*, and create on transit/verify/cosign, and nothing else. openbao:// is the sibling scheme and behaves the same, so this option survives the move off Vault if you ever make it. Mine is on the list. If you already run Vault or OpenBao for your cluster secrets, this is a config change and no new component.\nOption three: keyless, with Sigstore. No long-lived key at all. GitLab hands cosign an OIDC token, Fulcio issues a certificate that lives for ten minutes, cosign signs with it and throws the key away, and the whole event is recorded in Rekor . The identity in that certificate is your project and your CI config file, which is a far better claim than “someone who had the key”.\nOn GitLab.com the whole of it is the id_tokens block, and it goes in the job that already builds:\nbuild: stage: build image: docker:cli services: - docker:dind variables: COSIGN_YES: \"true\" IMAGE_URI: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA id_tokens: SIGSTORE_ID_TOKEN: aud: sigstore before_script: - apk add --no-cache cosign - docker login -u gitlab-ci-token -p \"$CI_JOB_TOKEN\" \"$CI_REGISTRY\" script: - docker build -t \"$IMAGE_URI\" . - docker push \"$IMAGE_URI\" - IMAGE_DIGEST=$(docker inspect --format='{{index .RepoDigests 0}}' \"$IMAGE_URI\") - cosign sign \"$IMAGE_DIGEST\" One job, because IMAGE_DIGEST is a shell variable and a second job gets a different shell. Split the signing out and you’re passing the digest through a dotenv report, and the image sits in the registry unsigned for however long the pipeline takes to reach the next stage.\ncosign picks SIGSTORE_ID_TOKEN out of the environment on its own. COSIGN_YES replaces the interactive confirmation, since a pipeline has nobody to confirm anything.\nNow the part the tutorials skip. On a self-hosted GitLab, keyless costs you three services. The public Sigstore deployment only trusts GitLab.com pipelines, so GitLab’s own documentation for self-managed instances tells you to run your own Fulcio, your own Rekor, and your own Certificate Transparency log. The CT log isn’t optional, because a Fulcio without one issues certificates cosign refuses to verify. Rekor needs a persistent signing key, because an in-memory signer regenerates on restart and invalidates everything logged before it.\nWe costed this on the client’s self-managed instance and took the key in Vault instead. Three more stateful services, one of them an append-only log where losing the key is unrecoverable, is a real bill even for a team with an operations budget, and what it buys is a better identity story on top of signing that already works. Sovereignty means making that call with the costs visible, and here the simpler option was the honest one. If you’re on GitLab.com, take the keyless path. You get the strictly better identity for four lines of YAML.\nTwo gotchas from that page if you do build the stack. Container image signing needs cosign v3 and a registry on a resolvable HTTPS hostname, so an instance whose registry_external_url is a bare IP can sign files and can’t sign images. And Fulcio’s /healthz reports healthy even when its OIDC provider failed to initialise, which is a fun afternoon.\nVerifying, and the double slash Whichever option you pick, the verify command is where the identity shows up:\ncosign verify \"$IMAGE_DIGEST\" \\ --certificate-identity \"https://gitlab.com/my-group/my-project//.gitlab-ci.yml@refs/heads/main\" \\ --certificate-oidc-issuer \"https://gitlab.com\" That double slash between the project path and the CI config path is deliberate and required. Write one slash and you get:\nError: none of the expected identities matched what was in the certificate, got subjects which tells you nothing about the actual mistake. GitLab documents it. I still lost twenty minutes to it.\nOne more constraint worth knowing before you plan around it. The id_tokens block has to live in the project being built. Auto DevOps, CI files included from another repository, and child pipelines don’t work, which GitLab tracks as Support keyless signing when CI config is located outside of the project. If your organisation centralises pipeline templates, that’s a design constraint and you want to know it before the templates exist.\nStep three: attach the SBOM to the image Signing proves an image came from your pipeline. Attesting attaches a signed statement about that image, and the SBOM is exactly such a statement. Trivy documents the pairing:\ntrivy image --format cyclonedx -o sbom.cdx.json \"$IMAGE_DIGEST\" cosign attest --key hashivault://cosign --type cyclonedx --predicate sbom.cdx.json \"$IMAGE_DIGEST\" The SBOM now travels with the image, in the registry, signed by the same key. Pull the image anywhere and the parts list comes with it. Compare that with a JSON file in artifact storage and thirty days to live.\nRead it back with:\ncosign verify-attestation --key cosign.pub --type cyclonedx \"$IMAGE_DIGEST\" \u003e sbom.intoto.jsonl trivy sbom ./sbom.intoto.jsonl Two practical notes. cosign attest writes to the OCI registry, so the job needs push permission on that repository, and --no-upload exists if you only want to look at the statement. And only CycloneDX-type attestations can be fed back to trivy sbom, so if you were going to pick SPDX for some other reason, pick it knowing that.\nStep four: make the cluster refuse Everything so far produces evidence. This is the part that acts on it.\nA note on which Kyverno API to write against, because this changed recently and it changes what you should type. The ClusterPolicy kind with its verifyImages rule is deprecated as of Kyverno v1.19 and slated for removal in v1.20. The replacement is ImageValidatingPolicy, which reached stable in the same release and expresses its checks in CEL. If you write a new policy today, write this one.\nStart in audit mode. I mean it, every time.\napiVersion: policies.kyverno.io/v1 kind: ImageValidatingPolicy metadata: name: require-signed-images spec: validationActions: [Audit] failurePolicy: Fail webhookConfiguration: timeoutSeconds: 15 matchConstraints: resourceRules: - apiGroups: [\"\"] apiVersions: [\"v1\"] operations: [\"CREATE\", \"UPDATE\"] resources: [\"pods\"] matchImageReferences: - glob: \"registry.example.com/platform/*\" attestors: - name: pipeline cosign: key: data: | -----BEGIN PUBLIC KEY----- MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE... -----END PUBLIC KEY----- validations: - expression: \u003e- images.containers.map(image, verifyImageSignatures(image, [attestors.pipeline])).all(e, e \u003e 0) message: \"Image is not signed by the platform pipeline\" The public key comes straight out of Vault, and it’s public, so pasting it into a policy is fine:\ncosign public-key --key hashivault://cosign Note matchImageReferences is a glob on your own registry and no *. Scope it deliberately. Everything you didn’t build is going to fail this check, starting with the ingress controller.\nLeave it on Audit and go read the policy reports for a week. You’ll find images you forgot you were running. That week is the whole point of audit mode, and skipping it is how a policy rollout becomes an outage.\nThen verify the SBOM too, which is the step that closes the loop I opened at the top. Add an attestations: block, and grow the validations: list you already have from one entry to three. There’s exactly one validations: key in a policy, so these replace it instead of sitting beside it:\nattestations: - name: sbom intoto: type: https://cyclonedx.org/bom validations: - expression: \u003e- images.containers.map(image, verifyImageSignatures(image, [attestors.pipeline])).all(e, e \u003e 0) message: \"Image is not signed by the platform pipeline\" - expression: \u003e- images.containers.map(image, verifyAttestationSignatures(image, attestations.sbom, [attestors.pipeline])).all(e, e \u003e 0) message: \"Image has no signed SBOM attestation\" - expression: \u003e- images.containers.map(image, extractPayload(image, attestations.sbom).predicate.bomFormat == 'CycloneDX').all(e, e) message: \"SBOM attestation is not CycloneDX\" https://cyclonedx.org/bom is the predicate type cosign attest --type cyclonedx writes, and intoto: is how you name a predicate type. Kyverno also has a referrer: form that matches an OCI 1.1 artifact type, and it’s the wrong one here, because cosign attest writes a tag-based attestation at sha256-\u003cdigest\u003e.att by default and no referrer. Pick referrer: and the policy reports the attestation missing while step three reports success, which is a miserable hour.\nThe .predicate in the third expression is there for the same reason. cosign wraps your BOM in an in-toto statement, so bomFormat sits one level down and no longer at the top.\nextractPayload() errors unless verifyAttestationSignatures() ran first on the same attestation, which is the correct order anyway. Reading a payload you haven’t verified means trusting an attacker’s parts list.\nIf you run a self-hosted Sigstore after all, the keyless attestor takes a trustedRoot.expression that reads your own trust root JSON out of a ConfigMap, so the policy doesn’t have to trust the public Sigstore instance.\nWhen the reports are quiet, flip one word:\nvalidationActions: [Deny] Prove that it blocks Don’t take the policy’s word for it. Push an unsigned image and watch the API server refuse it:\ndocker pull alpine:3.22 docker tag alpine:3.22 registry.example.com/platform/unsigned-test:1 docker push registry.example.com/platform/unsigned-test:1 kubectl run blocked --image=registry.example.com/platform/unsigned-test:1 Error from server: admission webhook \"ivpol.validate.kyverno.svc-fail\" denied the request: ImageValidatingPolicy require-signed-images: Image is not signed by the platform pipeline Then run the same test with an image your pipeline built and signed, and watch it start. Both halves matter. A policy that blocks everything is broken too, and that failure looks like success on a compliance report.\nPut both commands in a scheduled pipeline. Policies drift, webhooks get disabled during an incident and nobody turns them back on, and a cached result can hide a problem for an hour. Which brings me to a caching detail worth knowing. Kyverno caches image verification results, with imageVerifyCacheTTLDuration defaulting to 60m. A signature you revoke in the registry is still accepted until that entry expires.\nSLSA level 3, and what a GitLab runner actually reaches Now the compliance claim from the top, taken seriously.\nFirst, a citation correction that catches a lot of writing on this subject, including things I had bookmarked. SLSA v1.0 is retired. The current specification is v1.2, and it reorganised the URLs, so the level descriptions now live at build track basics and the detail at build requirements. The Build track text itself barely changed; v1.2’s addition is a Source track. If you’re citing spec/v1.0, you’re citing a retired document.\nThe Build track goes like this. L1 is provenance that exists. L2 is signed provenance from a hosted build platform, aimed at “tampering after the build”. L3 is a hardened build platform, aimed at “tampering during the build”, and it adds two requirements: Isolated and Provenance is Unforgeable.\nGitLab’s own claim is precise and easy to miss. The documentation states that “GitLab offers a SLSA level 2 compliant provenance statement”, produced by the runner itself when you set RUNNER_GENERATE_ARTIFACTS_METADATA: \"true\". Level two. There’s a separate SLSA level 3 feature, and reading its header is the fastest way to know whether it applies to you. Tier Ultimate, Offering GitLab.com, Status Experiment, behind a feature flag called slsa_provenance_statement that’s off by default, and the project has to be public. A self-managed instance can’t use it at all.\nSo where does that leave a self-hosted runner? GitLab publishes no gap analysis, so what follows is my reading of the spec text against how a runner works, and nobody’s official position.\nSLSA v1.2 Build L3 requirement A GitLab runner, docker executor Hosted, not an individual’s workstation Met Ephemeral environment provisioned for each build Met. Fresh container per job Overlapping builds cannot influence one another Met, to the extent container isolation holds Build output identical with and without the cache Not proven. cache: is shared by key across jobs Signing material inaccessible to user-defined build steps Not met Every provenance field generated in a trusted control plane Not met The last two are the ones that matter, and they’re structural, so no configuration you forgot fixes them. The spec says signing material “MUST NOT be accessible to the environment running the user-defined build steps”. Whether you hand the job a cosign key from a CI variable or a SIGSTORE_ID_TOKEN from id_tokens, the credential arrives as an environment variable in the same shell that runs your build script. And the provenance the runner generates is produced on the same machine that runs the job, so a tenant who can run arbitrary commands there is in a position to influence it. The spec’s word for the property you need is “strongly resistant to forgery by tenants”, and a control plane you operate as the same person who writes the pipeline isn’t that.\nThe Vault option narrows this. The private key genuinely never reaches the runner. The job can still ask Vault to sign something of its choosing, so it moves the problem from key theft to signing-oracle abuse. Smaller problem. Still a problem.\nOne line from the spec keeps the whole exercise honest: “Build L3 is limited to ensuring that a well-intentioned build runs securely.” It doesn’t stop you from writing a risky build. Nothing here does.\nMy conclusion, and I’d rather write it plainly than hedge. A self-hosted GitLab reaches Build L2 and most of L3’s isolation properties, and doesn’t reach L3. I’ve stopped putting a level number in documents. What I document instead is the verification command, because that’s a claim a reader can check.\nHardening the runner itself is a real subject and a large one, and it gets its own post.\nWhat this does not buy you Worth being blunt about, because the diagram makes it look like more than it is.\nA signature proves an image came from your pipeline. It says nothing about whether the code in it is any good. Sign a backdoor and you get a correctly signed backdoor, and the policy will happily admit it.\nVerification is a dependency too. cosign v3.1.3, released 6 August 2026, fixes GHSA-fx35-mq7g-6g98, rated 7.4. When a legacy JSON bundle’s certificate failed to parse as X.509, cosign fell back to reading it as a raw public key, which skipped chain validation and made --certificate-identity and --certificate-oidc-issuer do nothing. Image verification isn’t affected, so none of the commands in this post are, but verify-blob is. The shape of the bug is what I’d take away. The thing checking your signatures is software with bugs in it.\nAnd the versions in this post move fast. On the day I wrote it: cosign v3.1.3, Kyverno v1.19.1, Trivy v0.74.0. Kyverno’s removal of ClusterPolicy in v1.20 is an announced plan, and no release has shipped it yet. Check your own versions instead of trusting mine.\nThe bit that mattered Three of the four steps in this post produce evidence. One acts on it, and that pipeline had the three and not the one for about a year. Most supply chain work I see has that shape. Enormous effort goes into generating artefacts, and almost none into the thing that refuses.\nWhat I got out of it has nothing to do with the level number or the questionnaire. I can answer “what’s running in this cluster and where did it come from” with a command now, where before it was a belief. Everything on this site comes back to that preference. I’d rather understand what I run than trust it.\nStart with Audit and one glob, read the reports for a week, and flip it to Deny when they go quiet. Keep pushing the unsigned test image on a schedule after that.","permalink":"https://tommeurs.nl/en/posts/supply-chain-security-gitlab-ci/","title":"Supply Chain Security in GitLab CI: SBOM, cosign, and the Policy That Blocks"},{"content":" TL;DR A vendor freezing its free edition belongs in your risk register next to the expiring certificate. Thirty-six dated cases below, three kinds of event, and five questions to score your own stack. My own register had the Vault row missing for three years.\nOn 11 September, Portainer’s CEO published “Portainer 3.0 is coming, and here’s what it means for you”. I read it the Monday after and shrugged.\nHere’s what it says. 3.0 is a Kubernetes-first codebase for enterprise customers, and “we are not cutting a separate CE build of it”. Community Edition stays on 2.x. 2.45 LTS “is the last release in the 2.x line”. A few days later they added a clarification: 2.45 keeps getting security fixes, bug fixes and back-ports of any 3.x feature that has a Docker API equivalent, “for as long as that LTS is supported”. The lifecycle page puts that at May 2027. Want 3.x for free? There’s a Business Edition key called 3 Nodes Free, one per organisation.\nI want to be fair to Portainer, because the rest of this is going to sound like I’m not. Nothing in that post is sneaky. The licence on the code you’re running didn’t change. It’s still zlib, the repository says so. They explained themselves, they’re supporting what they shipped, and they wrote the uncomfortable sentence themselves: releasing 3.x as CE “would misrepresent what it is and who it is for”.\nSo if you run Portainer on a Docker host, you probably shrugged too. It still works. It’ll still work in May 2027.\nThat shrug is the whole post. It’s what a risk sounds like when nobody wrote it down.\nThe row my register was missing I keep a risk register for client platforms. It’s a boring spreadsheet. Certificates that expire, a Debian release going end of life, a Kubernetes version that drops an API. Each row gets a date, a blast radius, and a rough cost to get out. Nobody argues about those rows. They’ve all happened to someone in the room.\nHere’s the Portainer news as one of those rows:\nRisk. The vendor of the container management UI freezes the free edition.\nLikelihood. High. The vendor sells a commercial edition, and no foundation owns the code or the name.\nImpact. The upgrade path acquires an invoice or a node cap. The free edition receives security fixes until a date the vendor sets.\nTrigger. A blog post.\nMitigation. Know the exit before the post. Keep the data in a format the alternative reads.\nI’d never seen that row in anyone’s register. Mine included, until this month. We treat the licence on a dependency like the licence on the kernel, something that’s just true. Then a company changes it in a quarterly planning meeting and everyone’s surprised.\nNow imagine you’d written that row down in 2024. The Portainer post lands and you already know the answers. 3 Nodes Free covers you if you run three environments or fewer under one organisation. You might not care about 3.x at all, since a Docker home lab isn’t who it’s for. And if you ever want out, Komodo and Arcane both read compose stacks, though neither’s a fork. Ten minutes of reading, and you’re back to whatever you were doing.\nThirty-six rows, three kinds of event So I built the register I wish I’d had. I went through every case I could remember and a few I’d forgotten, and read each one on the vendor’s own page on 23 September 2026. Anything I couldn’t confirm there, I dropped. That left forty-three events. Some fold into each other, a reversal belongs with the move it reversed and a second relicensing with the first. Thirty-six rows.\nDate Project What changed The free path today Fork 2016-08-16 MariaDB MaxScale GPLv2 to BSL, the licence’s first use anywhere, seven years before HashiCorp reached for it. “The license of MaxScale has been changed from GPLv2 to MariaDB BSL” Converts to GPL after four years None 2018-08 Redis modules AGPL to Apache-2.0 with the Commons Clause, then RSAL in February 2019 The server stayed BSD until 2024 None 2018-10-16 MongoDB AGPLv3 to SSPL Free to self-host. The OSI said in 2021 it is not open source None survived 2018-11-29 Neo4j Enterprise Enterprise source out of the repository with 3.5, “requires a commercial license” Community stays GPLv3 2018-12 Emby 4.x closed source, by the fork’s account 3.5.2 was the last GPL release Jellyfin, from 3.5.2. 12.1 this month 2018-12-14 Confluent Platform KSQL, Schema Registry and REST Proxy from Apache-2.0 to the Confluent Community License Free to run, no SaaS Not a fork question. Kafka itself stays Apache-2.0 at the ASF 2019-11-06 Sentry BSD-3 to BSL , then to FSL in 2023 Free to self-host, competing SaaS excluded, converts to Apache-2.0 or MIT after two years None 2019-12-04 Chef Binaries behind the Chef EULA Source stays Apache-2.0 Cinc, “same code, same compatibility, no EULA”. 19.3.14 in May 2020-04-30 Anaconda Terms of service on the package repository Free under 200 employees, the line since 2024 Not a fork question. Miniforge and conda-forge, free for everyone 2020-05-26 CoreOS Container Linux End of life, two years after Red Hat bought CoreOS. Apache-2.0 throughout Fedora CoreOS is the successor Flatcar, forked in 2018, CNCF incubating since 2024. Stable 4757.2.0 this month 2020-11-16 Graylog To SSPL with 4.0 Free to self-host None 2020-12-08 CentOS Linux “CentOS Linux 8, as a rebuild of RHEL 8, will end at the end of 2021.” Stream takes its place, “serving as the upstream (development) branch” The no-cost RHEL is the developer subscription, 16 systems Rocky Linux, GA June 2021, 10.2 in May 2026. AlmaLinux, stable March 2021, 10.2 in May 2026 2020-12-08 Mapbox GL JS BSD-3 to Mapbox terms with v2.0.0 “a billable map load occurs whenever a Map object is initialized” MapLibre GL JS, from v1.13. 6.11.1 today 2021-01-14 Elasticsearch, Kibana Apache-2.0 to SSPL and Elastic License Free to self-host. AGPL added back in 2024 OpenSearch, under the Linux Foundation since 2024. 3.8.0 in August 2026 2021-04-21 Grafana, Loki, Tempo Apache-2.0 to AGPLv3 Still open source. In the table to show where the line is None needed 2021-08-31 Docker Desktop Subscription above 250 employees or 10M USD revenue Free below the line. The threshold still stands Not a fork question. Podman Desktop and Rancher Desktop 2021-09-27 Airbyte Core from MIT to Elastic License 2.0 Free to run, no hosted service. Connectors stay MIT 2022-09-07 Akka Apache-2.0 to BSL, free under 25M USD revenue Free below the line, converts after three years Apache Pekko, from 2.6.20 2022-10-24 Gitea Gitea Ltd takes the trademark and domains Licence unchanged, MIT Forgejo, under Codeberg e.V. Hard fork since 2024, GPLv3+ since v9. 16.0.5 this month 2023-06-21 RHEL source “CentOS Stream will now be the sole repository for public RHEL-related source code releases.” Customers keep source through the portal Unchanged for customers. Rebuilders lose the 1:1 input AlmaLinux dropped 1:1 for ABI compatibility. CIQ, Oracle and SUSE founded OpenELA 2023-08-10 Terraform, Vault MPL-2.0 to BSL, converts to MPL-2.0 four years after each release Free to self-host unless you sell a competing hosted product. Vault Community Edition was not discontinued OpenTofu, LF, 1.12.6 in August 2026. OpenBao, LF, 2.6.3 today 2023-12-11 VMware, by Broadcom Perpetual licences end of sale, nineteen days after the acquisition closed. “VMware vSphere Hypervisor (free edition)” on the list Subscription only, VCF or VVF. Support runs to the end of the contract. Free ESXi came back in April 2025, “for non-production use” No fork, there is no source. Alternatives, yes: Proxmox VE, XCP-ng, KubeVirt 2024-02-14 nginx Nothing. Lead developer left F5 over a security policy BSD-2, identical on both sides freenginx. A governance fork, in the table for the same reason as Grafana 2024-02-21 Linkerd “as of 2.15, we will no longer be producing open source stable release artifacts”. CNCF graduated since 2021 Edge releases stay open. Stable is Buoyant Enterprise, free under 50 employees 2024-03-20 Redis BSD-3 to RSALv2 and SSPLv1 Free to self-host. AGPL added back in May 2025 Valkey, LF, eight days later. 9.1.2 in August 2026 2024-04-09 Camunda 8 Camunda License 1.0 from 8.6, “Free for Development only. License needed for production use” Non-production only None 2024-08-22 Sourcegraph “Sourcegraph transitioned to a private monorepo.” Was Apache-2.0 A read-only snapshot, archived September 2024 Nothing left to fork 2024-11-07 Puppet Binaries to “a private, hardened, and controlled location”, EULA 25 nodes free. Source stays Apache-2.0 OpenVox, Vox Pupuli, “a drop-in replacement”. 8.29.0 this month 2024-11-18 CockroachDB Core retired. Enterprise free under 10M USD revenue The free edition is gone. The free licence is a revenue test None 2024-12-18 ScyllaDB “ScyllaDB OSS AGPL 6.2 will stand as the final OSS AGPL release”. Source-available from 2025.1 Free tier capped at 10 TB and 50 vCPUs 2025-05-24 MinIO Console reduced to an object browser, LDAP and OIDC login moved to the paid product AGPLv3, unchanged. See below None 2025-07-16 Bitnami Broadcom again, twenty months after VMware. Free catalogue cut to “the most recent ’latest’ tags of the limited community-tier subset”, the rest to a legacy repository with “no further updates” Source stays Apache-2.0 on GitHub Not a fork question. Build your own images 2025-10-15 MinIO “distributed as source code only”, no binaries, no images Last release the same day, a CVE fix None 2025-12-03 MinIO “under maintenance and is not accepting new changes” Security fixes “case-by-case” None 2026-04-25 MinIO Repository archived. “THIS REPOSITORY IS NO LONGER MAINTAINED” AIStor Free, a licence key, or AIStor Enterprise None 2026-09-11 Portainer No CE build of 3.x. 2.45 LTS is the last 2.x release zlib, unchanged. Supported until May 2027. 3 Nodes Free for 3.x None as of today Reading them in one sitting, three different kinds of event fell out, and the advice is different for each.\nRelicensing is the one everyone argues about. The code stays public, the licence on new releases changes. What people forget in the first week of outrage is that old releases keep their old licence forever. The binary you’re running tonight is fine. Your upgrade path is what changed. MongoDB, Elastic, Redis, HashiCorp, Sentry and Akka all did this.\nEdition freeze is quieter, and I think it’s the more dangerous one. The licence doesn’t move at all. The free edition just stops getting the product’s future. Portainer did this. CockroachDB did it in 2024, when it retired Core and replaced it with a free Enterprise licence for companies under 10 million dollars in revenue. Red Hat did it twice. December 2020, the free rebuild of RHEL stopped and an upstream development branch took over the CentOS name. June 2023, the public source that Rocky and Alma rebuilt from went behind the customer portal. Not one licence changed in either move. The GPL never came into it. Chef did it in 2019, source stays Apache-2.0 and the binaries go behind a EULA, and Puppet did the same in 2025. Both got a community rebuild. And MinIO did it in a way I’ll get to.\nProduct going paid is the third. Docker Desktop in 2021, where the engine stayed Apache-2.0 and the desktop app grew a subscription above 250 employees or 10 million dollars. And VMware in December 2023, which is the same thing at a scale that ends careers. Nineteen days after Broadcom closed the acquisition, perpetual licences went end of sale, the free hypervisor with them, and everything became a subscription to one of two bundles. There’s no fork, because there was never any source. There are alternatives, Proxmox VE, XCP-ng, or KubeVirt inside a cluster you already run, and every one of them means migrating every VM rather than swapping a binary underneath. Most of my clients live in this row. It’s also the row that convinced me the licence is a distraction. VMware was never open, and the risk was there all the same.\nTwo rows are in the table to keep me honest. Grafana Labs moved Grafana, Loki and Tempo to the AGPL in 2021, and the AGPL is an OSI-approved licence. That’s open source, full stop. If I called it the same thing as SSPL, anyone who’s read both licences would stop reading, and they’d be right to. nginx in 2024 was a governance fight. The LICENSE files on nginx.org and freenginx.org are byte for byte identical. A table that only shows the scary moves is a table you shouldn’t trust.\nThe MinIO row This one hurts, because it was in my cluster. For four months this year MinIO held Loki’s chunks, the Postgres backups and Longhorn’s backup target. And its free edition died without a single licence event. AGPLv3 in the first release, AGPLv3 in the last.\nWatch the dates. May 2025, a release tagged “Breaking Release” cuts the embedded console down to an object browser and moves LDAP and OIDC login to the paid product. October 2025, the README says “distributed as source code only”, and an issue asking for the Docker image is closed as “working as intended”. December 2025, “under maintenance and is not accepting new changes”. April 2026, repository archived.\nFour steps in eleven months, and the licence page never changed. If your register only watches the licence field, MinIO walks straight past it.\nWhat any of these costs Doesn’t matter which kind. The bill has the same four lines.\nThe free edition keeps working and stops growing. Fine next year. Then Kubernetes drops an API version and it doesn’t follow. Your upgrade path now has an invoice, a revenue test or a node cap attached. All three are in the table. If there’s a fork, moving to it is a migration. The data format matches on day one and drifts from there. There’s only a fork if the project was big enough to attract one. Portainer has none as I write this. MinIO has none. Small projects don’t get an OpenBao. What a foundation actually buys you I expected this section to be short. The forks did better than the outrage of 2023 predicted.\nOpenTofu forked Terraform fifteen days after the announcement, joined the Linux Foundation on 20 September 2023, went GA in January 2024, and shipped 1.12.6 in August 2026. OpenBao went GA in July 2024 and released 2.6.3 the day I’m writing this, with Adfinis, ControlPlane, GitLab and SAP on its supporters page. Valkey forked Redis eight days after the announcement, with AWS, Google Cloud and Oracle behind it, and shipped 9.1.2 in August. OpenSearch is five years old, has its own foundation under the Linux Foundation, and shipped 3.8.0 in August too.\nAll four alive. And here’s the part I didn’t expect. When Elastic added the AGPL back in August 2024, and Redis did the same in May 2025, none of the forks folded. A vendor changing its mind doesn’t unwind a fork. Once it exists you’ve got a second supplier for good, and the original gets cheaper for everyone.\n“Donated to a foundation” is going to be my recommendation, so I wanted to know what it actually buys. I read the CNCF charter, updated June 2025, and the graduation template. Here’s what’s guaranteed.\nThe trademark moves to the Linux Foundation, section 11(a)(i). A company that walks away can’t take the name. OpenTofu, OpenBao and Valkey all lacked this on day one, and each had to pick a new name and start marketing from zero. OSI-approved licences only, Apache-2.0 outbound, sections 11(a)(ii) and 11(b)(iii). A move to BSL or SSPL would need the Governing Board to grant an exception. Not impossible, but not one company’s call. Graduation needs maintainers from at least two organisations, with an org-balance mechanism so that “no single organization controls governance decisions”. And here’s what isn’t guaranteed, which nobody mentions when they say “it’s a CNCF project”:\nThat anyone keeps paying the maintainers. No clause obliges a company to fund anything. That the maintainer company doesn’t sell a commercial derivative right next to it. That the project stays alive at all. Section 6(a)(iii) lets the TOC archive it. That an incubating project has more than one organisation behind it. That test only comes at graduation. That list isn’t theoretical. Linkerd graduated in July 2021. In February 2024 Buoyant wrote “as of 2.15, we will no longer be producing open source stable release artifacts” and put stable builds behind Buoyant Enterprise, free below 50 employees. Licence untouched. Governance untouched. Graduated status untouched. The charter had nothing to say about it, because it never promised binaries.\nI checked the four CNCF projects my own cluster leans on hardest. Argo graduated in December 2022, Cilium in October 2023, Kyverno in March 2026. Longhorn has been incubating since November 2021 and hasn’t graduated, which means SUSE can still be the only organisation with maintainers. I run Longhorn. It gets a lower score than Cilium in my register, and I’m fine with that, as long as the score is written down and not just felt.\nLoki and Tempo get the same note. AGPL, so open source. Single vendor, so not a foundation. The exposure there is question three at the end, who pays the maintainers, rather than the licence. I run Loki too.\nAnd Forgejo, the git forge this whole cluster deploys from, is a fork itself. Gitea Ltd took the trademark and domains in October 2022, Forgejo started two months later under Codeberg e.V., went hard fork in February 2024, and moved from MIT to GPLv3+ with v9. So the fork I picked has done a licence change of its own, in the direction I happen to like. Question one applies to it all the same.\nMy own cluster, from the git log This is where I stop pointing at other people’s tables.\nThree years after HashiCorp’s announcement, my cluster runs hashicorp/vault:2.1. Renovate bumped it from 1.21 to 2 on 13 August and to 2.1 on 15 September, and I merged both without a second thought. The chart comes from helm.releases.hashicorp.com. The LICENSE file in that repository now names IBM as licensor, since the acquisition closed in February 2025. I put Vault into this cluster on 5 April 2026, when OpenBao was a year past its first stable release.\nDo I have reasons? Sure. It was the chart I knew. None of them was a register entry, and I’ve been recommending open source to clients the whole time, which is a little embarrassing to type out.\nSo why is it still Vault? Partly because Vault works, and the BSL says in so many words that internal hosting is “not considered a competitive offering”, so a home lab is exactly the user the licence leaves alone. Mostly because I’d never counted what leaving would cost. So I did, from the repository rather than from a whiteboard.\nThe chart. OpenBao ships its own. The values don’t map one to one, and mine has six months of small decisions in it, the telemetry stanza among them. Three CronJobs that use the Vault image. Auto-unseal, the nightly raft snapshot, and the nightly export that walks the whole KV tree and encrypts it to an age key. Each one calls vault. On the other side the CLI is bao, which reads BAO_ADDR first and falls back to VAULT_ADDR. A ClusterSecretStore called vault-backend. Every ExternalSecret in the cluster points at it. That name is the one piece of luck here, because nothing downstream changes if the store behind the name does. An alert group on vault_core_unsealed and vault_autopilot_failure_tolerance, plus 1072 vault_* series in Prometheus. OpenBao kept the vault prefix on its metrics but moved the service registration labels from vault to openbao. The chart’s Services select on those labels. So did my scrape config, until I pointed it at the headless Service. Small stuff, and small stuff is what empties a dashboard. The data. This is the cost that grew while I waited. OpenBao’s migration guide is an in-place node swap, with a warning box: “This guide was only tested against Vault 1.14.1. Newer versions are untested and unsupported: OpenBao makes no guarantees about storage compatibility with Vault.” I’m on 2.1, Vault doesn’t do downgrades, so there’s no way back to 1.14.1. The migration isn’t a swap anymore. It’s my disaster recovery export, replayed into an empty OpenBao with bao kv put, plus every policy, lease and auth mount rebuilt by hand. A document called 09-secrets-vault.md, and six months of typing vault kv put. Add it up and it’s a week of evenings, with a rollback plan for the one step that touches data. Vault for beginners on this blog still says Vault, and it’ll keep saying that until the migration’s done, because I’m not going to describe a lab I don’t run.\nAnd why hasn’t it happened yet? Paid work comes first. In my lab this row bothers nobody but me. At clients the same row, with VMware or Red Hat in it, costs serious money, and those clients know I’ve been talking about open source for years, so they’re queued up waiting for my time. My own migration is behind them in that queue. Vault keeps answering requests in the meantime, and no risk register I’ve ever seen has “a week of evenings” as a budget line.\nThe one I did do MinIO is the migration I did get done. It’s also the one time the register worked, even though it only existed in my head.\nI added MinIO on 5 April 2026, on a 250 GiB Longhorn volume. The community repository had been in maintenance mode since December and was archived on 25 April, twenty days after my commit. I have the commit date to prove it. It installed, it ran, it would’ve kept running for years. But I knew MinIO had gone vague about its source and its licence, and that’s why I was already looking around. Otherwise August would’ve replaced MinIO with a newer MinIO.\nStorage set the month. Longhorn was replicating the volume twice, MinIO wanted to replicate on top of that, and the pool was full. I didn’t want Ceph in a home lab, that’s overkill. Garage replicates itself onto a plain disk per node, so Garage it was. Five consumers moved one at a time, Loki’s chunks, the Postgres backups, Longhorn’s backup target, the Vault export, Forgejo’s storage, with MinIO kept running until its request counter read zero for half an hour. Then it went, on 11 August, and the snapshot chain under that volume handed back 470 GiB. Nearly twice what the volume claimed.\nTwo things broke, both in the cleanup. I tried to cut one field out of a KV secret with sed on the JSON response and wrote the whole API response back as the secret, request id and lease metadata included. vault kv rollback -version=3 saved me. Then I removed the OIDC client from one of two copies of the Keycloak realm and missed the second, which would’ve quietly stopped syncing within the hour. Neither mistake had anything to do with MinIO. Both had to do with doing a migration by hand on a weekday evening. That, more than any licence text, is what every row in that table costs.\nPortainer, the row this post started with, is the one I don’t have. The Docker host that runs my household apps, Nextcloud and a Minecraft server among them, is compose files and a shell. That announcement cost me nothing, and I did nothing to earn that. Write it down anyway. The next announcement will be about something you did install.\nFive questions per component The way out is smaller than the table makes it look. Five questions per component, an afternoon’s work, and the answers go in the register with a date on them.\nWho owns the copyright, and is there a CLA ? A CLA hands the company the right to relicense tomorrow. A DCO , which the kernel and the CNCF use, leaves the copyright with each contributor, so a relicensing needs every one of them to agree. Who owns the trademark? If a foundation does, a fork keeps the name and the search results. If the company does, the fork starts as a stranger. Who pays the maintainers, and how do they make money? Nearly every relicensing in the table came from a single vendor whose revenue depends on a hosted or supported version of the same code, and whose announcement names someone else selling it. You can read that on the pricing page years before the blog post. Could a fork or an alternative start from where you are? Is the data format documented, and does something else read it today? For Vault, a raft snapshot worked until 1.14 and now it’s the KV export. For Portainer, the stacks are compose files. For MinIO, mc mirror to anything that speaks S3. Has the vendor done this before? Sentry twice, CockroachDB twice, Red Hat twice, Broadcom twice, Redis three times if you count the modules in 2018, MinIO four times in eleven months. Elastic and Redis did it once and came halfway back. A vendor that’s done it once has shown you the procedure. Then decide per kind. Vault-shaped: price the exit and put a date on it, because the gap grows every month. MinIO-shaped: keep the data in a format the alternative reads and the decision can wait, because a frozen S3 server is still an S3 server. Portainer-shaped: take the free key and get on with your day.\nSelf-hosting only moves this risk. On a managed service the vendor’s business model changes your bill. On your own hardware it changes your upgrade path. A frozen edition keeps running on your hardware and a discontinued managed service doesn’t, and that difference is the sovereignty argument one layer down. You own the binary that runs tonight and the data in it. Next year’s release was always going to belong to whoever sets the vendor’s revenue target.\nNobody in that table did anything wrong. Companies with payroll changed the terms they were allowed to change, said so politely, and kept supporting what they’d shipped. I read Portainer’s post on a Monday and shrugged, and the shrug was the problem, because I’d never written the row down.\nThe Vault row is in the register now, dated 23 September 2026. OpenBao is on the list. Behind the clients. Sorry, lab.","permalink":"https://tommeurs.nl/en/posts/license-change-supply-chain-risk/","title":"Your Dependency's Business Model Is Part of Your Threat Model"},{"content":" TL;DR The Cyberbeveiligingswet has applied since 15 August 2026. Below, the ten measures of article 21 mapped onto controls that actually run, a Kyverno policy that proves one of them, the reporting deadlines as three Prometheus alerting rules, and the four duties no YAML covers. The first run of that policy found seven pods in my own cluster I’d have sworn were mirrored.\nThe Cyberbeveiligingswet has applied in the Netherlands since 15 August 2026. It’s the Dutch implementation of NIS2 , roughly 22 months after the directive said it should be, and it’s why the question at work stopped being hypothetical. Nobody asked me whether the cluster was secure. The question was narrower and a lot harder. Which measure in article 21 does this control satisfy, and where do we have nothing at all?\nAlmost every NIS2 text I could find was written by people who’ve never typed kubectl. Almost every Kubernetes security text reads as if there’s no such thing as a law. This post sits between the two. It maps the ten measures of the duty of care onto controls that run. Then a policy that blocks, and the reporting deadlines as alerting rules that fire while there’s still time to do something about them.\nI’m an engineer, not a lawyer. Every legal claim below traces to the Cyberbeveiligingswet, the Cyberbeveiligingsbesluit or Directive (EU) 2022/2555, all read on 2026-09-20. Articles get renumbered and ministers publish sector rules. Check the source before you bet a fine on my summary.\nThe five-minute scope test Skip the consultancy flowcharts. Three questions settle it for most organisations.\nAre you a public body? Cbw article 8(1)(h) makes provinces, municipalities, water authorities and qualifying joint arrangements, the gemeenschappelijke regelingen, essential entities by operation of law. No size test, no risk assessment. A village of 9,000 people sits in the same tier as a national telco. That’s a Dutch choice, and it’s the one that catches people out. NIS2 article 2(5)(a) leaves local government optional. The Netherlands took the option and then went further. Ministries and central government zbo’s sit one point higher, in 8(1)(g).\nAre you big enough? For everyone else the act points at Recommendation 2003/361/EC . Under 50 staff and under EUR 10 million on both turnover and balance sheet, you’re out, unless one of the size-independent categories catches you. Medium-sized, 50 to 249 staff or past that EUR 10 million, gets you important-entity status from either annex. Past the medium ceiling, 250 staff or EUR 50 million turnover with a balance sheet total over EUR 43 million, an annex 1 entity becomes essential, while annex 2 stays important however large you get.\nAre you size-independent? Trust service providers, TLD name registries, DNS service providers and providers of public electronic communications networks and services are in regardless of headcount. So are critical entities designated under the Wwke .\nSome organisations are carved out entirely. Defence, the intelligence and security services, the Public Prosecution Service, the police and the safety regions, plus the judiciary, both chambers of parliament and De Nederlandsche Bank. They keep a right to NCSC assistance under article 96.\nThe tier matters more than it looks. RDI supervises essential entities proactively, which means they can ask for information or turn up. For important entities, supervision mostly happens after an incident, and Cbw article 81 limits the enforcement instruments to cases where the authority already has evidence or indications of a breach. So if you’re essential, the supervisor doesn’t need an incident to ask for your policies. They can just ask.\nThe ten measures, mapped Article 21(3) lists ten measures. Cbb articles 6 to 18 turn each one into a concrete duty, so the decree is what you read when you want to know what “incident handling” means on paper. Here’s how they land on a cluster.\nMeasure Cbb What runs in my cluster Where the evidence lives a. risk analysis and security policy 6 , 7 The policy set in Git, with a risk register that names the systems it covers Repo history, ADRs b. incident handling, including logging 8 Falco for runtime detection, Kubernetes audit logs in Loki, Alertmanager routing Loki retention, alert history c. business continuity and crisis management 9 Scheduled Velero backup jobs, plus restore drills into a scratch cluster with a written outcome for each run The drill reports, with a date, a duration and what failed d. supply chain security 10 A registry I run, mirroring what we depend on, with a Kyverno allowlist in front of it, image signing and an SBOM per release Policy reports, cosign verify output, the mirror itself e. secure acquisition, development, maintenance 11 Trivy in CI, a patch window that exists, vulnerability disclosure contact in security.txt Pipeline artefacts f. assessing measure effectiveness 18 Scheduled evaluation with a written record. See the bottom of this post, there is a trap here The written record itself g. cyber hygiene and training 12 Mostly not technical. MFA rollout and onboarding are the parts that are Training records, IdP config h. cryptography policy 13 cert-manager for TLS, a written statement of which algorithms and key lengths are allowed The policy document, plus TLS scan output i. personnel, access control, asset management 14 , 15 , 16 RBAC without cluster-admin for humans, SSO, and an inventory that is generated rather than maintained RoleBindings in Git, inventory export j. MFA and secure communication none IdP enforcement, WebAuthn where possible IdP policy Two things stood out to me. Cbb article 17 adds a duty nothing in article 21(3) hints at. You have to act on targeted alerts, security advisories and threat intelligence. If you already read NCSC advisories and patch on them, you satisfy it, but only if you can show what you did. And Cbb article 4 pulls the digital entity types out of articles 6 to 18 entirely, because Implementing Regulation (EU) 2024/2690 already covers them. Cloud providers, data centres, CDNs, managed service and managed security providers read that regulation instead.\nIf you’re in the government sector, the regulation goes further and names the framework: NEN-EN-ISO/IEC 27001:2023 for the management system, and the controls of NEN-EN-ISO/IEC 27002:2022 minus 5.32 and 5.34, plus the relevant BIO2 measures. Equivalent frameworks are allowed if they reach the same level.\nThe smallest policy that proves a measure Measure d is where a cluster can do something a policy document can’t. Supply chain security starts with knowing where every image comes from, and an admission controller turns that from an intention into a property of the cluster.\nI use Kyverno for this, because the policy is a Kubernetes resource and lives in Git next to everything else. The smallest useful version is a registry allowlist:\napiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: registry-allowlist annotations: policies.kyverno.io/description: \u003e- Cbw art. 21(3)(d), supply chain security. Also the readiness condition for art. 21a: a supplier ban is only survivable if every image origin is known. spec: background: true rules: - name: only-approved-registries match: any: - resources: kinds: - Pod validate: failureAction: Enforce message: \u003e- Images come from registry.example.nl only (Cbw art. 21(3)(d)). pattern: spec: =(ephemeralContainers): - image: \"registry.example.nl/*\" =(initContainers): - image: \"registry.example.nl/*\" containers: - image: \"registry.example.nl/*\" Kyverno 1.13 moved failureAction onto the rule. On 1.12 and older it’s spec.validationFailureAction, which is what the older policies in my repo still use.\nThe command that proves it blocks:\n$ kubectl run test --image=nginx:1.27 Error from server: admission webhook \"validate.kyverno.svc-fail\" denied the request: resource Pod/default/test was blocked due to the following policies registry-allowlist: only-approved-registries: 'validation error: Images come from registry.example.nl only (Cbw art. 21(3)(d)). rule only-approved-registries failed at path /spec/containers/0/image/' That output is the artefact. A supervisor asking how I control the supply chain gets that policy and that denial, with a timestamp on it. I’d rather hand over that than a paragraph in a Word file.\nRun it in Audit first. background: true makes Kyverno scan what’s already running, and kubectl get policyreport -A then tells you how much of your cluster would’ve been refused. On my own cluster the first run turned up seven pods pulling from Docker Hub. I’d have sworn every one of them was mirrored.\nOne thing the policy assumes and doesn’t provide. registry.example.nl has to exist. An allowlist naming a registry nobody runs blocks every workload and calls it compliance, so measure d starts with the artifact repository. The admission rule comes second.\nMine is zot, an OCI-native registry that ships as a single binary and syncs upstream images on a schedule. Two reasons it fits a post about the duty of care. Its CVE search is Trivy underneath, so measure e gets answered by the same component that stores the artefact, instead of a second tool with its own database to keep current. And a single binary with no database beside it is a supply chain I can read end to end, which is the whole reason I run it. Holding the bits yourself buys you three more things. An image keeps existing after an upstream tag gets retagged or deleted, which is the kind of failure that takes a cluster down on a Sunday. The replacement period in article 21a becomes something you can actually act on, because swapping a supplier means rebuilding from artefacts already on your own disk. And Cbb article 10 , which wants a supply chain policy and vetting of direct suppliers, gets a boundary it can point at, since the list of direct suppliers is the list of upstreams the mirror pulls from.\nThat same policy is your answer to Cbw article 21a , which has no counterpart in the directive at all. It lets the responsible minister order an entity to stop using a named party’s products in named parts of its systems, with a replacement period set in the interest of continuity. Will that power ever get used? I have no idea. The readiness question underneath it is the useful part. If a supplier were named tomorrow, could you list every place their code runs? With an allowlist and an image inventory, that’s minutes. With a spreadsheet, weeks.\nThe reporting clock, as alerting rules Every other write-up states these deadlines as prose, and nothing in your monitoring reads prose. The deadlines first, then the rules.\nStep Deadline Measured from Vroegtijdige waarschuwing (early warning) 24 hours Becoming aware of the significant incident Melding (notification) 72 hours Becoming aware of the significant incident Tussentijds verslag on request A request from the CSIRT or authority Eindverslag (final report) one month The 72-hour notification, not the early warning Voortgangsverslag when the final report was due The incident still running at that moment Two traps in that table. The final report clock runs from the notification, so a slow 72-hour filing moves your month. And the Dutch text doesn’t say “within 24 hours”. It says “onverwijld of, indien dat niet mogelijk is, binnen 24 uur”, so immediately, and only if that’s impossible do you get the 24 hours. The outer limit matches the directive, and the duty to move immediately sits on top of it.\nAn incident is significant when it causes or can cause a serious operational disruption or financial loss, or when it has affected or can affect other entities through considerable damage. Sector ministers can add criteria, and a decision addressed to one specific entity is confidential inside that entity, so look up your own sector and skip the generic lists.\nEvery deadline in that table is a duration measured from one moment, the moment you became aware. Nothing in your infrastructure knows when that was, so nothing can tell you the 24 hours are nearly up. The fix is to turn that moment into a number the monitoring can read. Declaring an incident pushes one gauge to Pushgateway, holding the Unix timestamp of the declaration:\ncat \u003c\u003c'EOF' | curl --data-binary @- \\ https://pushgateway.example.nl/metrics/job/nis2/incident/INC-2026-014 # TYPE nis2_incident_declared_timestamp_seconds gauge nis2_incident_declared_timestamp_seconds 1758394800 EOF Pushgateway and not a scrape, because there’s nothing to scrape. A declaration happens once, at one moment, and a scrape target has to still be there when Prometheus comes looking. No process sits in a corner repeating “we’re in an incident”, and writing one would be a worse job than this curl. Pushgateway holds the value until something overwrites it or deletes it.\nThe grouping key in that URL, incident/INC-2026-014, arrives as a label on the metric. That label is what later lets a rule line up the declaration of one incident with the reports filed for that same incident, which is how an alert knows it’s been answered.\nFrom there the alerting is arithmetic on one number. time() is now, the gauge is then, and the difference is how long this incident has been running. Each rule fires ahead of its own deadline:\ngroups: - name: nis2-reporting rules: - alert: NIS2EarlyWarningDue expr: | (time() - nis2_incident_declared_timestamp_seconds \u003e 20 * 3600) unless on (incident) nis2_report_filed_timestamp_seconds{step=\"early-warning\"} labels: severity: page runbook: nis2-reporting annotations: summary: \u003e- Early warning for {{ $labels.incident }} is due within 4 hours (Cbw art. 26(1)). It goes to the CSIRT and the supervisor. - alert: NIS2NotificationDue expr: | (time() - nis2_incident_declared_timestamp_seconds \u003e 68 * 3600) unless on (incident) nis2_report_filed_timestamp_seconds{step=\"notification\"} labels: severity: page runbook: nis2-reporting - alert: NIS2FinalReportDue expr: | (time() - nis2_report_filed_timestamp_seconds{step=\"notification\"} \u003e 26 * 86400) unless on (incident) nis2_report_filed_timestamp_seconds{step=\"final\"} labels: severity: ticket runbook: nis2-reporting Four hours of margin on a 24-hour duty, four on 72, four days on the month. Filing a report clears its own alert, because the gauge for that step appears and the unless stops matching. If you want the reasoning behind alerting on a deadline and not on a metric, it’s the same argument I made in alerting that works. An alert exists to make a human do one specific thing, and this one names the thing in its summary.\nOne dependency I want to state before the runbook, because it decides whether any of this works. A deadline alert is only worth writing if a page still moves someone. In an organisation where alert fatigue has set in, where everyone’s learned that most of what fires is noise to silence, NIS2EarlyWarningDue joins the queue nobody opens. You’ve then built a legal obligation on top of a channel that stopped working, and you won’t see that until the night it costs you the deadline.\nIf that describes where you work, the alerting is the first job and these three rules are the second. Cut the volume until everything that fires means somebody has to do something now. That’s the whole argument of alerting that works, and here it’s a precondition.\nThe runbook, because one incident makes two notifications NIS2 lets a member state route reports to the CSIRT or the competent authority. Cbw articles 26(1) and 27(1) say CSIRT and competent authority. One submission on MijnNCSC covers both in practice because it gets forwarded, but the legal duty runs to two parties, and which two depends on your sector. A municipality reports to NCSC and RDI. A water authority reports to CERT-WM , housed at Het Waterschapshuis, and is supervised by ILT . Healthcare reports to Z-CERT and is supervised by IGJ . Find your own row now, before you need it.\nThe early warning also asks for more than the directive does. NIS2 wants two facts, whether the incident is suspected to be malicious and whether it may have cross-border consequences. Cbb article 24 adds four more, each qualified by “where possible”. The suspected start time, a description of the nature and the consequences noticeable so far, an estimated recovery time, and the measures taken or planned. Assembling seven data points inside 24 hours while the thing is still on fire is a different job from assembling two, and that’s why this lives in a runbook.\n# Runbook: nis2-reporting ## t+0 Declare - Assign INC-YYYY-NNN. Push the declared gauge (see alert annotations). - Decide significance: serious operational disruption, financial loss, or considerable damage to other entities. Record the reasoning as well as the verdict. ## t+0 to t+24h Early warning Seven fields, three from Cbw art. 26(2) and four from Cbb art. 24: 1. Suspected to be unlawful or malicious? (yes / no / unknown) 2. Possible cross-border consequences? (yes / no / unknown) 3. Contact details of the responsible reporter 4. Suspected start time of the incident 5. Nature and consequences noticeable so far 6. Estimated recovery time 7. Measures taken or intended Submit on MijnNCSC. Push the early-warning gauge. ## t+0 to t+72h Notification - Update the early warning where it has changed - Initial assessment: severity and impact - Indicators of compromise, where available - Whatever lets the CSIRT judge cross-border impact Push the notification gauge. The one-month clock starts here. ## Without delay Service recipients Inform them if the incident may adversely affect the service, and inform them of a significant cyber threat where they may be affected (Cbw art. 30). ## t+1 month Final report Detailed description, severity, consequences, probable root cause, mitigations applied and ongoing, cross-border consequences. Still running at that moment? File a progress report instead, then the final report within one month of the incident being handled. Your evidence is a retention setting Cbb article 8 folds logging into incident handling. That sounds mild until you’re writing the final report a month later and find your audit logs rolled over.\nThe Kubernetes audit policy I run keeps the expensive level only where it answers a question I’ll actually get asked:\napiVersion: audit.k8s.io/v1 kind: Policy rules: - level: RequestResponse resources: - group: \"\" resources: [\"secrets\", \"serviceaccounts/token\"] - group: \"rbac.authorization.k8s.io\" resources: [\"*\"] - level: Metadata omitStages: [\"RequestReceived\"] Who read a secret, and who changed an authorisation, at full fidelity. Everything else at metadata level. Ship it to Loki, and set the retention past the reporting horizon. Your final report can land a month after the notification, a progress report can push it further, and a supervisor can ask afterwards, so 30 days of retention runs out before the final report is even due. I keep 400 days for the audit stream and 30 for application logs.\nThe measures with no YAML behind them Not everything maps to a control, and pretending otherwise is where compliance theatre starts. These four don’t.\nThe board training certificate. Cbw article 24 goes well past NIS2 article 20 . Every board member needs the knowledge to identify risks, assess measures and judge their consequences, has to keep it demonstrably current, and has to hold a certificate naming them, the training dates, the subjects and the provider, in Dutch or English. Sitting members have until 15 August 2028, two years from entry into force, and a member appointed later gets two years from appointment. A board member who doesn’t comply can be fined EUR 25,000 personally under article 93 . Nothing in the directive asks for a certificate, a deadline or a personal fine. I read that article twice to be sure. For public bodies “the board” means the college van burgemeester en wethouders, gedeputeerde staten, or the dagelijks bestuur.\nRegistration, which is closer to your job than it looks. Cbw article 44 wants your name, addresses, contact details, IP ranges, sectors, the member states you serve, and your domain names, with changes reported within two weeks. Those are infrastructure facts. A merge request that adds a prefix to your IPAM or a domain to your DNS zone is a registration event. Treat it as one and the two-week duty is something you meet by accident. Don’t, and it’s something you find out you’ve been breaching for a year. Registration has been mandatory since 15 August 2026 and there’s no grace period.\nPeriodic evaluation, and the gap underneath it. NIS2 article 21(4) requires an entity that finds it doesn’t comply to take corrective measures without undue delay. The Dutch article 21 has no such paragraph. The nearest thing is Cbb article 18 , a periodic evaluation of effectiveness, written down, with adjustment where needed. That’s weaker, and the gap is yours to fill, because a drift you find on Tuesday and fix at the next quarterly review is legally defensible and operationally embarrassing. This is the one place I’d over-implement the law rather than meet it. If your infrastructure is declarative, the evaluation can lean on what the repo already proves, which is the argument in declarative infrastructure as compliance evidence.\nSupplier vetting. Cbb article 10 wants a supply chain policy and vetting of direct suppliers. The registry allowlist proves what runs. It says nothing about whether the vendor behind it patches.\nWhat the law actually asks of you Reading the Cyberbeveiligingswet next to the directive was more interesting than I expected, because the Dutch additions aren’t bureaucratic noise. They point at the same thing I keep landing on from the other direction. Every municipality is essential regardless of size, because the damage when it fails doesn’t scale with its headcount. Board members carry a personal fine, because a risk nobody in the room understands is a risk nobody manages. And a minister can ban a named supplier from named systems, which is a state saying out loud that depending on a vendor you can’t inspect is a national risk.\nThat last one is the argument I’ve been making about my own infrastructure for years, and now it’s arriving by law. If you can’t list what you run and where it came from, a supplier ban turns into an archaeology project. I run things I understand for my own reasons. An audit is just the first time somebody else asks me to prove it. The longer version of that is in sovereign infrastructure.\nStart with the mapping table. Fill the rows you can this week, and write down which ones stay empty, because a gap you’ve named is a register entry and a gap you haven’t is what the supervisor finds. Mine had row d empty until that first Audit run counted seven pods from Docker Hub, and seven is a number I could write down.","permalink":"https://tommeurs.nl/en/posts/nis2-for-platform-teams/","title":"NIS2 for Platform Teams: Which Control Satisfies Which Article"},{"content":"My neighbour is 70 years old. He’s used Windows his whole life. Last month he asked me: “Tom, I keep hearing about Linux. Is that something for me too?”\nHe’s not the first. Over the past year I’ve had the same question from friends, family and colleagues, each in their own shape. “I want to get off Windows, but where do I start?” “Is Linux hard?” “Can I still use my normal programs?” To me these are easy questions. I’ve been running Linux since SuSE 6.0, back when you picked your hardware by whether a Linux driver existed for it. If you’ve never seen a terminal, they’re not easy at all.\nAnd I get it. Search for “Linux” and you’re buried under distro comparisons, desktop environment flamewars, and terminal commands that look like someone fell asleep on their keyboard. The Linux community has a reputation problem. We’re very good at deep technical content. We’re terrible at saying “come on in, the water’s fine.”\nPicture this instead. My neighbour boots a USB stick, clicks through an installer that looks like every other installer he’s ever used, and fifteen minutes later he has a working desktop. No driver hunting. No config files. His browser, his email, his photos. A computer that does what he tells it and nothing he didn’t ask for. That’s Linux Mint today, on most laptops. He didn’t know that, and nobody had told him.\nThis blog has always pointed at platform engineering, Kubernetes, and self-hosted infrastructure. Content for people who already live in the terminal. But the philosophy underneath everything I write, sovereignty over convenience, understanding over blind trust, applies to anyone who wants to understand their own computer. Not only the people running production clusters.\nSo this blog is changing. The Kubernetes posts stay. I’m making room for people who are on their first install.\nWhat changed this year Three things came together.\nFirst, the questions kept coming. From people I know, not from strangers on a forum. My neighbour. A colleague who’s fed up with Windows telemetry. A friend who wants to self-host her photos and doesn’t know where to begin. They can learn this. What they’re missing is a starting point that assumes nothing.\nSecond, I looked at this blog and noticed something. There’s already a pile of posts that work fine for beginners. GPG explained, Pass as a password manager, Learn the Defaults, terminal multiplexing. None of these need a computer science degree. But they sat between the Kubernetes deep dives and the eBPF networking posts, and nothing told a newcomer which ones were theirs.\nThat costs more than it looks. A beginner lands on the blog, sees titles full of “etcd” and “Cilium ,” and quietly decides this place isn’t for them. The GPG guide is two clicks away. They never take the clicks. Tab closed, and Windows is still there. I’d written the right posts and filed them where nobody new would ever look.\nThird, and this one weighs the most. Linux is ready for everyone now. Hand-editing xorg.conf and compiling your own wifi drivers, that’s over. Fedora, Linux Mint and openSUSE work out of the box. The install stopped being the hard part years ago. Finding out where to start is the hard part, and when people go looking, they drown.\nTwo ways to find your level So I refiled the whole blog. All 66 posts now say how much you need to know before you start, and the ones that are specifically desktop or server work say which of the two they are.\nDifficulty has its own taxonomy Difficulty started as three tags sitting in the same cloud as Networking and Secrets, competing with them for the same click. It has its own taxonomy now, level, and every post carries exactly one of the three terms. The term measures what you already need to know, so a Beginner post about an advanced subject stays Beginner.\nLevel What it assumes Posts Examples Beginner A terminal and nothing else 18 GPG guide, Learn the Defaults, Terminal Multiplexing Intermediate You have run the thing in question at least once 24 ArgoCD for Beginners, K3s Cluster Setup Advanced You run it in production and want the parts that bite 24 etcd Deep Dive, Chaos Engineering Context stayed a tag Where you run Linux is a subject like any other, so it stays in the tag cloud with the rest of them.\nTag What you’re doing Posts Examples Desktop Using Linux as your daily driver 11 Dotfile Management, Terminal Multiplexing Server Running services, homelab, self-hosting 8 K3s Cluster Setup, Backup Strategy There was a third context tag, Platform, and I deleted it. It had landed on 59% of the posts. Click it and you got most of the blog back, which tells you nothing. If that scale is what you came for, the honest filter is Advanced, or the category for the subject you actually want.\nStart at Beginner if you’re new, and move up when the introductions start boring you.\nThe path is the same, the depth is different Here’s what 25+ years of Linux taught me about that ladder. There’s no jump in it.\nYou install Linux on a laptop you had lying around. You learn the terminal basics. You start poking at your dotfiles. You stand up a small server. You put something in a container. One day you’re running Kubernetes and you can’t quite point to the moment it stopped being scary. Each step rests on the one before it.\nThe values that make a good platform engineer are the same ones that make switching to Linux worth it in the first place. You want to know what your computer is doing. You refuse to let “it just works” quietly mean “you have no idea what it’s doing behind your back.” You’d rather run a system you can open up.\nThat’s why sovereignty is more than a fancy label for “I run my own servers.” It starts with an operating system that respects you, where you can open every setting and see every process. My neighbour installing Mint and an engineer wiring up multi-cluster networking are making the same choice. One of them just has more YAML.\nWhat’s coming I’ll be writing more for people at the start. Not “install Ubuntu in 5 minutes” content, there’s plenty of that already. I want to write about the things that actually trip people up:\nHow to think about choosing a distribution (and why it matters less than you think) Why the terminal is more efficient than it looks once it stops being scary Managing your files, photos and documents without cloud services Privacy and security basics that everyone should know Setting up your first home server These sit alongside the existing platform engineering content, and the level terms will point you at the ones written for you.\nThe same philosophy, for everyone So why hasn’t everyone switched already? Because the obstacles are real. People have software that only ships for Windows. They’ve built fifteen years of muscle memory into one operating system. They’ve been told, often by us, that Linux means a weekend of fighting their graphics card. We earned that reputation ourselves. It was true when I started. It’s still doing damage.\nMy neighbour doesn’t need to become a sysadmin, though. He needs a working desktop and someone to tell him his questions aren’t stupid. The engineer and the retiree want the same thing underneath. They want to know what their machine is doing, and they don’t want to ask a company’s permission to find out.\nI started on Linux when you bought a specific graphics card and network adapter just to get a working system. My neighbour can boot a USB stick into a full desktop in fifteen minutes. What’s left is that people don’t know this exists, or assume it isn’t for them.\nPick a level, pick a tag, and start where you are. And if you’ve got a question you think is too basic, send it anyway. My neighbour’s question is why this post exists.","permalink":"https://tommeurs.nl/en/posts/linux-is-for-everyone/","title":"Linux Is for Everyone: Why This Blog Now Welcomes Beginners Too"},{"content":"Here’s how privacy works today. You read the policy. You dismiss the cookie banner. You hunt through three settings menus for the opt-out toggle that someone buried on purpose. All of that to exercise a right that should’ve been the default. And while you’re still on the first paragraph of the terms of service, every click, scroll, hover and the exact timing of your keystrokes has already been harvested, packaged and sold. Somewhere along the way I stopped finding that strange, and that bothers me more than the harvesting does.\nThe defensive playbook is the standard answer. Use a VPN. Block trackers. Stay off social media. Self-host what you can. I do most of this, and I’d recommend it. But there’s a structural flaw in it that nags at me every time I think about it.\nDefence means you have to win every round You can only block what you know about. The widget embedded three layers deep in a page you visited once. The analytics SDK bundled into an app you forgot you installed. The tracking domain that spun up this morning. None of that is on your blocklist yet. The collector only has to slip through once to start building a profile. You have to catch it every time, on every device, forever.\nNobody plays that perfectly. I certainly don’t. Ad blockers work until a site detects them and degrades the page. A VPN hides your IP until fingerprinting routes around it. DNS blocking catches known trackers until tomorrow’s domains show up. On one side sits an industry with billions of dollars of incentive. On the other side, you and a browser extension.\nSo I keep coming back to a different question. What if you stopped trying to win every round?\nLet them collect garbage Imagine you let the collection happen. The tracker fires, the SDK phones home, the profile gets built. But the profile is garbage. The model trained on your data learns the wrong things. The advertiser bidding on your attention is bidding on noise.\nThat’s the shift. You’re going to lose the fight to prevent collection eventually, so you stop fighting it and make the collected data unreliable instead. The system still runs. It just captures nothing worth having.\nThink of giving a fake name at a coffee shop. The barista still makes your coffee. The loyalty system still logs a visit. But the data point they wrote down points at a person who doesn’t exist. Scale that up and you get data poisoning. You deliberately feed misleading, noisy or corrupted information into the systems that profile you, until the quality drops below the point where anyone can use it.\nOld idea. Resistance groups in occupied territory filed false reports and seeded deliberate misinformation to degrade what their occupiers knew. When you can’t stop the surveillance, you can still poison what it learns.\nThe tools already exist This is the part that surprised me when I first dug in. None of this is a research curiosity. The tooling ships today, and a lot of it runs in your browser.\nAdNauseam clicks every ad it sees, quietly, in the background. To the ad networks you suddenly look interested in baby clothes, retirement homes, skateboards and farming equipment, all at once. Your profile gets so noisy that targeting you becomes pointless. The network still collects its per-click fee, but the advertiser pays for nothing. Google pulled AdNauseam from the Chrome Web Store, which I take as a decent sign that it does what it claims.\nTrackMeNot does the same trick to your search history. It fires off random queries in the background, so your real searches sit buried in a pile of fake ones. Same principle as AdNauseam. If you can’t hide the signal, drown it in noise.\nPoisoning AI training data This is where it gets interesting for me, because it lines up with the moment we’re in.\nLarge language models are trained on enormous datasets scraped off the open internet. Your blog posts, your forum replies, your Stack Overflow answers, the things you wrote about yourself. All of it feeds the pipeline. Nobody asked you. Nobody paid you. And there’s no meaningful way to pull your contribution back out after the fact.\nNightshade, built at the University of Chicago, adds perturbations to images that your eye can’t see. The picture looks identical to you. To a training pipeline it reads as something else entirely. A dog parses as a cat, a car as a cow. Get enough poisoned images into the dataset and the model’s ability to generate certain content degrades.\nGlaze, from the same group, targets artists specifically. It wraps a style cloak around artwork so models can’t learn and replicate an artist’s style, while the art still looks unchanged to humans.\nText is the one that gets me, because text is what I put online. If you know your writing is being scraped for training, you can embed adversarial patterns into it. Subtly contradictory claims, carefully built edge cases, statements that mislead in context. All of it can drag down model quality in a specific domain. Some sites have started planting invisible text aimed straight at scrapers, a robots.txt with actual teeth. Researchers have shown that poisoning as little as 0.1% of a training set can measurably hurt performance on targeted tasks. That number is the whole argument. Poisoning a dataset costs you almost nothing, and cleaning it costs them a lot.\nYes, this is sabotage. Read the context. Some of you are already uncomfortable, and I get it. This looks like vandalism. So let me be honest about the context rather than dodge it.\nThese companies took your data without asking. They built products worth billions on your creative output, your technical answers, your personal writing, and handed back nothing except “you agreed to our terms.” There’s no honest way to leave that arrangement, so I’ve stopped calling obfuscation an attack and started calling it self-defence.\nThis is the sovereignty principle applied to data. As I argued in Sovereign Infrastructure, depending on systems you don’t control and can’t inspect is a vulnerability. I self-host for that reason, not because it’s the easy path. The same logic carries over here. If I can’t control whether my data gets collected, I can at least control what it’s worth once it’s been taken from me without real consent.\nWhat this looks like in practice You can start today. None of this needs a lab.\nBrowser noise:\nInstall AdNauseam to poison your ad profile Run separate browsers with different personas for different activities Clear all cookies now and then to break tracking continuity Search pollution:\nUse TrackMeNot to generate background queries Spread your searches across engines so no single company holds the full picture AI training resistance:\nRun any visual art through Glaze before you post it Think about what signals your scraped content sends to a model Use your robots.txt, while remembering it’s a request and not enforcement Infrastructure level:\nKeep a Pi-hole or AdGuard Home for DNS-level blocking as your baseline Hand out per-service email aliases, which a catch-all domain makes trivial Look at tools like Chaff that generate fake browsing traffic The obstacles are real I’d be selling you something if I skipped the downsides, so here they are.\nResource cost. AdNauseam burns bandwidth clicking ads. TrackMeNot generates traffic. You’re spending your own resources to manufacture noise.\nCollateral damage. Small publishers who live on ad revenue catch phantom clicks. Those clicks cost advertisers money, which eventually tightens the whole ad market. You might hit people you never meant to hit.\nArms race. Collectors adapt. Google already blocked AdNauseam from the Chrome Web Store. Expect smarter fingerprinting and better poisoning detection over time.\nLegal grey areas. Depending on where you live, some of these techniques may brush up against terms of service or computer fraud law. Know your jurisdiction before you act.\nEthical ambiguity. Poisoning LLM training data hits everyone who uses those models, not only the firm that scraped you. That’s a genuine collective action problem, and I don’t have a clean answer for it. It’s also the reason I still hesitate on the text side.\nThese costs are real. Whether they’re worth paying depends on how much more you value privacy than convenience, and how much collateral damage you’re willing to own.\nClosing the gap Data poisoning isn’t a magic fix, and I wouldn’t trust anyone who sold it that way. It belongs inside a wider strategy that still includes blocking, self-hosting, encryption and pushing for stronger privacy law through actual politics.\nWhat it adds is the thing pure defence can’t give you. Degrading data is cheap for you and cleaning it is expensive for them, and for once you’re on the cheap side. That makes mass surveillance a little less profitable, and it seeds doubt into systems that only work when the data is clean. You don’t have to migrate your whole life to feel it. Pick one tool, run it for a month, and watch what changes.\nHiding was never the only answer to surveillance. The other one is to make the surveillance unreliable, without breaking anything, by feeding it data too noisy to use. I still do the defensive part. I’ve just stopped counting every tracker that slips past it as a loss.","permalink":"https://tommeurs.nl/en/posts/data-poisoning-privacy/","title":"Data Poisoning: Reclaiming Your Privacy Through Offence"},{"content":"I run VMs all day. A Kubernetes deployment I want to test, a distro I read about, Windows for that one stubborn application, an experiment I’d rather not run on my actual machine. KVM/QEMU is the right tool for all of it. It lives in the Linux kernel and runs close to native speed. Nothing in it phones home, and no vendor gets a say in what I do with my own hardware.\nThe commands, though.\nvirt-install --name test --ram 2048 --vcpus 2 --disk path=/var/lib/libvirt/images/test.qcow2,size=20 --cdrom /home/user/Downloads/ubuntu-22.04.iso --os-variant ubuntu22.04 --network bridge=virbr0 --graphics spice --console pty,target_type=serial Nobody types that from memory. You copy it off a wiki, swap a few values, get the disk path wrong, and try again. By the time the VM boots, whatever made you want to test something has cooled off. I’ve lost count of the experiments I skipped because getting a VM up took longer than I was willing to wait.\nSo at some point I stopped fighting it and wrote a small pile of scripts and aliases. A VM from an ISO with defaults I don’t have to think about, and a snapshot that takes one word. The bar was low. If I can type it faster than I can click through virt-manager, I’ll use it. Everything below clears that bar.\nScripts instead of the GUI I lean on this kind of tooling for reasons I went into in Working with an AuDHD Brain. A GUI hides everything behind a layout I have to reload into my head each time. Where’s the snapshot button again? Which tab had the disk settings? That lookup costs me focus I’d rather spend on the problem I opened the VM for.\nA script doesn’t ask me to remember anything. It runs the same way every time, and the file is its own documentation. Tab completion beats any mouse, and I can pipe one script into the next without thinking about it.\nAnd the boring win. Scripts work over SSH. Most of my VMs live on a headless box in the corner, and virt-manager isn’t much use there.\nStart with the aliases The aliases are the cheapest win, so start there. Drop these in your .bashrc or .zshrc:\n# Quick status alias vms='virsh list --all' alias vmsr='virsh list' # Running only alias vmnets='virsh net-list --all' alias vmpools='virsh pool-list --all' # Power management alias vmstart='virsh start' alias vmstop='virsh shutdown' alias vmkill='virsh destroy' # Force stop alias vmreboot='virsh reboot' # Console access alias vmconsole='virsh console' alias vmviewer='virt-viewer' # Quick info alias vminfo='virsh dominfo' alias vmip='virsh domifaddr' alias vmblk='virsh domblklist' # Snapshots alias vmsnap='virsh snapshot-create-as' alias vmsnaps='virsh snapshot-list' alias vmrevert='virsh snapshot-revert' alias vmsnapdel='virsh snapshot-delete' # Edit alias vmedit='virsh edit' Now virsh list --all is vms. On its own that’s nothing. But I run a handful of these dozens of times a day, and the keystrokes matter less than the “what was that flag again” lookups I no longer do. It changed how often I bother with a VM at all.\nThe workhorse: vmquick Aliases shorten things you already know. Creating a VM is where the real time went, so this is the script I use most. A VM from an ISO with as little typing as I could get away with:\n#!/bin/bash # vmquick - Quick VM creation with sane defaults # Usage: vmquick \u003cname\u003e \u003ciso-path\u003e [ram-gb] [cpus] [disk-gb] set -euo pipefail # Defaults (easily upgradeable later) DEFAULT_RAM=2 # GB DEFAULT_CPUS=2 DEFAULT_DISK=20 # GB VM_PATH=\"/var/lib/libvirt/images\" NETWORK=\"default\" # Or your bridge name # Colors for output RED='\\033[0;31m' GREEN='\\033[0;32m' YELLOW='\\033[1;33m' NC='\\033[0m' # No Color usage() { echo \"Usage: vmquick \u003cname\u003e \u003ciso-path\u003e [ram-gb] [cpus] [disk-gb]\" echo \"\" echo \"Examples:\" echo \" vmquick ubuntu ubuntu-22.04.iso # 2GB RAM, 2 CPUs, 20GB disk\" echo \" vmquick fedora fedora-39.iso 4 # 4GB RAM\" echo \" vmquick arch archlinux.iso 4 4 50 # 4GB RAM, 4 CPUs, 50GB disk\" echo \"\" echo \"Defaults: ${DEFAULT_RAM}GB RAM, ${DEFAULT_CPUS} CPUs, ${DEFAULT_DISK}GB disk\" exit 1 } # Check arguments [[ $# -lt 2 ]] \u0026\u0026 usage NAME=\"$1\" ISO=\"$2\" RAM=\"${3:-$DEFAULT_RAM}\" CPUS=\"${4:-$DEFAULT_CPUS}\" DISK=\"${5:-$DEFAULT_DISK}\" # Convert RAM to MB for virt-install RAM_MB=$((RAM * 1024)) # Validate ISO exists if [[ ! -f \"$ISO\" ]]; then # Check common download locations for dir in ~/Downloads ~/ISOs /tmp; do if [[ -f \"$dir/$ISO\" ]]; then ISO=\"$dir/$ISO\" break fi done fi if [[ ! -f \"$ISO\" ]]; then echo -e \"${RED}Error: ISO not found: $ISO${NC}\" echo \"Checked: $ISO, ~/Downloads/$ISO, ~/ISOs/$ISO, /tmp/$ISO\" exit 1 fi # Check if VM already exists if virsh dominfo \"$NAME\" \u0026\u003e/dev/null; then echo -e \"${RED}Error: VM '$NAME' already exists${NC}\" echo \"Use 'vmrm $NAME' to remove it first\" exit 1 fi DISK_PATH=\"${VM_PATH}/${NAME}.qcow2\" echo -e \"${GREEN}Creating VM: $NAME${NC}\" echo \" RAM: ${RAM}GB (${RAM_MB}MB)\" echo \" CPUs: $CPUS\" echo \" Disk: ${DISK}GB at $DISK_PATH\" echo \" ISO: $ISO\" echo \"\" # Detect OS variant (best effort) OS_VARIANT=\"linux2022\" case \"$ISO\" in *ubuntu*22*|*jammy*) OS_VARIANT=\"ubuntu22.04\" ;; *ubuntu*24*|*noble*) OS_VARIANT=\"ubuntu24.04\" ;; *ubuntu*20*|*focal*) OS_VARIANT=\"ubuntu20.04\" ;; *debian*12*|*bookworm*) OS_VARIANT=\"debian12\" ;; *debian*11*|*bullseye*) OS_VARIANT=\"debian11\" ;; *fedora*39*) OS_VARIANT=\"fedora39\" ;; *fedora*40*) OS_VARIANT=\"fedora40\" ;; *arch*) OS_VARIANT=\"archlinux\" ;; *alpine*) OS_VARIANT=\"alpinelinux3.18\" ;; *centos*9*|*stream*9*) OS_VARIANT=\"centos-stream9\" ;; *rocky*9*) OS_VARIANT=\"rocky9\" ;; *alma*9*) OS_VARIANT=\"almalinux9\" ;; *windows*11*) OS_VARIANT=\"win11\" ;; *windows*10*) OS_VARIANT=\"win10\" ;; *talos*) OS_VARIANT=\"linux2022\" ;; esac echo -e \"${YELLOW}Detected OS variant: $OS_VARIANT${NC}\" echo \"\" # Create the VM virt-install \\ --name \"$NAME\" \\ --ram \"$RAM_MB\" \\ --vcpus \"$CPUS\" \\ --disk \"path=$DISK_PATH,size=$DISK,format=qcow2,bus=virtio\" \\ --cdrom \"$ISO\" \\ --os-variant \"$OS_VARIANT\" \\ --network \"network=$NETWORK,model=virtio\" \\ --graphics spice,listen=none \\ --video virtio \\ --channel spicevmc \\ --console pty,target_type=serial \\ --boot cdrom,hd \\ --noautoconsole echo \"\" echo -e \"${GREEN}VM '$NAME' created and started!${NC}\" echo \"\" echo \"Next steps:\" echo \" vmview $NAME # GUI console (virt-viewer)\" echo \" vmcon $NAME # Serial console (if OS supports)\" echo \" vmip $NAME # Get IP address (after OS install)\" echo \"\" echo \"To upgrade later:\" echo \" vmupgrade $NAME 8 # Set RAM to 8GB\" echo \" vmupgrade $NAME 8 4 # Set RAM to 8GB, CPUs to 4\" echo \" vmupgrade $NAME 8 4 50 # Also expand disk to 50GB\" Save it as ~/.local/bin/vmquick, chmod +x it, and you’re done.\nUsage examples # Minimal - just name and ISO vmquick testvm ubuntu-22.04.4-live-server-amd64.iso # With more RAM vmquick k8s-node talos-amd64.iso 8 # Full specs for a beefy workload vmquick gitlab fedora-server-40.iso 16 8 100 It looks for the ISO in the usual places, so if the file sits in ~/Downloads or ~/ISOs you give the filename and skip the full path.\nResizing later: vmupgrade I start every VM small on purpose. 2GB and two cores is plenty to find out whether something works, and if I need more later, bumping it up is one command instead of a reinstall. That’s what this script handles:\n#!/bin/bash # vmupgrade - Upgrade VM resources (requires VM to be stopped) # Usage: vmupgrade \u003cname\u003e [ram-gb] [cpus] [disk-gb] set -euo pipefail RED='\\033[0;31m' GREEN='\\033[0;32m' YELLOW='\\033[1;33m' NC='\\033[0m' usage() { echo \"Usage: vmupgrade \u003cname\u003e [ram-gb] [cpus] [disk-gb]\" echo \"\" echo \"Examples:\" echo \" vmupgrade myvm 8 # Set RAM to 8GB\" echo \" vmupgrade myvm 8 4 # Set RAM to 8GB, CPUs to 4\" echo \" vmupgrade myvm 8 4 50 # Also expand disk to 50GB\" echo \"\" echo \"Note: VM must be stopped. Disk can only be increased, not decreased.\" exit 1 } [[ $# -lt 2 ]] \u0026\u0026 usage NAME=\"$1\" RAM=\"${2:-}\" CPUS=\"${3:-}\" DISK=\"${4:-}\" # Check VM exists if ! virsh dominfo \"$NAME\" \u0026\u003e/dev/null; then echo -e \"${RED}Error: VM '$NAME' not found${NC}\" exit 1 fi # Check VM is stopped STATE=$(virsh domstate \"$NAME\" 2\u003e/dev/null) if [[ \"$STATE\" != \"shut off\" ]]; then echo -e \"${RED}Error: VM must be stopped first${NC}\" echo \"Current state: $STATE\" echo \"Run: vmstop $NAME\" exit 1 fi echo -e \"${GREEN}Upgrading VM: $NAME${NC}\" echo \"\" # Update RAM if [[ -n \"$RAM\" ]]; then RAM_MB=$((RAM * 1024)) echo \"Setting RAM to ${RAM}GB...\" virsh setmaxmem \"$NAME\" \"${RAM_MB}M\" --config virsh setmem \"$NAME\" \"${RAM_MB}M\" --config echo -e \"${GREEN} RAM: ${RAM}GB${NC}\" fi # Update CPUs if [[ -n \"$CPUS\" ]]; then echo \"Setting CPUs to $CPUS...\" virsh setvcpus \"$NAME\" \"$CPUS\" --config --maximum virsh setvcpus \"$NAME\" \"$CPUS\" --config echo -e \"${GREEN} CPUs: $CPUS${NC}\" fi # Expand disk if [[ -n \"$DISK\" ]]; then # Find the disk DISK_PATH=$(virsh domblklist \"$NAME\" | grep -E '\\.qcow2|\\.img' | awk '{print $2}' | head -1) if [[ -z \"$DISK_PATH\" ]]; then echo -e \"${YELLOW}Warning: Could not find disk to resize${NC}\" else CURRENT_SIZE=$(qemu-img info \"$DISK_PATH\" | grep 'virtual size' | grep -oP '\\d+(?= GiB)') if [[ \"$DISK\" -le \"$CURRENT_SIZE\" ]]; then echo -e \"${YELLOW}Warning: New size ($DISK GB) is not larger than current ($CURRENT_SIZE GB)${NC}\" echo \"Disk resize skipped (can only increase)\" else echo \"Expanding disk from ${CURRENT_SIZE}GB to ${DISK}GB...\" qemu-img resize \"$DISK_PATH\" \"${DISK}G\" echo -e \"${GREEN} Disk: ${DISK}GB${NC}\" echo -e \"${YELLOW} Note: You'll need to expand the partition inside the VM${NC}\" fi fi fi echo \"\" echo -e \"${GREEN}Upgrade complete!${NC}\" echo \"Start the VM with: vmstart $NAME\" Snapshots without the ceremony: vmsnap A snapshot is what lets me do reckless things on purpose. Break the VM, revert, try again. The raw virsh snapshot-* commands are fine but verbose, so this wrapper shortens them:\n#!/bin/bash # vmsnap - Easy snapshot management # Usage: vmsnap \u003ccommand\u003e \u003cvm-name\u003e [snapshot-name] set -euo pipefail RED='\\033[0;31m' GREEN='\\033[0;32m' YELLOW='\\033[1;33m' CYAN='\\033[0;36m' NC='\\033[0m' usage() { echo \"Usage: vmsnap \u003ccommand\u003e \u003cvm-name\u003e [snapshot-name]\" echo \"\" echo \"Commands:\" echo \" create \u003cvm\u003e [name] Create snapshot (default name: timestamp)\" echo \" list \u003cvm\u003e List all snapshots\" echo \" revert \u003cvm\u003e [name] Revert to snapshot (default: latest)\" echo \" delete \u003cvm\u003e \u003cname\u003e Delete a snapshot\" echo \" info \u003cvm\u003e \u003cname\u003e Show snapshot details\" echo \"\" echo \"Examples:\" echo \" vmsnap create myvm # Create with timestamp name\" echo \" vmsnap create myvm before-upgrade # Create with custom name\" echo \" vmsnap list myvm # Show all snapshots\" echo \" vmsnap revert myvm # Revert to latest\" echo \" vmsnap revert myvm before-upgrade # Revert to specific snapshot\" exit 1 } [[ $# -lt 2 ]] \u0026\u0026 usage CMD=\"$1\" VM=\"$2\" SNAP_NAME=\"${3:-}\" # Verify VM exists if ! virsh dominfo \"$VM\" \u0026\u003e/dev/null; then echo -e \"${RED}Error: VM '$VM' not found${NC}\" exit 1 fi case \"$CMD\" in create) # Generate name if not provided if [[ -z \"$SNAP_NAME\" ]]; then SNAP_NAME=\"snap-$(date +%Y%m%d-%H%M%S)\" fi echo -e \"${GREEN}Creating snapshot '$SNAP_NAME' for VM '$VM'...${NC}\" virsh snapshot-create-as \"$VM\" \"$SNAP_NAME\" --description \"Created by vmsnap on $(date)\" echo -e \"${GREEN}Snapshot created!${NC}\" ;; list) echo -e \"${CYAN}Snapshots for VM '$VM':${NC}\" echo \"\" virsh snapshot-list \"$VM\" --tree 2\u003e/dev/null || virsh snapshot-list \"$VM\" ;; revert) if [[ -z \"$SNAP_NAME\" ]]; then # Get the latest snapshot SNAP_NAME=$(virsh snapshot-list \"$VM\" --name | tail -1) if [[ -z \"$SNAP_NAME\" ]]; then echo -e \"${RED}Error: No snapshots found for VM '$VM'${NC}\" exit 1 fi echo -e \"${YELLOW}No snapshot specified, using latest: $SNAP_NAME${NC}\" fi echo -e \"${GREEN}Reverting VM '$VM' to snapshot '$SNAP_NAME'...${NC}\" virsh snapshot-revert \"$VM\" \"$SNAP_NAME\" echo -e \"${GREEN}Reverted!${NC}\" ;; delete) if [[ -z \"$SNAP_NAME\" ]]; then echo -e \"${RED}Error: Snapshot name required for delete${NC}\" usage fi echo -e \"${YELLOW}Deleting snapshot '$SNAP_NAME' from VM '$VM'...${NC}\" virsh snapshot-delete \"$VM\" \"$SNAP_NAME\" echo -e \"${GREEN}Deleted!${NC}\" ;; info) if [[ -z \"$SNAP_NAME\" ]]; then echo -e \"${RED}Error: Snapshot name required for info${NC}\" usage fi virsh snapshot-info \"$VM\" \"$SNAP_NAME\" ;; *) echo -e \"${RED}Unknown command: $CMD${NC}\" usage ;; esac How that plays out # Before doing something risky vmsnap create testvm before-experiment # Do risky thing... # Oh no, it broke! # Revert to safety vmsnap revert testvm before-experiment # Or just revert to the latest snapshot vmsnap revert testvm Finding the IP: vmip This one exists because virsh domifaddr lied to me one too many times. It only answers once the guest agent is up, so right after boot, when I want the IP, it tells me nothing. This script tries the guest agent, then the DHCP lease, then the ARP table, and returns whatever answers first:\n#!/bin/bash # vmip - Get VM IP address(es) # Usage: vmip \u003cvm-name\u003e [-w] set -euo pipefail VM=\"${1:-}\" WAIT=\"${2:-}\" if [[ -z \"$VM\" ]]; then echo \"Usage: vmip \u003cvm-name\u003e [-w]\" echo \" -w Wait for IP to become available\" exit 1 fi get_ip() { # Try multiple methods # Method 1: Guest agent (most reliable if installed) IP=$(virsh domifaddr \"$VM\" --source agent 2\u003e/dev/null | grep -oP '\\d+\\.\\d+\\.\\d+\\.\\d+' | head -1) [[ -n \"$IP\" ]] \u0026\u0026 echo \"$IP\" \u0026\u0026 return 0 # Method 2: DHCP lease (works for NAT network) IP=$(virsh domifaddr \"$VM\" --source lease 2\u003e/dev/null | grep -oP '\\d+\\.\\d+\\.\\d+\\.\\d+' | head -1) [[ -n \"$IP\" ]] \u0026\u0026 echo \"$IP\" \u0026\u0026 return 0 # Method 3: ARP table MAC=$(virsh domiflist \"$VM\" 2\u003e/dev/null | grep -oP '([0-9a-f]{2}:){5}[0-9a-f]{2}' | head -1) if [[ -n \"$MAC\" ]]; then IP=$(arp -an | grep -i \"$MAC\" | grep -oP '\\d+\\.\\d+\\.\\d+\\.\\d+' | head -1) [[ -n \"$IP\" ]] \u0026\u0026 echo \"$IP\" \u0026\u0026 return 0 fi return 1 } if [[ \"$WAIT\" == \"-w\" ]]; then echo \"Waiting for IP address for '$VM'...\" \u003e\u00262 for i in {1..60}; do if IP=$(get_ip); then echo \"$IP\" exit 0 fi sleep 2 done echo \"Timeout waiting for IP\" \u003e\u00262 exit 1 else if IP=$(get_ip); then echo \"$IP\" else echo \"No IP found for '$VM'\" \u003e\u00262 echo \"VM might not be running or IP not yet assigned\" \u003e\u00262 exit 1 fi fi Use it like this:\n# Get IP immediately (if available) vmip myvm # Wait for IP (useful right after boot) vmip myvm -w # SSH directly ssh user@$(vmip myvm) Doing things in bulk: vmall Once you’re running a cluster of test VMs, doing things one VM at a time gets old fast. Starting four k8s nodes by hand is the kind of repetitive thing my brain refuses to do reliably, so vmall does it for me:\n#!/bin/bash # vmall - Run operation on all VMs (or filtered set) # Usage: vmall \u003ccommand\u003e [filter] set -euo pipefail CMD=\"${1:-list}\" FILTER=\"${2:-}\" case \"$CMD\" in list|ls) virsh list --all ;; start) for vm in $(virsh list --name --state-shutoff | grep -E \"${FILTER:-.}\"); do echo \"Starting $vm...\" virsh start \"$vm\" done ;; stop|shutdown) for vm in $(virsh list --name --state-running | grep -E \"${FILTER:-.}\"); do echo \"Stopping $vm...\" virsh shutdown \"$vm\" done ;; kill|destroy) for vm in $(virsh list --name --state-running | grep -E \"${FILTER:-.}\"); do echo \"Force stopping $vm...\" virsh destroy \"$vm\" done ;; ips) for vm in $(virsh list --name --state-running | grep -E \"${FILTER:-.}\"); do IP=$(vmip \"$vm\" 2\u003e/dev/null || echo \"N/A\") printf \"%-20s %s\\n\" \"$vm\" \"$IP\" done ;; snap) SNAP_NAME=\"bulk-$(date +%Y%m%d-%H%M%S)\" for vm in $(virsh list --name | grep -E \"${FILTER:-.}\"); do echo \"Snapshotting $vm...\" virsh snapshot-create-as \"$vm\" \"$SNAP_NAME\" 2\u003e/dev/null || echo \" Failed (external disk?)\" done ;; *) echo \"Usage: vmall \u003ccommand\u003e [filter]\" echo \"\" echo \"Commands:\" echo \" list List all VMs\" echo \" start [filter] Start stopped VMs matching filter\" echo \" stop [filter] Gracefully stop running VMs\" echo \" kill [filter] Force stop running VMs\" echo \" ips [filter] Show IPs of running VMs\" echo \" snap [filter] Snapshot all VMs\" echo \"\" echo \"Filter is a regex pattern (e.g., 'k8s' for all k8s-* VMs)\" ;; esac In practice # Start all k8s nodes vmall start k8s # Get IPs for all running VMs vmall ips # Snapshot everything before cluster upgrade vmall snap # Stop all test VMs vmall stop test Skipping the installer entirely: vmcloud For anything that speaks cloud-init, which is most modern distros, sitting through an installer is wasted time. A cloud image is already a working system. You hand it a hostname, your SSH key and a package list, and it boots straight into a usable machine. This script wires that up:\n#!/bin/bash # vmcloud - Create VM from cloud image with cloud-init # Usage: vmcloud \u003cname\u003e \u003ccloud-image\u003e [ram-gb] [cpus] [disk-gb] set -euo pipefail VM_PATH=\"/var/lib/libvirt/images\" CLOUD_INIT_PATH=\"/tmp/cloud-init-$$\" DEFAULT_RAM=2 DEFAULT_CPUS=2 DEFAULT_DISK=20 DEFAULT_USER=\"admin\" DEFAULT_PASSWORD=\"changeme\" # Override with env var NAME=\"${1:-}\" IMAGE=\"${2:-}\" RAM=\"${3:-$DEFAULT_RAM}\" CPUS=\"${4:-$DEFAULT_CPUS}\" DISK=\"${5:-$DEFAULT_DISK}\" SSH_KEY=\"${SSH_KEY:-$(cat ~/.ssh/id_ed25519.pub 2\u003e/dev/null || cat ~/.ssh/id_rsa.pub 2\u003e/dev/null || echo '')}\" PASSWORD=\"${VM_PASSWORD:-$DEFAULT_PASSWORD}\" usage() { echo \"Usage: vmcloud \u003cname\u003e \u003ccloud-image\u003e [ram-gb] [cpus] [disk-gb]\" echo \"\" echo \"Environment variables:\" echo \" SSH_KEY SSH public key to inject (default: ~/.ssh/id_ed25519.pub)\" echo \" VM_PASSWORD Password for default user (default: changeme)\" echo \"\" echo \"Examples:\" echo \" vmcloud myvm ubuntu-22.04-minimal-cloudimg-amd64.img\" echo \" vmcloud myvm debian-12-genericcloud-amd64.qcow2 4 4 50\" exit 1 } [[ $# -lt 2 ]] \u0026\u0026 usage # Create disk from cloud image DISK_PATH=\"${VM_PATH}/${NAME}.qcow2\" if [[ -f \"$DISK_PATH\" ]]; then echo \"Error: Disk already exists: $DISK_PATH\" exit 1 fi echo \"Creating disk from cloud image...\" cp \"$IMAGE\" \"$DISK_PATH\" qemu-img resize \"$DISK_PATH\" \"${DISK}G\" # Create cloud-init config mkdir -p \"$CLOUD_INIT_PATH\" cat \u003e \"$CLOUD_INIT_PATH/meta-data\" \u003c\u003cEOF instance-id: ${NAME} local-hostname: ${NAME} EOF cat \u003e \"$CLOUD_INIT_PATH/user-data\" \u003c\u003cEOF #cloud-config hostname: ${NAME} manage_etc_hosts: true users: - name: ${DEFAULT_USER} sudo: ALL=(ALL) NOPASSWD:ALL shell: /bin/bash lock_passwd: false passwd: $(openssl passwd -6 \"$PASSWORD\") ssh_authorized_keys: - ${SSH_KEY} package_update: true package_upgrade: false packages: - qemu-guest-agent runcmd: - systemctl enable --now qemu-guest-agent EOF # Create cloud-init ISO genisoimage -output \"$CLOUD_INIT_PATH/cloud-init.iso\" \\ -volid cidata -joliet -rock \\ \"$CLOUD_INIT_PATH/user-data\" \\ \"$CLOUD_INIT_PATH/meta-data\" 2\u003e/dev/null CLOUD_INIT_ISO=\"${VM_PATH}/${NAME}-cloud-init.iso\" mv \"$CLOUD_INIT_PATH/cloud-init.iso\" \"$CLOUD_INIT_ISO\" rm -rf \"$CLOUD_INIT_PATH\" # Create VM RAM_MB=$((RAM * 1024)) virt-install \\ --name \"$NAME\" \\ --ram \"$RAM_MB\" \\ --vcpus \"$CPUS\" \\ --disk \"path=$DISK_PATH,format=qcow2,bus=virtio\" \\ --disk \"path=$CLOUD_INIT_ISO,device=cdrom\" \\ --os-variant linux2022 \\ --network network=default,model=virtio \\ --graphics spice,listen=none \\ --console pty,target_type=serial \\ --import \\ --noautoconsole echo \"\" echo \"VM '$NAME' created!\" echo \"\" echo \"Default credentials:\" echo \" User: $DEFAULT_USER\" echo \" Password: $PASSWORD (if SSH key didn't work)\" echo \"\" echo \"Connect:\" echo \" ssh $DEFAULT_USER@\\$(vmip $NAME -w)\" Download the cloud image once, and from then on a fresh VM is a few seconds away:\n# Download a cloud image once wget https://cloud-images.ubuntu.com/minimal/releases/jammy/release/ubuntu-22.04-minimal-cloudimg-amd64.img -O ~/ISOs/ubuntu-cloud.img # Spin up VMs instantly vmcloud web-server ~/ISOs/ubuntu-cloud.img 2 2 20 vmcloud db-server ~/ISOs/ubuntu-cloud.img 8 4 100 # SSH in immediately ssh admin@$(vmip web-server -w) Moving VMs around: vmexport and vmimport I move VMs between machines often enough that this matters. A test VM graduates from my workstation to the server, or gets backed up before I do something I might regret. virsh can do all of it, but it’s dumpxml, copy the disks, fix the paths, redefine, and I forget a step every single time. So I wrapped the whole sequence.\nExport: vmexport #!/bin/bash # vmexport - Export VM to portable archive (disk + XML) # Usage: vmexport \u003cvm-name\u003e [output-dir] set -euo pipefail RED='\\033[0;31m' GREEN='\\033[0;32m' YELLOW='\\033[1;33m' CYAN='\\033[0;36m' NC='\\033[0m' NAME=\"${1:-}\" OUTPUT_DIR=\"${2:-.}\" usage() { echo \"Usage: vmexport \u003cvm-name\u003e [output-dir]\" echo \"\" echo \"Exports VM definition and disks to a directory.\" echo \"The VM should be shut down for consistent export.\" echo \"\" echo \"Examples:\" echo \" vmexport myvm # Export to current directory\" echo \" vmexport myvm /backup/vms # Export to specific directory\" echo \" vmexport myvm - # Stream to stdout (for piping to ssh)\" exit 1 } # Interactive selection with fzf if no VM specified if [[ -z \"$NAME\" ]]; then if command -v fzf \u0026\u003e/dev/null; then NAME=$(virsh list --all --name | grep -v '^$' | \\ fzf --prompt=\"Select VM to export: \" \\ --preview 'virsh dominfo {} 2\u003e/dev/null; echo \"---\"; virsh domblklist {} 2\u003e/dev/null') [[ -z \"$NAME\" ]] \u0026\u0026 exit 1 else usage fi fi # Verify VM exists if ! virsh dominfo \"$NAME\" \u0026\u003e/dev/null; then echo -e \"${RED}Error: VM '$NAME' not found${NC}\" exit 1 fi # Warn if VM is running STATE=$(virsh domstate \"$NAME\" 2\u003e/dev/null) if [[ \"$STATE\" != \"shut off\" ]]; then echo -e \"${YELLOW}Warning: VM is $STATE. For consistent export, shut it down first.${NC}\" read -p \"Continue anyway? [y/N] \" confirm [[ ! \"$confirm\" =~ ^[Yy]$ ]] \u0026\u0026 exit 1 fi # Create export directory EXPORT_DIR=\"${OUTPUT_DIR}/${NAME}-export-$(date +%Y%m%d-%H%M%S)\" mkdir -p \"$EXPORT_DIR\" echo -e \"${GREEN}Exporting VM: $NAME${NC}\" echo \" Output: $EXPORT_DIR\" echo \"\" # Export XML definition echo -e \"${CYAN}Exporting VM definition...${NC}\" virsh dumpxml \"$NAME\" \u003e \"$EXPORT_DIR/${NAME}.xml\" # Export each disk echo -e \"${CYAN}Exporting disks...${NC}\" while IFS= read -r line; do TARGET=$(echo \"$line\" | awk '{print $1}') SOURCE=$(echo \"$line\" | awk '{print $2}') # Skip empty lines and CDROMs [[ -z \"$SOURCE\" || \"$SOURCE\" == \"-\" ]] \u0026\u0026 continue [[ ! -f \"$SOURCE\" ]] \u0026\u0026 continue DISK_NAME=$(basename \"$SOURCE\") echo \" $TARGET: $SOURCE -\u003e $DISK_NAME\" # Use qemu-img convert to compress and potentially convert format qemu-img convert -O qcow2 -c \"$SOURCE\" \"$EXPORT_DIR/$DISK_NAME\" done \u003c \u003c(virsh domblklist \"$NAME\" --details | grep -E \"file\\s+disk\" | awk '{print $3, $4}') # Create metadata file cat \u003e \"$EXPORT_DIR/metadata.txt\" \u003c\u003cEOF VM Export ========= Name: $NAME Date: $(date) Source host: $(hostname) Original state: $STATE Files: $(ls -lh \"$EXPORT_DIR\") EOF # Calculate total size TOTAL_SIZE=$(du -sh \"$EXPORT_DIR\" | cut -f1) echo \"\" echo -e \"${GREEN}Export complete!${NC}\" echo \" Location: $EXPORT_DIR\" echo \" Size: $TOTAL_SIZE\" echo \"\" echo \"To import on another host:\" echo \" scp -r $EXPORT_DIR user@target:/path/\" echo \" vmimport /path/$(basename \"$EXPORT_DIR\")\" Import: vmimport #!/bin/bash # vmimport - Import VM from exported archive # Usage: vmimport \u003cexport-dir\u003e [new-name] set -euo pipefail RED='\\033[0;31m' GREEN='\\033[0;32m' YELLOW='\\033[1;33m' CYAN='\\033[0;36m' NC='\\033[0m' VM_PATH=\"/var/lib/libvirt/images\" EXPORT_DIR=\"${1:-}\" NEW_NAME=\"${2:-}\" usage() { echo \"Usage: vmimport \u003cexport-dir\u003e [new-name]\" echo \"\" echo \"Imports a VM from vmexport archive.\" echo \"\" echo \"Examples:\" echo \" vmimport ./myvm-export-20240101-120000\" echo \" vmimport ./myvm-export-20240101-120000 myvm-restored\" exit 1 } [[ -z \"$EXPORT_DIR\" ]] \u0026\u0026 usage [[ ! -d \"$EXPORT_DIR\" ]] \u0026\u0026 { echo -e \"${RED}Error: Directory not found: $EXPORT_DIR${NC}\"; exit 1; } # Find the XML file XML_FILE=$(find \"$EXPORT_DIR\" -name \"*.xml\" -type f | head -1) [[ -z \"$XML_FILE\" ]] \u0026\u0026 { echo -e \"${RED}Error: No XML definition found in $EXPORT_DIR${NC}\"; exit 1; } # Get original name from XML ORIG_NAME=$(grep -oP \"(?\u003c=\u003cname\u003e)[^\u003c]+\" \"$XML_FILE\") NAME=\"${NEW_NAME:-$ORIG_NAME}\" echo -e \"${GREEN}Importing VM: $NAME${NC}\" echo \" Source: $EXPORT_DIR\" echo \" Original name: $ORIG_NAME\" [[ -n \"$NEW_NAME\" ]] \u0026\u0026 echo \" New name: $NEW_NAME\" echo \"\" # Check if VM already exists if virsh dominfo \"$NAME\" \u0026\u003e/dev/null; then echo -e \"${RED}Error: VM '$NAME' already exists${NC}\" echo \"Use a different name: vmimport $EXPORT_DIR different-name\" exit 1 fi # Copy disks to libvirt images directory echo -e \"${CYAN}Copying disks...${NC}\" declare -A DISK_MAP for disk in \"$EXPORT_DIR\"/*.qcow2 \"$EXPORT_DIR\"/*.img; do [[ ! -f \"$disk\" ]] \u0026\u0026 continue DISK_NAME=$(basename \"$disk\") # If renaming VM, also rename disk files if [[ -n \"$NEW_NAME\" ]]; then NEW_DISK_NAME=\"${DISK_NAME/$ORIG_NAME/$NEW_NAME}\" else NEW_DISK_NAME=\"$DISK_NAME\" fi DEST_PATH=\"$VM_PATH/$NEW_DISK_NAME\" # Check if destination exists if [[ -f \"$DEST_PATH\" ]]; then echo -e \"${YELLOW}Warning: $DEST_PATH already exists${NC}\" read -p \"Overwrite? [y/N] \" confirm [[ ! \"$confirm\" =~ ^[Yy]$ ]] \u0026\u0026 { echo \"Skipping...\"; continue; } fi echo \" $DISK_NAME -\u003e $DEST_PATH\" cp \"$disk\" \"$DEST_PATH\" # Store mapping for XML update DISK_MAP[\"$DISK_NAME\"]=\"$DEST_PATH\" done # Prepare XML with updated paths and name echo -e \"${CYAN}Preparing VM definition...${NC}\" TEMP_XML=$(mktemp) cp \"$XML_FILE\" \"$TEMP_XML\" # Update VM name if changed if [[ -n \"$NEW_NAME\" ]]; then sed -i \"s|\u003cname\u003e$ORIG_NAME\u003c/name\u003e|\u003cname\u003e$NEW_NAME\u003c/name\u003e|g\" \"$TEMP_XML\" fi # Update disk paths for orig_disk in \"${!DISK_MAP[@]}\"; do new_path=\"${DISK_MAP[$orig_disk]}\" # Update any path ending with the original disk name sed -i \"s|file='[^']*${orig_disk}'|file='${new_path}'|g\" \"$TEMP_XML\" sed -i \"s|source file=\\\"[^\\\"]*${orig_disk}\\\"|source file=\\\"${new_path}\\\"|g\" \"$TEMP_XML\" done # Remove UUID (let libvirt generate new one) sed -i '/\u003cuuid\u003e/d' \"$TEMP_XML\" # Remove MAC addresses (let libvirt generate new ones to avoid conflicts) sed -i '/\u003cmac address=/d' \"$TEMP_XML\" # Define the VM echo -e \"${CYAN}Defining VM...${NC}\" virsh define \"$TEMP_XML\" rm \"$TEMP_XML\" echo \"\" echo -e \"${GREEN}Import complete!${NC}\" echo \"\" echo \"Start the VM with: vmstart $NAME\" Migrating live: vmmigrate Sometimes the VM needs to move while it’s still running, or at least without me babysitting the export/import shuffle. This script handles the three cases I hit: a true live migration when there’s shared storage, a live migration that drags the disk along when there isn’t, and a plain offline move that stops, copies, and restarts on the far side:\n#!/bin/bash # vmmigrate - Migrate VM to another libvirt host # Usage: vmmigrate \u003cvm-name\u003e \u003ctarget-host\u003e [--live] [--offline] set -euo pipefail RED='\\033[0;31m' GREEN='\\033[0;32m' YELLOW='\\033[1;33m' CYAN='\\033[0;36m' NC='\\033[0m' NAME=\"${1:-}\" TARGET=\"${2:-}\" MODE=\"${3:---offline}\" # Known hosts for fzf selection (customize this) KNOWN_HOSTS=\"${VM_MIGRATE_HOSTS:-}\" usage() { echo \"Usage: vmmigrate \u003cvm-name\u003e \u003ctarget-host\u003e [--live|--offline]\" echo \"\" echo \"Migrate a VM to another libvirt host.\" echo \"\" echo \"Options:\" echo \" --live Live migration (VM keeps running, requires shared storage\" echo \" or same CPU model)\" echo \" --offline Offline migration (default, VM is stopped, disk copied)\" echo \" --copy Live migration with disk copy (slower but no shared storage needed)\" echo \"\" echo \"Examples:\" echo \" vmmigrate myvm server2 --offline # Stop, copy, start on server2\" echo \" vmmigrate myvm server2 --live # Live migrate (shared storage)\" echo \" vmmigrate myvm server2 --copy # Live migrate with disk copy\" echo \"\" echo \"Environment:\" echo \" VM_MIGRATE_HOSTS Space-separated list of hosts for fzf selection\" exit 1 } # Interactive VM selection if [[ -z \"$NAME\" ]] \u0026\u0026 command -v fzf \u0026\u003e/dev/null; then NAME=$(virsh list --all --name | grep -v '^$' | \\ fzf --prompt=\"Select VM to migrate: \" \\ --preview 'virsh dominfo {} 2\u003e/dev/null') [[ -z \"$NAME\" ]] \u0026\u0026 exit 1 fi # Interactive host selection if [[ -z \"$TARGET\" ]] \u0026\u0026 command -v fzf \u0026\u003e/dev/null; then if [[ -n \"$KNOWN_HOSTS\" ]]; then TARGET=$(echo \"$KNOWN_HOSTS\" | tr ' ' '\\n' | \\ fzf --prompt=\"Select target host: \" \\ --preview 'ssh {} \"virsh list --all\" 2\u003e/dev/null || echo \"Cannot connect\"') else read -p \"Target host: \" TARGET fi [[ -z \"$TARGET\" ]] \u0026\u0026 exit 1 fi [[ -z \"$NAME\" || -z \"$TARGET\" ]] \u0026\u0026 usage # Verify VM exists if ! virsh dominfo \"$NAME\" \u0026\u003e/dev/null; then echo -e \"${RED}Error: VM '$NAME' not found${NC}\" exit 1 fi # Test SSH connectivity echo -e \"${CYAN}Testing connection to $TARGET...${NC}\" if ! ssh -o ConnectTimeout=5 \"$TARGET\" \"virsh version\" \u0026\u003e/dev/null; then echo -e \"${RED}Error: Cannot connect to libvirt on $TARGET${NC}\" echo \"Ensure SSH works and libvirt is running: ssh $TARGET 'virsh version'\" exit 1 fi STATE=$(virsh domstate \"$NAME\" 2\u003e/dev/null) echo -e \"${GREEN}Migrating VM: $NAME${NC}\" echo \" From: $(hostname)\" echo \" To: $TARGET\" echo \" Mode: $MODE\" echo \" Current state: $STATE\" echo \"\" case \"$MODE\" in --live) if [[ \"$STATE\" != \"running\" ]]; then echo -e \"${RED}Error: Live migration requires VM to be running${NC}\" exit 1 fi echo -e \"${CYAN}Starting live migration...${NC}\" echo \"This requires shared storage between hosts.\" virsh migrate --live --persistent --undefinesource \\ \"$NAME\" \"qemu+ssh://$TARGET/system\" echo -e \"${GREEN}Live migration complete!${NC}\" ;; --copy) if [[ \"$STATE\" != \"running\" ]]; then echo -e \"${RED}Error: Copy migration requires VM to be running${NC}\" exit 1 fi echo -e \"${CYAN}Starting live migration with disk copy...${NC}\" echo \"This will copy all disks over the network (can be slow for large disks).\" virsh migrate --live --persistent --undefinesource \\ --copy-storage-all \\ \"$NAME\" \"qemu+ssh://$TARGET/system\" echo -e \"${GREEN}Migration with disk copy complete!${NC}\" ;; --offline|*) echo -e \"${CYAN}Starting offline migration...${NC}\" # Shut down if running if [[ \"$STATE\" == \"running\" ]]; then echo \"Shutting down VM...\" virsh shutdown \"$NAME\" # Wait for shutdown for i in {1..60}; do STATE=$(virsh domstate \"$NAME\" 2\u003e/dev/null) [[ \"$STATE\" == \"shut off\" ]] \u0026\u0026 break sleep 2 done if [[ \"$STATE\" != \"shut off\" ]]; then echo -e \"${YELLOW}VM didn't shut down gracefully, forcing...${NC}\" virsh destroy \"$NAME\" fi fi # Export TEMP_DIR=$(mktemp -d) echo \"Exporting to $TEMP_DIR...\" virsh dumpxml \"$NAME\" \u003e \"$TEMP_DIR/${NAME}.xml\" # Copy disks while IFS= read -r line; do SOURCE=$(echo \"$line\" | awk '{print $2}') [[ -z \"$SOURCE\" || \"$SOURCE\" == \"-\" || ! -f \"$SOURCE\" ]] \u0026\u0026 continue DISK_NAME=$(basename \"$SOURCE\") echo \" Copying $DISK_NAME...\" cp \"$SOURCE\" \"$TEMP_DIR/\" done \u003c \u003c(virsh domblklist \"$NAME\" --details | grep -E \"file\\s+disk\" | awk '{print $3, $4}') # Transfer to remote echo \"Transferring to $TARGET...\" REMOTE_PATH=\"/var/lib/libvirt/images\" # Copy disks first for disk in \"$TEMP_DIR\"/*.qcow2 \"$TEMP_DIR\"/*.img; do [[ ! -f \"$disk\" ]] \u0026\u0026 continue scp \"$disk\" \"$TARGET:$REMOTE_PATH/\" done # Update XML with new paths and copy sed -i \"s|/var/lib/libvirt/images/|$REMOTE_PATH/|g\" \"$TEMP_DIR/${NAME}.xml\" sed -i '/\u003cuuid\u003e/d' \"$TEMP_DIR/${NAME}.xml\" scp \"$TEMP_DIR/${NAME}.xml\" \"$TARGET:/tmp/\" # Define on remote ssh \"$TARGET\" \"virsh define /tmp/${NAME}.xml \u0026\u0026 rm /tmp/${NAME}.xml\" # Clean up local rm -rf \"$TEMP_DIR\" # Optionally undefine locally read -p \"Remove VM from this host? [y/N] \" confirm if [[ \"$confirm\" =~ ^[Yy]$ ]]; then virsh undefine \"$NAME\" --remove-all-storage echo \"VM removed from local host.\" fi echo \"\" echo -e \"${GREEN}Offline migration complete!${NC}\" echo \"Start the VM on $TARGET with: ssh $TARGET 'vmstart $NAME'\" ;; esac Cloning: vmclone When I want a throwaway copy of a machine I know is good, cloning beats building from scratch. Keep a clean ubuntu-base around, clone it, and the copy is ready in seconds:\n#!/bin/bash # vmclone - Clone an existing VM # Usage: vmclone \u003csource-vm\u003e \u003cnew-name\u003e set -euo pipefail RED='\\033[0;31m' GREEN='\\033[0;32m' YELLOW='\\033[1;33m' NC='\\033[0m' SOURCE=\"${1:-}\" NEW_NAME=\"${2:-}\" usage() { echo \"Usage: vmclone \u003csource-vm\u003e \u003cnew-name\u003e\" echo \"\" echo \"Creates a full clone of an existing VM.\" echo \"Source VM should be shut down for consistent clone.\" echo \"\" echo \"Examples:\" echo \" vmclone ubuntu-base ubuntu-test\" echo \" vmclone k8s-template k8s-worker-3\" exit 1 } # Interactive selection with fzf if [[ -z \"$SOURCE\" ]] \u0026\u0026 command -v fzf \u0026\u003e/dev/null; then SOURCE=$(virsh list --all --name | grep -v '^$' | \\ fzf --prompt=\"Select VM to clone: \" \\ --preview 'virsh dominfo {} 2\u003e/dev/null') [[ -z \"$SOURCE\" ]] \u0026\u0026 exit 1 fi if [[ -z \"$NEW_NAME\" ]]; then read -p \"New VM name: \" NEW_NAME [[ -z \"$NEW_NAME\" ]] \u0026\u0026 { echo \"Name required\"; exit 1; } fi [[ -z \"$SOURCE\" || -z \"$NEW_NAME\" ]] \u0026\u0026 usage # Verify source exists if ! virsh dominfo \"$SOURCE\" \u0026\u003e/dev/null; then echo -e \"${RED}Error: Source VM '$SOURCE' not found${NC}\" exit 1 fi # Check if target already exists if virsh dominfo \"$NEW_NAME\" \u0026\u003e/dev/null; then echo -e \"${RED}Error: VM '$NEW_NAME' already exists${NC}\" exit 1 fi # Warn if source is running STATE=$(virsh domstate \"$SOURCE\" 2\u003e/dev/null) if [[ \"$STATE\" != \"shut off\" ]]; then echo -e \"${YELLOW}Warning: Source VM is $STATE.${NC}\" echo \"Clone may be inconsistent. Shut down source for best results.\" read -p \"Continue anyway? [y/N] \" confirm [[ ! \"$confirm\" =~ ^[Yy]$ ]] \u0026\u0026 exit 1 fi echo -e \"${GREEN}Cloning VM: $SOURCE -\u003e $NEW_NAME${NC}\" echo \"\" # Use virt-clone for the heavy lifting virt-clone \\ --original \"$SOURCE\" \\ --name \"$NEW_NAME\" \\ --auto-clone echo \"\" echo -e \"${GREEN}Clone complete!${NC}\" echo \"\" echo \"Start the clone with: vmstart $NEW_NAME\" # Show disk locations echo \"\" echo \"Clone disks:\" virsh domblklist \"$NEW_NAME\" --details | grep -E \"file\\s+disk\" | awk '{print \" \" $4}' Putting the migration tools together # Clone a template for testing vmclone ubuntu-base test-ubuntu vmstart test-ubuntu # Export VM for backup vmexport important-vm /backup/vms/ # Move VM to another server (offline) vmmigrate webserver server2.local --offline # Live migration with shared storage (NFS/Ceph) vmmigrate database server2.local --live # Move VM to server without shared storage vmmigrate webserver server2.local --copy # Import a VM from backup vmimport /backup/vms/important-vm-export-20240101/ vmimport /backup/vms/important-vm-export-20240101/ restored-vm # With new name Installation All scripts are available on GitHub: github.com/kapott/vm-scripts\n# Clone and install git clone https://github.com/kapott/vm-scripts.git cd vm-scripts ./install.sh Or by hand. Drop the scripts in ~/.local/bin/ and make sure it’s on your PATH:\n# In .bashrc or .zshrc export PATH=\"$HOME/.local/bin:$PATH\" # Source the aliases source ~/.local/share/vm-scripts/vm-aliases.sh Going interactive with fzf If you’ve got fzf around, the whole thing gets a lot nicer. Half the pain with virsh is remembering exact VM names. With fzf you type a couple of letters, see a live preview of the VM, and hit enter. These functions wrap the scripts above with that selector:\n# ~/.vm-aliases - fzf enhanced section # Interactive VM selector (use in other scripts) vmf() { virsh list --all --name | grep -v '^$' | \\ fzf --prompt=\"Select VM: \" \\ --preview 'virsh dominfo {} 2\u003e/dev/null; echo \"---\"; virsh domifaddr {} 2\u003e/dev/null' \\ --preview-window=right:50% } # SSH into VM with fuzzy selection vmssh() { local user=\"${1:-root}\" local vm=$(virsh list --name | grep -v '^$' | \\ fzf --prompt=\"SSH to VM: \" \\ --preview 'echo \"IP: $(vmip {} 2\u003e/dev/null || echo \"N/A\")\"; echo \"---\"; virsh dominfo {} 2\u003e/dev/null | grep -E \"State|CPU|memory\"') [[ -z \"$vm\" ]] \u0026\u0026 return 1 local ip=$(vmip \"$vm\" 2\u003e/dev/null) [[ -z \"$ip\" ]] \u0026\u0026 { echo \"No IP for $vm\"; return 1; } ssh \"${user}@${ip}\" } # Start VMs with multi-select vmstartf() { virsh list --name --state-shutoff | grep -v '^$' | \\ fzf --multi --prompt=\"Start VMs (TAB to select multiple): \" \\ --preview 'virsh dominfo {} 2\u003e/dev/null' | \\ xargs -I{} virsh start {} } # Stop VMs with multi-select vmstopf() { virsh list --name --state-running | grep -v '^$' | \\ fzf --multi --prompt=\"Stop VMs (TAB to select multiple): \" \\ --preview 'virsh dominfo {} 2\u003e/dev/null; echo \"---\"; echo \"IP: $(vmip {} 2\u003e/dev/null)\"' | \\ xargs -I{} virsh shutdown {} } # Delete VM with safety preview vmrmf() { local vm=$(virsh list --all --name | grep -v '^$' | \\ fzf --prompt=\"DELETE VM (careful!): \" \\ --preview 'echo \"=== VM Info ===\"; virsh dominfo {} 2\u003e/dev/null; echo; echo \"=== Disks (WILL BE DELETED) ===\"; virsh domblklist {} 2\u003e/dev/null' \\ --preview-window=right:60% \\ --header=\"WARNING: This will delete the VM and ALL storage\") [[ -z \"$vm\" ]] \u0026\u0026 return 0 echo \"Delete '$vm' and all storage? [y/N]\" read -r confirm [[ \"$confirm\" =~ ^[Yy]$ ]] \u0026\u0026 virsh undefine \"$vm\" --remove-all-storage } # Snapshot management with fzf vmsnapf() { local vm=$(vmf) [[ -z \"$vm\" ]] \u0026\u0026 return 1 local action=$(echo -e \"create\\nlist\\nrevert\\ndelete\" | fzf --prompt=\"Snapshot action for $vm: \") case \"$action\" in create) read -p \"Snapshot name (empty for timestamp): \" name vmsnap create \"$vm\" \"$name\" ;; list) vmsnap list \"$vm\" ;; revert) local snap=$(virsh snapshot-list \"$vm\" --name 2\u003e/dev/null | grep -v '^$' | \\ fzf --prompt=\"Revert to snapshot: \" \\ --preview \"virsh snapshot-info $vm {} 2\u003e/dev/null\" \\ --tac) [[ -n \"$snap\" ]] \u0026\u0026 vmsnap revert \"$vm\" \"$snap\" ;; delete) local snap=$(virsh snapshot-list \"$vm\" --name 2\u003e/dev/null | grep -v '^$' | \\ fzf --prompt=\"Delete snapshot: \" \\ --preview \"virsh snapshot-info $vm {} 2\u003e/dev/null\") [[ -n \"$snap\" ]] \u0026\u0026 vmsnap delete \"$vm\" \"$snap\" ;; esac } # Quick VM from ISO with fzf selection vmquickf() { local iso=$(find ~/Downloads ~/ISOs /var/lib/libvirt/images -maxdepth 2 \\ \\( -name \"*.iso\" -o -name \"*.img\" \\) 2\u003e/dev/null | \\ fzf --prompt=\"Select ISO/image: \" \\ --preview 'ls -lh {}; file {}') [[ -z \"$iso\" ]] \u0026\u0026 return 1 local suggested=$(basename \"$iso\" | sed 's/\\.[^.]*$//' | tr '[:upper:]' '[:lower:]' | tr ' _' '-' | cut -c1-20) read -p \"VM name [$suggested]: \" name name=\"${name:-$suggested}\" read -p \"RAM in GB [2]: \" ram read -p \"CPUs [2]: \" cpus read -p \"Disk in GB [20]: \" disk vmquick \"$name\" \"$iso\" \"${ram:-2}\" \"${cpus:-2}\" \"${disk:-20}\" } I haven’t typed a VM name in full since. A few letters and a preview.\nThe full aliases file Here’s the complete .vm-aliases file in one place, ready to source:\n# ~/.vm-aliases # Source this in .bashrc: source ~/.vm-aliases # === Status === alias vms='virsh list --all' alias vmsr='virsh list' alias vmnets='virsh net-list --all' alias vmpools='virsh pool-list --all' # === Power === alias vmstart='virsh start' alias vmstop='virsh shutdown' alias vmkill='virsh destroy' alias vmreboot='virsh reboot' alias vmsuspend='virsh suspend' alias vmresume='virsh resume' # === Console === alias vmcon='virsh console' alias vmview='virt-viewer' # === Info === alias vminfo='virsh dominfo' alias vmblk='virsh domblklist' alias vmnet='virsh domiflist' # === Management === alias vmedit='virsh edit' alias vmxml='virsh dumpxml' alias vmrm='virsh undefine --remove-all-storage' # === Completions (bash) === _vm_complete() { local cur=\"${COMP_WORDS[COMP_CWORD]}\" COMPREPLY=($(compgen -W \"$(virsh list --all --name 2\u003e/dev/null)\" -- \"$cur\")) } complete -F _vm_complete vmstart vmstop vmkill vmreboot vmcon vmview vminfo vmblk vmnet vmedit vmxml vmrm vmip vmsnap vmupgrade vmsuspend vmresume What ten seconds changed Back to that virt-install line at the top. KVM itself was fine the whole time. What cost me was the distance between wanting a VM and having one, and these scripts bring that down to about ten seconds of typing.\nTen seconds changed my habits more than I expected. When a clean VM is faster than making coffee, you stop testing things in production because isolating them feels like work. You stop picking a container when a VM was the right call. The test you’d have skipped gets run. My work got safer, and I never planned that.\nThere’s a sovereignty angle too, the one I keep coming back to in Sovereign Infrastructure. These are a couple hundred lines of shell. I can read all of them, and when one breaks I fix it, because nothing in here is hidden from me. That cost me some time up front. It bought me tooling I own and understand. virt-manager would’ve been quicker to adopt, and I’d still be clicking through tabs.\nThe scripts are on GitHub, linked above. Take them, gut them, rename everything to match your own muscle memory. That’s what I did with the commands I copied off a wiki in the first place.","permalink":"https://tommeurs.nl/en/posts/kvm-qemu-scripts-aliases/","title":"KVM/QEMU Scripts and Aliases: Making Virtualization Actually Usable"},{"content":"Someone opens a cloud console, clicks through a wizard, accepts a few defaults, and a resource appears. The dashboard goes green and everyone moves on. That’s still how a lot of infrastructure gets built in 2026, and I’ve built some of it that way myself.\nI can’t work like that for long. When I click a button and infrastructure appears, it feels borrowed. I want the configuration written down somewhere I can read it back, and I want to know why something exists, not only that it exists. The console gives me a green tick. It doesn’t give me any of that.\nSo this post is about the distance between infrastructure that happens to you and infrastructure you can hold in your head.\nWhat click-ops costs Click your way to a working system and you get a working system. You also get a pile of decisions nobody wrote down.\nEvery resource that comes out of a GUI carries settings you never consciously chose and dependencies you never saw form. State piles up where nobody audits it. None of that hurts on day one. It hurts six months later, when someone asks why a security group has 47 rules, or whether that orphaned load balancer is safe to delete, and the honest answer is that nobody knows.\nWon’t the documentation save you? It won’t. Docs describe what things should be. The moment you change something in the console without updating the wiki, the two drift apart, and they always drift apart. I’ve done this myself, more than once. After a while you’re reading the wiki and hoping it’s still true.\nWhen everything is written down Now picture the same system, except every piece of it exists as text you can read and recreate.\n# This exists because we need to allow traffic from the monitoring namespace apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-prometheus namespace: production spec: podSelector: matchLabels: app: api ingress: - from: - namespaceSelector: matchLabels: name: monitoring ports: - port: 9090 protocol: TCP The comment explains why. The code shows what. Both live in the same file, under version control, readable by anyone who clones the repo. Nobody has to do archaeology later, because the reason never got separated from the thing.\nThat one property, explicit rather than implicit, is why I trust Infrastructure as Code. Best practice or not, I can read it.\nThe cost is in your head Let me go back to the messy side, because the cost I described above is the cheap one. The expensive one is what click-ops does to your head.\nYou made a configuration decision for a reason. Months pass. Was that timeout value load-bearing? Did that IP whitelist protect against something specific, or did someone add it during an incident and forget? Nobody remembers, so the setting stays forever and slowly turns into the kind of technical debt everyone is a bit afraid to touch.\nAnd there’s always one person on the team who happens to remember why the network looks the way it does. When something breaks, everyone turns to that person. That’s a single point of failure, and it’s a human. Humans forget things. I know, because I’ve been that person.\nWritten-down infrastructure moves that knowledge from one tired person into a file the whole team can read.\nWhat you get back Once the system lives in Git, a few things stop being heroics.\nHistory is no longer a guess. Every change carries a commit message and an author, so you can trace a decision back through time:\ngit log --oneline -- network-policy.yaml a1b2c3d Add network policy for Prometheus scraping d4e5f6g Restrict to specific port (was allowing all) g7h8i9j Initial creation for monitoring access A second pair of eyes gets a look before production does. The classic “allow all” mistake is hard to spot in a console. In a diff it jumps off the screen:\n# Pull request diff - cidrBlocks: - - 0.0.0.0/0 # DANGER: allows all traffic + cidrBlocks: + - 10.0.0.0/8 # Internal network only Recovery is a command rather than an anxious afternoon. When a cluster dies, you don’t rebuild it from memory:\n# Recreate everything terraform apply kubectl apply -k ./infrastructure argocd app sync --all The declarative part is what holds this together. You write down the state you want, and making reality match it is the system’s job.\n# I declare: there should be 3 replicas apiVersion: apps/v1 kind: Deployment spec: replicas: 3 A pod dies, Kubernetes notices there are two instead of three, and starts a new one without anyone clicking anything. You declared the contract. The system enforces it.\nThere’s room to write down intent too, and not only mechanics. Tags and comments turn configuration into a message to whoever reads it next, usually a future version of you:\n# Terraform: Why this VPC exists resource \"aws_vpc\" \"main\" { cidr_block = \"10.0.0.0/16\" tags = { Name = \"production\" Environment = \"production\" Purpose = \"Primary VPC for production workloads\" Owner = \"platform-team\" CostCenter = \"infrastructure\" } } This is roughly how my homelab is built. Everything that runs there is defined in one repo:\nhomelab/ ├── infrastructure/ │ ├── terraform/ # VMs, networks, DNS │ └── kubernetes/ # Cluster configuration ├── applications/ │ ├── argocd/ # GitOps controller │ ├── monitoring/ # Prometheus, Grafana, Loki │ └── services/ # Home Assistant, Forgejo, etc. └── secrets/ └── external-secrets/ # ExternalSecret manifests (secrets live in Vault) If the whole thing burned down tomorrow, I could rebuild it with three commands and a fresh disk, because I never trusted my own memory to hold the configuration in the first place.\nWhy people don’t switch If this is so clearly better, why does click-ops survive? Because the objections feel real in the moment. Here’s what I hear most, and what I make of each.\n“It’s slower than clicking.” At first, sure. Writing the YAML once takes ten minutes against a five-minute click. But you write the YAML once and it works in every environment. The clicking you repeat by hand, every time, and it transfers to nothing.\n“Not everything can be Infrastructure as Code.” True. Some steps need a human, and the honest answer is to write those down instead of pretending they don’t exist:\n# manual-steps.md ## Setting up external DNS provider 1. Log into Cloudflare dashboard 2. Create API token with Zone:Edit permission 3. Add token to Vault at `secret/cloudflare/api-token` Note: This is manual because Cloudflare doesn't support organization-level tokens via API. Last performed: 2026-03-15 by @tom The goal is that nothing goes undocumented, including the parts you can’t automate.\n“My team won’t adopt it.” Then don’t ask them to adopt all of it. Automate one painful thing, do it well, and let the result do the talking. The three hours you didn’t spend rebuilding a server. The outage a code review would’ve caught. That convinces people faster than any argument I’ve made.\nStart from where you are You don’t have to rewrite your infrastructure to get here. Start by capturing what already exists.\n# Export existing Kubernetes resources kubectl get deployment my-app -o yaml \u003e deployments/my-app.yaml # Import existing Terraform resources terraform import aws_instance.web i-1234567890abcdef0 Then give it some structure, split by environment, so the differences are visible instead of buried:\ninfrastructure/ ├── base/ # Common configuration │ ├── namespaces.yaml │ ├── rbac.yaml │ └── network-policies.yaml ├── environments/ │ ├── development/ │ │ └── kustomization.yaml # Overlays for dev │ ├── staging/ │ │ └── kustomization.yaml │ └── production/ │ └── kustomization.yaml # Production-specific settings └── applications/ ├── api/ ├── frontend/ └── database/ And stop applying it by hand. Wire it into CI, or better, hand it to GitOps and let a controller keep reality in sync with the repo:\napiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: infrastructure spec: source: repoURL: https://gitlab.internal/infrastructure path: environments/production destination: server: https://kubernetes.default.svc syncPolicy: automated: selfHeal: true Where this touches sovereignty For me this lands on something bigger than tidy YAML. Digital sovereignty goes well past owning the hardware. It means understanding and controlling the systems you depend on, and you can’t control what you can’t see.\nInfrastructure as Code is what makes that control real. Moving providers becomes a config change. Recreating everything from scratch becomes a command. And for any piece of the stack I can answer “why does this exist” without hunting down whoever last clicked the buttons and hoping they remember.\nThat’s the part I care about. I understand my infrastructure because I wrote it down, line by line, in files I can read. When something breaks, I don’t open a console. I open the repo, and it tells me what should be running and why. For a brain that treats an unexplained system as a low-grade itch, the afternoon it took to write it all down was cheap.","permalink":"https://tommeurs.nl/en/posts/infrastructure-as-code-control/","title":"Infrastructure as Code for People Who Need to Understand"},{"content":" TL;DR Three copies, two kinds of media, one of them offsite. Restic for files, Velero for the cluster, a Backblaze bucket for the offsite copy, and a restore test on a schedule. Mine costs about €5 a month, and the restore test is the part I still have to make myself do.\nPicture the disk in your homelab dying tonight. Not corrupting, not throwing SMART warnings for a week first, just gone. The git server with every repo you care about, the password vault, fifteen years of family photos, the home automation that runs the house. All of it on a drive that’s decided it’s done.\nCan you say what happens next without your stomach dropping? If not, you don’t have backups. You have hope, and I ran on hope for longer than I’d like to admit.\nThe good news is that you don’t need anything clever. The 3-2-1 rule has been around for decades and it’s almost boringly simple. This post takes you from “I think I have something” to a setup you’ve tested. I’ll start with the version you can run today and layer the rest on top.\nThe 3-2-1 rule Three copies, two different media, one offsite. That’s the whole rule. The rest of this post is filling in those three numbers with real tools.\n3-2-1 Backup Rule\n1 Offsite\n2 Media Types\n3 Copies\nCloud/Remote\nOriginal Data\nCopy 1\n(Original)\nCopy 2\n(Local Backup)\nCopy 3\n(Offsite)\nNVMe/SSD\nHDD/NAS\nWhy three copies Copy 1: Your live data (original) Copy 2: Local backup (fast restore) Copy 3: Offsite backup (disaster recovery) One copy is your data waiting to fail. Two copies feel safer, right up until one event takes out both. A fire does that. So does a ransomware payload that walks every mount it can reach. Three copies with real distance between them is where I’d start calling it resilience.\nWhy two media types Different hardware fails in different ways, and you want your copies spread across those failure modes so one bad day can’t reach all of them.\nSSDs can fail silently through bit rot, with data quietly going bad before you notice HDDs die mechanically, often with enough warning to react if you’re watching RAID is not a backup. It protects against a drive dropping out, and does nothing about corruption, deletion, or a bad write that gets faithfully mirrored across every disk Put your copies on an SSD, a spinning disk and a remote bucket, and whatever kills one rarely kills the others.\nWhy one offsite Your house can burn down. A power surge can cook everything on the same circuit. Local copies are fast and convenient and worth nothing against anything that hits the whole building. The offsite copy is just the one that sits in a different building.\nWhat to back up Before you back up anything, sort your data into three buckets. Most people back up far too much, or they quietly skip the one thing they can’t rebuild. A few minutes of sorting fixes both.\nCritical, backed up daily Data Why Tool Databases Can’t recreate pg_dump, Velero Secrets/credentials Security critical Vault export, External Secrets Configuration System state Git (already offsite) Personal files Irreplaceable Restic Important, backed up weekly Data Why Tool Container images Rebuild takes time Registry backup Persistent volumes Stateful workloads Longhorn /Velero Logs (compressed) Forensics Loki snapshots Rebuildable, not backed up Base OS (reinstall from ISO) Downloaded packages (re-download) Cached data (regenerates) Temporary files If you can get it back with a command or a download, leave it out. Backup space costs money and backup time costs time, and spending either on the base OS is a waste.\nBackup tools Right, that’s the theory. Here’s where it gets practical. I’ll start with the simplest tool that gives you a real second copy, then build up to the Kubernetes-aware stuff.\nRestic for file-level backups If you do nothing else after reading this, set up Restic. It’s what I use for file backups. It’s fast, it encrypts by default, it deduplicates so repeat backups stay small, and it talks to a local disk or an S3 bucket without you changing anything. This is the minimum viable backup, and you can have it running in five minutes.\n# Initialize repository restic init --repo /mnt/backup/restic # Or with S3 backend restic init --repo s3:s3.amazonaws.com/my-bucket # Backup a directory restic backup /home/user/documents # Backup with exclusions restic backup /data \\ --exclude=\"*.tmp\" \\ --exclude=\".cache\" \\ --exclude=\"node_modules\" # List snapshots restic snapshots # Restore restic restore latest --target /restore/location Automated Restic backups A backup you have to remember to run is one you’ll forget. Wrap the commands in a script, point it at your offsite bucket, and let it prune so the repo doesn’t grow forever.\n#!/bin/bash # /usr/local/bin/backup.sh export RESTIC_REPOSITORY=\"s3:s3.eu-west-1.amazonaws.com/homelab-backups\" export RESTIC_PASSWORD_FILE=\"/etc/restic/password\" export AWS_ACCESS_KEY_ID=\"your-key\" export AWS_SECRET_ACCESS_KEY=\"your-secret\" # Backup restic backup /data/important \\ --exclude-caches \\ --tag homelab \\ --tag daily # Prune old snapshots (keep 7 daily, 4 weekly, 12 monthly) restic forget \\ --keep-daily 7 \\ --keep-weekly 4 \\ --keep-monthly 12 \\ --prune # Check repository integrity restic check That restic check at the end matters more than it looks. It reads the repository back and verifies it. Without it you’ve got a folder full of bytes and no idea whether they’re readable.\nCron job:\n# /etc/cron.d/restic-backup 0 3 * * * root /usr/local/bin/backup.sh \u003e\u003e /var/log/restic-backup.log 2\u003e\u00261 Velero for Kubernetes Once you run a cluster, file-level backups stop being enough. You also want the cluster’s own state, the Deployments and ConfigMaps and PersistentVolumeClaims. Velero backs up Kubernetes resources and their persistent volumes together, so one restore brings back the workloads and their data.\n# Install Velero with S3 backend velero install \\ --provider aws \\ --plugins velero/velero-plugin-for-aws:v1.8.0 \\ --bucket velero-backups \\ --backup-location-config region=eu-west-1 \\ --secret-file ./credentials-velero \\ --use-volume-snapshots=true \\ --snapshot-location-config region=eu-west-1 Scheduled backups apiVersion: velero.io/v1 kind: Schedule metadata: name: daily-backup namespace: velero spec: schedule: \"0 3 * * *\" template: includedNamespaces: - production - gitlab - monitoring excludedResources: - events - pods ttl: 720h # Keep for 30 days storageLocation: default volumeSnapshotLocations: - default Restore from Velero # List backups velero backup get # Describe a backup velero backup describe daily-backup-20260518030000 # Restore entire backup velero restore create --from-backup daily-backup-20260518030000 # Restore specific namespace velero restore create --from-backup daily-backup-20260518030000 \\ --include-namespaces gitlab Longhorn backups If you run Longhorn for storage, you already have a backup engine under your volumes, and it talks straight to S3:\n# Configure backup target apiVersion: longhorn.io/v1beta1 kind: Setting metadata: name: backup-target namespace: longhorn-system value: \"s3://longhorn-backups@eu-west-1/\" --- apiVersion: longhorn.io/v1beta1 kind: Setting metadata: name: backup-target-credential-secret namespace: longhorn-system value: \"longhorn-s3-secret\" Schedule recurring backups:\napiVersion: longhorn.io/v1beta1 kind: RecurringJob metadata: name: daily-backup namespace: longhorn-system spec: cron: \"0 3 * * *\" task: backup groups: - default retain: 7 concurrency: 2 Offsite options The “1” in 3-2-1 is the part people skip, usually because it sounds like the expensive, complicated one. It isn’t. You’ve got three honest options. They trade cost against control, and how far down that list you go says a lot about how much sovereignty you want.\nCloud storage The cheapest way to get offsite. Pick a provider, point Restic or Velero at it, done.\nProvider Cost Pros Cons Backblaze B2 $0.005/GB Cheap, S3-compatible US-based Wasabi $0.0059/GB No egress fees 90-day minimum AWS S3 Glacier $0.004/GB Very cheap Slow retrieval Hetzner Storage Box €3.81/1TB EU-based, cheap SFTP/WebDAV only Want the offsite copy inside the EU? The Hetzner Storage Box is hard to beat on price. Backblaze is the easy default when you just want it working. Mine is at Backblaze, US-based, so I don’t get to be smug about the EU option.\nSecond location Got a friend who also runs a homelab? Back each other up. They host an encrypted copy of your data, you host one of theirs, and neither of you can read what you’re storing.\nTheir Home\nYour Home\nEncrypted\nEncrypted\nYour Data\nTheir Backup\n(encrypted)\nTheir Data\nYour Backup\n(encrypted)\nThis costs you some disk and some upload bandwidth, and your offsite copy never touches a cloud provider.\nSelf-hosted cloud The full sovereignty version. Run your own S3-compatible storage at a second physical location and back up to that.\n# MinIO at remote location apiVersion: apps/v1 kind: Deployment metadata: name: minio spec: template: spec: containers: - name: minio image: minio/minio args: - server - /data env: - name: MINIO_ROOT_USER valueFrom: secretKeyRef: name: minio-credentials key: user - name: MINIO_ROOT_PASSWORD valueFrom: secretKeyRef: name: minio-credentials key: password Reach it over Tailscale so the storage never has to sit on the public internet.\nDatabase backups Databases need their own treatment. Copy the files out from under a running database and you get a snapshot that may be mid-write, and a mid-write snapshot may not restore. Dump them properly instead.\nPostgreSQL #!/bin/bash # Kubernetes PostgreSQL backup NAMESPACE=\"gitlab\" POD=$(kubectl get pod -n $NAMESPACE -l app=postgresql -o jsonpath='{.items[0].metadata.name}') DATE=$(date +%Y%m%d_%H%M%S) # Dump all databases kubectl exec -n $NAMESPACE $POD -- \\ pg_dumpall -U postgres | \\ gzip \u003e /backup/postgres_${DATE}.sql.gz # Upload to S3 restic backup /backup/postgres_${DATE}.sql.gz --tag postgres --tag daily Vault backup Vault is the one I lose sleep over, because losing it means losing access to everything else. Take a raft snapshot, encrypt it before it leaves the machine, and only then ship it off.\n# Export Vault data (requires root token) vault operator raft snapshot save /backup/vault_$(date +%Y%m%d).snap # Encrypt and upload gpg --encrypt --recipient backup@example.com /backup/vault_$(date +%Y%m%d).snap restic backup /backup/vault_$(date +%Y%m%d).snap.gpg --tag vault Testing restores Here’s the part everyone nods along to and almost nobody does. I’ve nodded along too. A backup you haven’t tested is a guess. The first time you run a restore shouldn’t be the time you need it, with production down, and that’s when you find out the dumps have been empty for three months.\nMonthly restore test So test on a schedule. Restore into a throwaway namespace, check the thing comes up, tear it down.\n#!/bin/bash # test-restore.sh # Create test namespace kubectl create namespace restore-test # Restore from Velero velero restore create test-restore \\ --from-backup $(velero backup get -o json | jq -r '.items[0].metadata.name') \\ --include-namespaces gitlab \\ --namespace-mappings gitlab:restore-test # Wait for restore velero restore wait test-restore # Verify pods are running kubectl get pods -n restore-test # Test application (example: GitLab) kubectl port-forward -n restore-test svc/gitlab 8080:80 \u0026 curl -s http://localhost:8080/health | grep \"ok\" # Cleanup kubectl delete namespace restore-test Document recovery procedures When something is on fire you don’t want to reverse-engineer your own restore process from memory. Write the runbook down for each critical system, with how long it took and when you last proved it works.\n# GitLab Recovery Procedure ## Prerequisites - Access to Velero backups - Access to PostgreSQL backups - GitLab Helm values ## Steps 1. Restore PostgreSQL from backup 2. Restore GitLab PVCs with Velero 3. Deploy GitLab with same Helm values 4. Verify user login works 5. Verify repositories are accessible ## Estimated Time: 45 minutes ## Last Tested: 2026-05-01 That “Last Tested” line keeps the test honest. If the date is a year old, nobody knows whether the procedure still works, and that includes you.\nMonitoring backups A backup job that silently stopped two weeks ago is worse than no backup, because you think you’re covered. Wire the jobs into your monitoring so a failure shouts at you.\nPrometheus alerts groups: - name: backup-alerts rules: - alert: BackupFailed expr: restic_backup_last_successful_timestamp \u003c (time() - 86400) for: 1h labels: severity: critical annotations: summary: \"Backup hasn't succeeded in 24 hours\" - alert: BackupStorageLow expr: restic_repository_size_bytes / restic_repository_max_bytes \u003e 0.9 for: 1h labels: severity: warning annotations: summary: \"Backup storage over 90% full\" Backup dashboard Track in Grafana:\nLast successful backup time Backup duration trend Storage usage Restore test results My backup setup Here’s how it fits together in my own homelab, which is the layers above stacked into one flow. Local copies for fast restores, an offsite bucket for the disaster case, and Git for anything that’s already text.\nOffsite (Backblaze B2)\nLocal Backup (NAS)\nHomelab (K3s)\nPersistent Volumes\nDatabases\nConfigs (Git)\nLonghorn Snapshots\nRestic Repository\nVelero Backups\nRestic Offsite\nForgejo (self-hosted)\nGitHub Mirror\nSchedule What Frequency Retention Location Longhorn snapshots Hourly 24 hours Local NVMe Longhorn backups Daily 7 days NAS Velero full backup Daily 30 days Backblaze B2 Database dumps Daily 30 days Backblaze B2 Git repos Push Forever GitHub mirror Costs Backblaze B2: ~€5/month for 200GB NAS storage: Already owned Total: ~€5/month for peace of mind Five euros a month, and the NAS was already there. The photos are the reason I pay it.\nCommon mistakes I’ve made most of these myself, or watched friends make them. They all sound reasonable until the day they cost you something.\n“RAID is my backup” RAID protects against a drive failing. It does nothing about:\nAccidental deletion Ransomware Software bugs corrupting data Fire/flood/theft An rm -rf in the wrong directory propagates across every disk in the array instantly. RAID keeps the array up when a drive dies. Nothing on that list is a drive dying.\n“I’ll restore when I need to” If you’ve never restored, you don’t know whether your backups work, and the day you need them is the day you’ll find out. Test quarterly at the absolute minimum.\n“I back up everything” Backing up 10TB of films you can re-download burns money and time you could spend protecting the data you can’t replace. Sort first, then back up the irreplaceable stuff.\n“My backup is in the same room” One fire takes out the server and the backup drive next to it. Offsite is the part that makes the rest of the rule worth doing.\nWhy this matters Data loss isn’t a maybe. Drives carry a 3-5% annual failure rate, people run the wrong command, software corrupts databases in ways that pass every health check, and houses do occasionally catch fire. Run a homelab long enough and you’ll meet at least one of these.\nWhat changes the outcome is whether you did the boring work ahead of time. It’s the same thing I keep coming back to on this blog, the difference between a system you understand and control and one you’re hoping holds together. With a tested backup, losing a disk costs you an afternoon. Without one it costs you the photos.\nSo spend the five euros and the afternoon. Then put the restore test in your calendar, because that’s the step I keep having to push myself to do.","permalink":"https://tommeurs.nl/en/posts/homelab-backup-strategy/","title":"Backup Strategy for Your Homelab: The 3-2-1 Rule in Practice"},{"content":"The first time it bit me I was halfway through a database migration over SSH. The train went into a tunnel, the WiFi dropped, and the process on the other end died with the connection. I sat looking at a dead prompt, wondering how much of the migration had committed.\nA multiplexer would have saved me that hour. The process keeps running on the server whether I’m attached or not, so a dropped connection costs me a reconnect and nothing else. I reattach and the work is where I left it. That one property is why a multiplexer goes on every machine I touch, before the editor and before the dotfiles.\nOne thing before the comparison. I’ve driven tmux for years and I’m not planning to stop, so read the rest with that in mind. I have spent real time in Screen and in Zellij too, enough to know where each one beats tmux, and I’ll say so when it does.\nWhat a terminal multiplexer does A multiplexer sits between your shell and your terminal emulator. It owns the session, and your terminal window is a viewport onto it. Close the viewport and the session carries on without you. That one layer of indirection gives you four things:\nSession persistence: Detach, reconnect later, everything still running Screen splits: Multiple panes in one window Window management: Switch between workspaces Remote parity: Same interface whether you’re local or SSH’d into a box on the other side of the country Terminal Emulator\nMultiplexer\nShell 1\nShell 2\nShell 3\nBefore I walk through the three, here’s what I judge them on. Not feature count, since all three can split a screen. Four things matter to me day to day. How likely the tool is to be installed already when I land on a strange box. How much config I have to write before it’s usable. Whether I can find a command without a cheat sheet. And how far I can script it. Performance barely registers; all three are light enough on anything built this decade.\nGNU Screen, the original Screen has been around since 1987. It’s installed everywhere, and it looks like it was designed in 1987.\nBasic usage # Start new session screen # Start named session screen -S development # Detach (from inside screen) Ctrl-a d # List sessions screen -ls # Reattach screen -r development # Split horizontally Ctrl-a S # Split vertically Ctrl-a | # Switch between regions Ctrl-a Tab # Create new window Ctrl-a c # Switch windows Ctrl-a n # next Ctrl-a p # previous Ctrl-a 0 # window 0 What Screen has going for it is that it’s already there. A hardened box, a vendor appliance, a rescue shell, the Solaris machine a client forgot existed. On those, Screen is often the only multiplexer present and you can’t install another one. It’s stable because nobody has touched it in years, and attach and detach is all I ever ask of it.\nThe cost shows up the moment you want more. There’s no visual feedback, the key bindings are cryptic, the splits are “regions” rather than real panes, and scrollback inside a split is a mess when you’re reading a log. For a quick detach I don’t care about any of that. As a daily driver I’d care about all of it, which is why almost nobody picks Screen for one anymore.\nSo Screen stays in my back pocket for one situation. A server I don’t control, where I can’t install software, and a process that has to outlive my connection. Everything else gets something newer.\ntmux, the standard tmux is what most people moved to once they wanted more than Screen gives. More features, a better interface, active development. It’s also the one I live in.\nBasic usage # Start new session tmux # Start named session tmux new -s development # Detach Ctrl-b d # List sessions tmux ls # Attach tmux attach -t development # Split horizontally Ctrl-b \" # Split vertically Ctrl-b % # Navigate panes Ctrl-b Arrow keys # Resize panes Ctrl-b Ctrl-Arrow keys # Create window Ctrl-b c # Switch windows Ctrl-b n # next Ctrl-b p # previous Ctrl-b 0 # window 0 # Kill pane Ctrl-b x Configuration Default tmux is awkward, and that’s the fair knock against it. Out of the box the prefix is Ctrl-b, the split bindings are \" and %, which nobody guesses, and your Screen habits fight you. tmux does pay you back for the config work. Here’s a ~/.tmux.conf that fixes the worst of the defaults:\n# Better prefix unbind C-b set -g prefix C-a bind C-a send-prefix # Start windows and panes at 1, not 0 set -g base-index 1 setw -g pane-base-index 1 # Renumber windows when one is closed set -g renumber-windows on # Easy splits (more intuitive) bind | split-window -h -c \"#{pane_current_path}\" bind - split-window -v -c \"#{pane_current_path}\" # Vim-style pane navigation bind h select-pane -L bind j select-pane -D bind k select-pane -U bind l select-pane -R # Resize panes with vim keys bind -r H resize-pane -L 5 bind -r J resize-pane -D 5 bind -r K resize-pane -U 5 bind -r L resize-pane -R 5 # Mouse support (optional, some prefer keyboard-only) set -g mouse on # Increase scrollback set -g history-limit 50000 # Don't wait for escape sequences set -sg escape-time 0 # Status bar set -g status-style 'bg=#333333 fg=#ffffff' set -g status-left '#[fg=#00ff00][#S] ' set -g status-right '%H:%M %d-%b' # Reload config bind r source-file ~/.tmux.conf \\; display \"Config reloaded\" Plugin manager (TPM) Install TPM for plugins:\n# Install TPM git clone https://github.com/tmux-plugins/tpm ~/.tmux/plugins/tpm # In .tmux.conf set -g @plugin 'tmux-plugins/tpm' set -g @plugin 'tmux-plugins/tmux-sensible' set -g @plugin 'tmux-plugins/tmux-resurrect' # Save/restore sessions set -g @plugin 'tmux-plugins/tmux-continuum' # Auto-save sessions # Initialize TPM (at bottom of .tmux.conf) run '~/.tmux/plugins/tpm/tpm' Press prefix + I to install plugins.\nSession management Save and restore sessions:\n# With tmux-resurrect plugin prefix + Ctrl-s # Save prefix + Ctrl-r # Restore # Manual session scripts # ~/.tmux/dev-session.sh tmux new-session -d -s dev -n editor tmux send-keys -t dev:editor 'nvim' C-m tmux new-window -t dev -n server tmux send-keys -t dev:server 'npm run dev' C-m tmux new-window -t dev -n git tmux attach -t dev Where tmux earns its keep is configurability and scripting. Everything is adjustable and the plugin ecosystem covers anything you’d want. The scripting is what I’d miss most. I can describe a whole working session in a shell script and recreate it, identically, on any machine. That’s the same habit I have with infrastructure, where the config is written down and reproducible rather than clicked together once and forgotten. The trade is the learning curve and the config you write up front. Nobody gets a comfortable tmux without writing the config for it.\nZellij, the modern option Zellij is the newcomer. Written in Rust, and built around the idea that you shouldn’t have to memorise anything to use it.\nBasic usage # Start new session zellij # Start named session zellij -s development # Attach to session zellij attach development # List sessions zellij list-sessions What sets Zellij apart is that it shows you the keybindings as you go. The status bar lists what’s available in the current mode, so you find your way by looking instead of by remembering. On a box without my config I still hesitate over which of \" and % splits which way, after all these years. Zellij never gave me that pause.\nDefault keybindings Zellij uses a modal system:\nCtrl-p → Pane mode n: new pane d: close pane h/j/k/l: navigate Ctrl-t → Tab mode n: new tab x: close tab h/l: navigate r: rename Ctrl-n → Resize mode h/j/k/l: resize Ctrl-s → Scroll mode j/k: scroll d/u: page down/up Ctrl-o → Session mode d: detach w: session manager Ctrl-q → Quit Every mode lists its actions in the status bar, so you never go looking for them.\nConfiguration ~/.config/zellij/config.kdl:\n// Theme theme \"catppuccin-mocha\" // Default layout default_layout \"compact\" // Keybindings keybinds { normal { // Vim-like pane navigation bind \"Alt h\" { MoveFocus \"Left\"; } bind \"Alt j\" { MoveFocus \"Down\"; } bind \"Alt k\" { MoveFocus \"Up\"; } bind \"Alt l\" { MoveFocus \"Right\"; } } } // Options pane_frames false // No borders between panes simplified_ui true // Cleaner status bar default_shell \"zsh\" scrollback_editor \"/usr/bin/nvim\" Layouts Define reusable layouts in ~/.config/zellij/layouts/:\n// dev.kdl layout { pane split_direction=\"vertical\" { pane { command \"nvim\" } pane split_direction=\"horizontal\" { pane { command \"lazygit\" } pane } } } Start with layout:\nzellij --layout dev Plugins Zellij supports WebAssembly plugins:\n// In config.kdl plugins { tab-bar { path \"tab-bar\"; } status-bar { path \"status-bar\"; } strider { path \"strider\"; } // File browser compact-bar { path \"compact-bar\"; } } Zellij’s defaults are sensible, the layouts are reusable, and floating panes for a quick throwaway command are lovely. The catches are real but small. It’s young, so the plugin ecosystem is thin next to tmux. You have to install it yourself. And if you come from tmux, your muscle memory fights you for a week. That’s the normal price of a tool that hasn’t been on every server for fifteen years yet.\nComparison Feature Screen tmux Zellij Learning curve Medium High Low Configuration Minimal Extensive Good Default UX Poor Poor Good Plugin support None Excellent Growing Availability Everywhere Most places Install required Session persistence Yes Yes (plugins) Yes Scripting Basic Excellent Good Performance Light Medium Medium Development Minimal Active Active My workflow So which one did I land on? tmux, every day. Here’s the setup behind that.\nConfig philosophy # Minimal, keyboard-first # Ctrl-a as prefix (easier than Ctrl-b) # Vim-style navigation # No mouse (keeps hands on keyboard) # Minimal status bar Session structure Development session: ├── Window 1: editor (nvim) ├── Window 2: terminal (general commands) ├── Window 3: git (lazygit) └── Window 4: servers (split panes for logs) Homelab session: ├── Window 1: k9s (Kubernetes) ├── Window 2: monitoring (Prometheus/Grafana access) └── Window 3: logs (Loki queries) Startup script #!/bin/bash # ~/.local/bin/dev SESSION=\"dev\" # Check if session exists tmux has-session -t $SESSION 2\u003e/dev/null if [ $? != 0 ]; then # Create session tmux new-session -d -s $SESSION -n editor tmux send-keys -t $SESSION:editor 'nvim' C-m tmux new-window -t $SESSION -n term tmux new-window -t $SESSION -n git tmux send-keys -t $SESSION:git 'lazygit' C-m tmux new-window -t $SESSION -n servers tmux split-window -h -t $SESSION:servers fi # Attach tmux attach -t $SESSION Why tmux and not Zellij The answer is mostly path dependence. Years of tmux muscle memory, a plugin setup (resurrect and continuum) I’d have to rebuild, scripted sessions I lean on, and tmux being on every server I touch. Switching would cost me more than it pays back.\nThat’s a statement about my situation, and I’d hate for it to be read as a verdict on Zellij. Starting from zero today, with no muscle memory to protect, I’d give Zellij a serious look. Reading commands off the status bar beats remembering them, and I’ve watched people new to multiplexers get productive in it in an afternoon. If your context has less tmux in it than mine, the answer may well flip.\nRecommendations When Screen is the right pick Working on servers you don’t control Need something that’s definitely installed Simple attach/detach for long processes When tmux is the right pick You want maximum customization You need plugin support You’ll invest time in configuration You work across many machines When Zellij is the right pick You’re new to multiplexers You want good UX immediately You prefer modern tooling You value discoverability Getting started Week 1, the basics Pick one. Learn these operations:\nCreate/attach/detach session Split panes Navigate between panes Create windows Navigate between windows Week 2, make it yours Change prefix key (tmux/Screen) Set up comfortable keybindings Configure status bar Set scrollback size Week 3, wire it into the workflow Create session scripts Set up session persistence Integrate with editor/shell Why this matters Keeping a process alive through a dropped connection is the obvious win, and it’s the one I opened with. The reason a multiplexer stays on every machine I own is that it changes how the work feels. Each project gets its own session, so the context lives somewhere persistent instead of in my head. Windows and panes become places. After a while my hands know that logs are bottom-right and the editor is window one, and my brain isn’t consulted. Nothing pulls me out of flow to go hunt for a window.\nPair a multiplexer with an editor and a shell you’ve tuned and you get the same working environment on your laptop and on every server you SSH into. That portability is the whole point for me. I’m not helpless on a fresh box, because the same key bindings work everywhere, and the config travels in a git repo I control rather than sitting in some GUI’s settings.\nThe terminal is already low friction computing, and a multiplexer takes more of the friction out. The next time a tunnel eats my connection, I’ll reattach on the other side and pick up the migration where it was.","permalink":"https://tommeurs.nl/en/posts/terminal-multiplexing/","title":"Terminal Multiplexing: tmux vs Zellij vs Screen"},{"content":" TL;DR Tailscale puts every device I own on one WireGuard mesh, so I reach the cluster in the closet from a coffee shop without a single port forwarded on the router. Subnet router, exit node, the Kubernetes operator, ACLs, and Headscale for the part I’d rather host myself, in that order.\nMy homelab cluster runs in a closet at home. I don’t. I work from coffee shops, from a client’s office, and now and then from a hotel with WiFi that seems to resent me. I still want my own machines while I’m out there.\nFor years the standard answer was to poke a hole in your own front door. Forward a port on the router, set up dynamic DNS so the changing home IP doesn’t break everything, write the firewall rules, and hope nobody scanning the internet finds the SSH daemon you’ve just exposed. It works, you can reach your stuff. It also means part of your home network answers questions from strangers around the clock.\nWhat I wanted was to reach the homelab from anywhere with nothing open to the internet. Tailscale does that for me, and this post is how I set it up, from one node to the whole cluster.\nWhat Tailscale is Tailscale is a mesh VPN built on WireGuard. Every device you enrol gets a stable address in the 100.x.y.z range, and every device talks directly to every other one. There’s no central box the traffic has to pass through, which is the difference from the VPN concentrator you may remember from an office job.\nTailscale Network (100.x.y.z)\nLaptop\n100.64.0.1\nPhone\n100.64.0.2\nHomelab\n100.64.0.3\nVPS\n100.64.0.4\nWhen the network cooperates, two devices connect straight to each other. When NAT gets in the way, and it often does, the traffic falls back to a relay through Tailscale’s DERP servers. You don’t manage any of that. Both paths are encrypted end to end, so even the relay can’t read what passes through it.\nOne trade-off before I go further. The coordination plane is Tailscale’s, not yours. They run the control servers that hand out keys and tell devices where to find each other. Your traffic stays peer to peer and encrypted, but the brokering depends on a company you don’t host. For a blog built on sovereignty that matters, and there’s a fix. I’ll get to it near the end.\nWhy I use it anyway What sold me is what disappeared from my router config. Nothing is forwarded, nothing is exposed, and there’s no port for a scanner to find. The service I used to keep hardened against the whole internet doesn’t exist any more.\nIt also gets through network setups that would otherwise be a dead end. CGNAT from an ISP that won’t give you a real public IP, double NAT, hotel WiFi that blocks half the internet. Both ends reach out instead of waiting to be reached, so the mess of routers in between stops mattering. My homelab is just there, on my tailnet.\nThen the smaller things. Each device keeps the same 100.x.y.z address, so I’m not chasing dynamic DNS records that lag behind reality. MagicDNS gives every machine a name like homelab.tail-abc123.ts.net, so I’ve stopped memorising IPs. And ACLs let me say who reaches what, so a borrowed laptop doesn’t automatically get the keys to the cluster.\nGetting it running Start with the homelab node itself.\n# Ubuntu/Debian curl -fsSL https://tailscale.com/install.sh | sh # Start Tailscale sudo tailscale up # Check status tailscale status tailscale up prints a login link. Open it, log in, and the node joins your tailnet. Then put the client on whatever you carry around:\nYour laptop Your phone Anything else that needs in Every device authenticates against the same Tailscale account, and once they have, they see each other. Check it from your laptop:\n# From laptop ping homelab.tail-abc123.ts.net # SSH to homelab ssh user@homelab.tail-abc123.ts.net # Access services curl http://homelab.tail-abc123.ts.net:3000 # Grafana No port forwarding, no firewall rules. The first time I SSH’d into a machine at home from a train, having opened nothing, it felt slightly illegal.\nReaching the devices that can’t run Tailscale Plenty of things on a home network will never run a Tailscale client. The NAS firmware doesn’t allow it, and the printer is a printer. A subnet router fixes that by turning one enrolled node into a doorway to the rest of the LAN.\n# On homelab node sudo tailscale up --advertise-routes=192.168.1.0/24 # Accept the route in Tailscale admin console # Or use --accept-routes on clients Once you accept the advertised route, your laptop reaches the NAS on 192.168.1.50 by routing through the homelab node, and the printer the same way.\nHome Network\nRemote (Coffee Shop)\nTailscale\nLocal\nLocal\nLaptop\n100.64.0.1\nHomelab\n100.64.0.3\n192.168.1.10\nNAS\n192.168.1.50\nPrinter\n192.168.1.60\nSending all traffic home There’s a related trick for when you don’t trust the network around you at all. Turn a home node into an exit node and your laptop routes everything through it, so your traffic leaves from your home IP instead of the café’s.\n# On homelab node sudo tailscale up --advertise-exit-node # Accept in admin console # On laptop (when needed) sudo tailscale up --exit-node=homelab I flip this on when I’m on WiFi I don’t control, which gets me out from under whatever that network is logging. It also works for reaching geo-restricted things as if you were home, and for using your home Pi-hole wherever you are.\nPutting cluster services on the tailnet This is where it gets nice for a Kubernetes homelab. The Tailscale Kubernetes Operator publishes services straight onto your tailnet, so they’re reachable by name without an ingress facing the internet.\n# Install operator helm repo add tailscale https://pkgs.tailscale.com/helmcharts helm repo update helm install tailscale-operator tailscale/tailscale-operator \\ --namespace tailscale \\ --create-namespace \\ --set oauth.clientId=\"\u003cclient-id\u003e\" \\ --set oauth.clientSecret=\"\u003cclient-secret\u003e\" Create the OAuth client in the Tailscale admin console first, then hand those credentials to the install. With the operator running, exposing a service is one annotation:\napiVersion: v1 kind: Service metadata: name: grafana annotations: tailscale.com/expose: \"true\" spec: selector: app: grafana ports: - port: 3000 Grafana is now reachable at grafana.tail-abc123.ts.net:3000 from any device on the tailnet, and from nowhere else. If you’d rather treat Tailscale as a proper ingress controller, that works too:\napiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: grafana spec: ingressClassName: tailscale rules: - host: grafana http: paths: - path: / pathType: Prefix backend: service: name: grafana port: number: 3000 That gives you grafana.tail-abc123.ts.net with HTTPS handled for you.\nDeciding who can reach what Enrolling devices is the easy half. By default everything on your tailnet talks to everything else, and that’s fine until you add a device you trust less. ACLs tighten it.\n{ \"acls\": [ // Admin can access everything { \"action\": \"accept\", \"src\": [\"group:admin\"], \"dst\": [\"*:*\"] }, // Developers can access cluster services { \"action\": \"accept\", \"src\": [\"group:dev\"], \"dst\": [\"tag:k8s:80,443,6443\"] }, // Everyone can access monitoring { \"action\": \"accept\", \"src\": [\"*\"], \"dst\": [\"tag:monitoring:3000,9090\"] } ], \"groups\": { \"group:admin\": [\"user@example.com\"], \"group:dev\": [\"dev1@example.com\", \"dev2@example.com\"] }, \"tagOwners\": { \"tag:k8s\": [\"group:admin\"], \"tag:monitoring\": [\"group:admin\"] } } You tag devices in the admin console, and the ACL decides what each tag reaches. A dev box gets the cluster API and the web ports, the monitoring stack is readable by anyone on the tailnet, and admin gets the rest. It’s the same reason I write network policies inside the cluster, a smaller blast radius when something does go wrong.\nNames and certificates MagicDNS already gives every device a device-name.tail-abc123.ts.net name. In the admin console you can turn it on and swap the random tailnet string for something you’ll remember.\nFor TLS, Tailscale issues real certificates for tailnet domains:\ntailscale cert grafana.tail-abc123.ts.net That’s valid HTTPS for an internal service without dragging Let’s Encrypt or cert-manager into it. If you want your own domain in front, point a CNAME at the tailnet name and tell Tailscale to accept it:\n# In your DNS grafana.internal.example.com CNAME grafana.tail-abc123.ts.net When you do need something public Now and then you want one thing reachable from the open internet. A webhook endpoint, a link to hand someone for an hour, a quick demo. Tailscale Funnel gives you a public URL for that without touching your router.\n# Expose local port 3000 publicly tailscale funnel 3000 You get back something like https://homelab.tail-abc123.ts.net/. Funnel sits outside your ACLs, so whatever you funnel is public in the plain sense of the word. Treat it like a forwarded port, because that’s what it is, and only funnel things you’re happy for the world to poke at.\nOwning the control plane Back to the trade-off from the start. What nags at me about Tailscale’s coordination servers is what nags at me about any black box. If they’re down, or the company changes course, my access changes with it. For a homelab built on owning what you run, handing the key-and-discovery layer to someone else is a real concession.\nHeadscale is the answer. It’s an open-source reimplementation of the Tailscale control server that you host yourself, and the standard Tailscale clients talk to it without complaint.\n# headscale deployment apiVersion: apps/v1 kind: Deployment metadata: name: headscale spec: replicas: 1 selector: matchLabels: app: headscale template: spec: containers: - name: headscale image: headscale/headscale:latest args: - serve ports: - containerPort: 8080 volumeMounts: - name: config mountPath: /etc/headscale - name: data mountPath: /var/lib/headscale Same WireGuard underneath, same client experience, except the coordination now runs on something you control. The cost is that you run and update it yourself. In return, the mesh keeps working on the day Tailscale the company doesn’t.\nWiring it into the rest of your tools A few integrations earn their keep every day. Tailscale can handle SSH auth, so you stop juggling keys for hosts that are already on your tailnet:\n# Enable Tailscale SSH on server tailscale up --ssh # Connect (no keys needed!) ssh user@homelab.tail-abc123.ts.net Tailscale authenticates the connection with your tailnet identity, so there’s no key to copy around. Cluster access works the same way once the API server is on the tailnet:\n# ~/.kube/config apiVersion: v1 clusters: - cluster: server: https://k3s.tail-abc123.ts.net:6443 name: homelab contexts: - context: cluster: homelab user: admin name: homelab current-context: homelab kubectl from a hotel room hits the cluster as if you were sitting next to it. And for editing on the box itself, one SSH config entry is enough for VS Code Remote to connect:\n// .ssh/config Host homelab HostName homelab.tail-abc123.ts.net User youruser What my setup looks like Nothing exotic. One node does the heavy lifting and my handful of devices connect to it.\nMy Tailnet\nDevices\nHome\nLaptop\nK3s Cluster\nSubnet Router\nExit Node\nPhone\nTablet\nSynology NAS\nThe K3s node is the subnet router for 192.168.1.0/24, so the NAS and everything else on the LAN are reachable. It’s also my exit node for the days I’m on WiFi I’d rather not trust. The Tailscale Operator publishes Grafana, ArgoCD and Forgejo onto the tailnet, and an ACL keeps the whole thing scoped to my own devices.\nI check monitoring from anywhere. I’ve kicked off an ArgoCD sync from my phone on a train. And the router has never had a single port forwarded to make any of it work.\nWhen it misbehaves A few things that have tripped me up.\n“Tailscale is blocked.” Some corporate networks block it outright. Move it to a different DERP port, or run it over TCP/443 so it looks like ordinary HTTPS and gets through.\n“The connection is slow.” Usually the direct path failed and you’re going through a DERP relay. Check whether a firewall eats the UDP traffic Tailscale uses for direct connections, and whether your NAT type forces relaying.\n“I can’t reach the subnet.” Subnet routes have to be accepted, not just advertised. Advertise on the router node with --advertise-routes, accept the route in the admin console, and turn on --accept-routes on the clients that need it.\nWhy this sticks Instead of opening the network and then defending the opening, I open nothing. Reaching anything means authenticating onto the mesh first, encryption is on whether I think about it or not, and access is decided in one ACL file I can read in a minute.\nSo the cluster stays in the closet and I keep working from wherever I am, and from my side it behaves as if I never left the house. With Headscale in the mix, not even the connection brokering runs on somebody else’s servers. The router still has no ports forwarded, and it’s staying that way.","permalink":"https://tommeurs.nl/en/posts/tailscale-homelab/","title":"Tailscale for Homelab: Secure Remote Access Without Port Forwarding"},{"content":" TL;DR An Internal Developer Platform is a thin layer between what a developer wants and what the cluster demands. Mine is a Helm base chart, ArgoCD and a handful of Kyverno policies, about 500 lines of my own code. Start with those three, and add observability and a portal only when the tickets tell you to.\nShould you build an Internal Developer Platform? Usually, yes. It’s the how that wrecks teams.\nI’ve watched a platform that cost a small fortune go live and then sit unused, because nobody wanted it. I’ve also seen a couple of Helm charts and one Kyverno policy change how a whole team ships software. Budget didn’t separate those two. Neither did the choice of tools. One team started from a problem developers had. The other built the platform it imagined developers should want.\nSo instead of a reference architecture to copy, I’ll build this up from the smallest thing worth deploying. Stop reading at any point and you should still have something you can ship.\nWhat an IDP is An Internal Developer Platform is a self-service layer that keeps the infrastructure plumbing away from the people building applications. Developers describe what they need. The platform works out how to make it real.\nWith Platform\nDeveloper\nPlatform API\nKubernetes\nCI/CD\nSecurity\nMonitoring\nWithout Platform\nDeveloper\nKubernetes YAML\nCI Pipeline\nSecurity Config\nMonitoring Setup\nThe platform sits between what a developer wants and what the cluster requires. Everything in this post is about keeping that layer thin, and easy to get out of on the day it’s in the way.\nWhy so many platforms die The failure modes first, then the build steps. Most of them are predictable and you can design around them.\nThe classic one is building in isolation. The platform team shuts itself in a room, ships what it thinks people need, and gets annoyed when nobody adopts it. The developers weren’t being difficult. They got handed a tool for problems they didn’t have.\nClose behind is doing too much too soon. A service catalog, a custom UI, self-service for every conceivable resource, all planned up front. Six months later there’s an impressive demo and nothing in production.\nThen there’s the wrong altitude. Too low and the platform is Kubernetes with extra steps, so why bother. Too high and there’s no escape hatch on the day someone hits a case the abstraction never anticipated. Someone always does.\nAnd the one that has sunk more platform efforts than any technical mistake. The platform only works for greenfield, existing services can’t move onto it, and now you run two systems forever.\nStart with problems, not a solution This is the part people skip. Before you write a single template, go find out where the pain is.\nWhere do developers lose time? Waiting on infrastructure requests, copy-pasting the same boilerplate config, debugging deployments that fail for reasons nobody documented, hunting down production access. What do they keep asking for? Your ticketing system knows. Your Slack channels know. And, revolutionary idea, you can ask them.\nThen look at where incidents come from. Misconfigured services, missing network policies, resource limits nobody set. Automate those away first, because every one of them is a page you stop getting.\nThe answers say what to build. Build anything before you have them and you’re guessing.\nThe simplest useful platform Here’s the minimum I’d bother deploying. Three pieces, all from tools you already run.\nGolden paths Opinionated templates that bake in the right defaults so nobody has to remember them.\n# service-template/ ├── deployment.yaml # Standard deployment pattern ├── service.yaml # Service with sensible defaults ├── networkpolicy.yaml # Security by default ├── servicemonitor.yaml # Automatic monitoring └── values.yaml # Customization points A developer gets a working, monitored service with a network policy out of the box, and only touches the values that make their service different. The safe configuration is also the one with the least typing, and that’s why people use it.\nSelf-service deployment Push code, get a deployment. No ticket, no waiting on someone in another timezone.\nWith GitOps:\n# Application definition in Git apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: my-service namespace: argocd spec: source: repoURL: https://gitlab.internal/my-service path: deploy targetRevision: main destination: server: https://kubernetes.default.svc namespace: my-service syncPolicy: automated: prune: true selfHeal: true Merge to main, ArgoCD picks it up, the service ships. You don’t need a portal for this to count as self-service. Git already is one.\nGuardrails The flip side of self-service is that people deploy things that hurt them. Kyverno policies catch the obvious mistakes before they land:\n# Require resource limits apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: require-limits spec: validationFailureAction: Enforce rules: - name: require-limits match: resources: kinds: - Pod validate: message: \"CPU and memory limits are required\" pattern: spec: containers: - resources: limits: cpu: \"?*\" memory: \"?*\" A pod without limits never reaches the cluster. Nobody has to remember the rule, the admission controller remembers it for them.\nThat’s a real platform. Templates, GitOps, guardrails. Ship only this and developers deploy safely without learning the Kubernetes internals, and the most common tickets are gone. You could stop here for a year and be fine.\nThe building blocks Once the basics stick, you flesh out the pieces. None of this is exotic. It’s mostly configuration of tools you already have.\nService templates with Helm A base chart teams extend instead of forking:\n# base-service/Chart.yaml apiVersion: v2 name: base-service version: 1.0.0 description: Standard service template # base-service/templates/deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: {{ .Values.name }} spec: replicas: {{ .Values.replicas | default 2 }} selector: matchLabels: app: {{ .Values.name }} template: metadata: labels: app: {{ .Values.name }} annotations: prometheus.io/scrape: \"true\" spec: containers: - name: {{ .Values.name }} image: {{ .Values.image }} ports: - containerPort: {{ .Values.port | default 8080 }} resources: requests: cpu: {{ .Values.resources.requests.cpu | default \"100m\" }} memory: {{ .Values.resources.requests.memory | default \"128Mi\" }} limits: cpu: {{ .Values.resources.limits.cpu | default \"500m\" }} memory: {{ .Values.resources.limits.memory | default \"512Mi\" }} readinessProbe: httpGet: path: {{ .Values.healthPath | default \"/health\" }} port: {{ .Values.port | default 8080 }} A team consumes it with a small values file:\n# my-service/values.yaml name: my-service image: registry.internal/my-service:v1.2.3 replicas: 3 port: 8080 resources: requests: cpu: 200m memory: 256Mi limits: cpu: 1 memory: 1Gi Fix a bug in the base chart and every service picks up the fix on its next deploy. That’s what the base chart is for.\nThe namespace as a boundary Give each team a namespace that arrives with its guardrails in place:\napiVersion: v1 kind: Namespace metadata: name: team-payments labels: team: payments environment: production --- apiVersion: v1 kind: ResourceQuota metadata: name: team-quota namespace: team-payments spec: hard: requests.cpu: \"10\" requests.memory: 20Gi limits.cpu: \"20\" limits.memory: 40Gi pods: \"50\" --- apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny namespace: team-payments spec: podSelector: {} policyTypes: - Ingress - Egress Inside the namespace the team decides. The quota and the default-deny policy belong to the platform. A runaway workload in payments can’t eat the cluster, and traffic is closed until someone deliberately opens it.\nSelf-service through pull requests Teams change their own config the same way they change code:\ninfrastructure/ ├── teams/ │ ├── payments/ │ │ ├── namespace.yaml │ │ ├── applications/ │ │ │ ├── api.yaml │ │ │ └── worker.yaml │ │ └── secrets/ │ │ └── external-secret.yaml │ ├── orders/ │ └── ... A pull request triggers review, the merge triggers an ArgoCD sync, and the change is live. Every infrastructure change has an author, a reviewer and a diff. That’s an audit trail, and nobody had to build one.\nGrowing the platform You don’t bolt all of this on at once. Each phase is a place you can stop and live for a while.\nPhase one is what I just walked through. Helm charts for the common patterns, Kyverno for safety, GitOps for deployment, and enough docs to find your way around. People deploy safely without a deep Kubernetes background.\nPhase two is observability nobody wires up by hand:\nAutomatic Prometheus scraping Loki log aggregation Tempo trace collection Pre-built Grafana dashboards A service deployed through the platform shows up in dashboards with no extra config. When it breaks, the data is already there.\nPhase three is when a UI starts to earn its keep, whether that’s Backstage or something you build yourself:\nDeveloper Portal\nService Catalog\nCreate from Template\nDocumentation\nService Status\nGit Repository\nArgoCD\nKubernetes\nThe portal doesn’t replace any of the earlier layers. It generates Git commits that flow through the same ArgoCD you already trust. Same machinery, friendlier front door.\nPhase four is the deeper self-service most teams get to eventually. Secret management with Vault, database provisioning, environment cloning, cost visibility. By now the platform handles the long tail of requests that used to land in your queue.\nWhat not to build The refusals matter as much as the builds. Four I’d make every time.\nDon’t invent your own YAML DSL. Someone always wants to, and the result is a configuration language only your team understands, with worse tooling than Helm, Kustomize or cdk8s give you for free. Say no.\nIf developers still raise tickets to get things done, the automation isn’t finished. Each kind of ticket is a workflow you haven’t built yet.\nDon’t make one size fit all. Golden paths are defaults. The team that knows what it’s doing should be able to drop down to raw manifests without asking permission.\nAnd keep the platform boring on purpose. A platform that ships new features every week usually hasn’t found its real problems yet. I’d rather run one nobody talks about.\nKnowing if it’s working You’ll want to know whether any of this paid off. A few signals I watch.\nTime to production first. How long from “new service idea” to “running in prod”? Weeks means the platform isn’t doing its job. Days is fine.\nTicket volume should trend down as the platform absorbs the routine work:\nBefore: 50 infrastructure tickets/week After: 10 infrastructure tickets/week (edge cases only) Ask developers a blunt question now and then. Would you recommend this platform to a colleague? That answer beats any dashboard. And check whether services deployed through the platform have fewer incidents than the hand-rolled ones. If the guardrails work, they should.\nThe team behind it A platform is a product with users, so it needs people who own it. A rough rule is one platform engineer for every ten to fifteen application developers. Too few and the platform stops evolving. Too many and they start inventing features nobody asked for, which is how the six-month demo happens.\nThe split of responsibility matters. The platform team maintains the templates and tools, writes the policies, handles platform-level incidents, and helps teams migrate onto the thing. Application teams own their services end to end. They deploy their own code, define their own resource needs, and take the first page when their service breaks.\nStart centralised so the platform has a clear owner. As it matures, embed platform engineers part-time in product teams. Sitting next to the people who live with the platform is how you find out what hurts.\nWhat this looks like in practice My homelab platform runs on the pieces above.\nThe golden path is a Helm dependency:\n# Standard service template dependencies: - base-service # Helm dependency values: name: my-app image: registry/my-app:latest Self-service is GitOps. Push to main and ArgoCD deploys, open a PR for a namespace change and it auto-merges after review. Guardrails are Kyverno enforcing limits, labels and security context, plus a default-deny NetworkPolicy. Observability comes with the template, a ServiceMonitor created automatically and a Grafana dashboard generated alongside it.\nCustom code, all in, is somewhere around 500 lines of Helm templates and Kyverno policies. Everything else is configuration of tools that already existed. That ratio matters to me more than any feature. The less bespoke code the platform carries, the less of it I have to understand when it breaks, and I want to be able to read my own platform top to bottom.\nThe mistakes I see most A few patterns show up again and again.\nStarting with the portal is the most common. The UI is the fun part, so people build it first, before the workflows underneath are proven. Get GitOps and the CLI working, then put a face on it.\nIgnoring existing services is the expensive one. A platform that only handles greenfield leaves you running two worlds in parallel forever. Migration is a feature, and it needs to be in the first version.\nOver-engineering security early strangles adoption. Block every edge case on day one and people route around the platform entirely. Start permissive and tighten based on what incidents teach you.\nHiding all the complexity sounds nice until a senior developer needs to see what’s underneath and can’t. Some people want to see the machinery, and that instinct is healthy. Give them escape hatches and real documentation.\nWhy bother Developer time is the most expensive thing on the team, and every hour spent fighting infrastructure is an hour not spent on the product. With a platform that works, a junior can deploy safely on day one, and the senior who used to answer their tickets gets those hours back.\nMine started as one base chart and one Kyverno policy. 500 lines later, it’s still mostly that.","permalink":"https://tommeurs.nl/en/posts/internal-developer-platform/","title":"Building an Internal Developer Platform: Where to Start"},{"content":" TL;DR Each dashboard answers one question, and within five seconds I know whether I need to care. Four kinds of dashboard cover everything, variables and links tie them together, and the JSON lives in Git. My homelab runs five, and anything that goes a month unopened gets deleted.\nYou have Grafana. You have Prometheus metrics. You have logs in Loki and traces in Tempo. The data’s all there.\nYou also have 47 dashboards nobody opens.\nI’ve done this to myself more than once. Something breaks, I bolt a dashboard together to see what’s going on, and it sits there forever. A year of incidents and a few “let me add a panel for that” moments later, most of my Grafana was archaeology. Nobody remembered what half the panels meant, me included. Most of it should go. But before you delete, it helps to know what the survivors have that the rest don’t.\nThe dashboards I still open have one thing in common. I open one and know within five seconds whether I need to care. That’s the whole bar, and this post is how I get there, from the smallest dashboard worth keeping up to the five I run on my homelab cluster.\nThere’s a reason I keep coming back to this. A pile of metrics you can’t read is a black box, and this one you built yourself. Observability is supposed to buy understanding. A dashboard nobody can parse buys nothing.\nWhere dashboards go wrong The failure modes first. I’ve shipped every one of these.\nToo many panels. Thirty panels, thirty metrics, no answers, because nobody reads thirty graphs at a glance. They compete for attention and none of them wins.\nWrong abstraction level. CPU, memory and disk for every individual pod is what you want when you’re elbow-deep in a problem. It’s useless when you want to know if the service is up. Those are different questions, and they want different dashboards.\nNo context. “500 requests/second.” Good? Bad? Normal? Without something to compare it against, the number is decoration.\nBuilt and forgotten. The incident dashboard from two years ago, still querying services that got renamed in a refactor nobody told it about. Flat lines, and everyone’s learned to look past it.\nOne dashboard, one question The rule that fixes most of this is boring. Each dashboard answers exactly one question.\nQuestion Dashboard “Is the system healthy?” Service overview “Why is this service slow?” Service deep dive “What happened at 3 AM?” Incident investigation “Are we meeting SLOs ?” SLO dashboard Keep those apart. The moment you bolt a debugging panel onto the health-check dashboard, you’ve started the slide back to 47.\nThe simplest useful dashboard is “is everything OK?”. A handful of stat panels, big and colour-coded, red when something’s wrong. If that’s the only dashboard you ever build, you’re ahead of most teams I’ve worked with.\nLink down instead of cramming it all in Once the one-glance dashboard works, the next question is “OK, which service?”. Don’t answer that on the same screen. Link to a deeper one. The model I use:\nLevel 1: Is everything OK? (green/yellow/red) ↓ Level 2: Which service has problems? ↓ Level 3: What specific metric is wrong? ↓ Level 4: Raw data for debugging Each level is its own dashboard or section, wired together with links. You stop at whichever level answers your question. Most days that’s level 1.\nGive every number something to compare against A raw number tells you almost nothing. The same value is fine on a Tuesday afternoon and an incident in the middle of the night. So I add context wherever I can.\nThresholds. The panel changes colour when a metric crosses an SLO boundary, so you don’t do the arithmetic in your head. Comparisons. Today against last week, this deploy against the previous one. Annotations. Deployments, incidents and config changes drawn straight onto the graph. The annotations have saved me more debugging time than anything else on that list. Latency spikes, there’s a deployment marker right under the spike, done.\nReadable in five seconds If I can’t read a dashboard in five seconds it’s too complex, and I’ll stop opening it, which defeats the point. So I stick to three panel types.\nStat panels for current state, big numbers, colour-coded Time series for trends, so you can see when something started Heatmaps for distributions like latency percentiles And I skip the things that look informative and aren’t. Tables with 50 rows, graphs with 20 overlapping lines, pie charts. Especially pie charts.\nThe dashboards you need With the rules in place, these are the dashboards that get a permanent spot. Four kinds. I’ve never needed a fifth kind, only more copies of these four.\nService overview The “is everything OK?” dashboard, one per service or service group. This is the one you’ll open most.\n# Panels (top to bottom, left to right) Row 1: Status (stat panels) - Availability (% uptime last 24h) - Error Rate (current, color-coded) - P99 Latency (current, color-coded) - Active Instances (count) Row 2: Trends (time series) - Request Rate (stacked by status code) - Latency (P50, P95, P99 lines) - Error Rate (percentage over time) Row 3: Resources (gauges) - CPU Usage (% of limit) - Memory Usage (% of limit) - Pod Count (current vs desired) The queries behind it:\n# Availability (percentage of successful requests) sum(rate(http_requests_total{status!~\"5..\"}[5m])) / sum(rate(http_requests_total[5m])) * 100 # Error rate sum(rate(http_requests_total{status=~\"5..\"}[5m])) / sum(rate(http_requests_total[5m])) * 100 # P99 latency histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le) ) SLO dashboard This one tracks Service Level Objectives and tells you whether you’re burning your error budget faster than you can afford. It’s the backbone of meaningful alerting, because you want to page someone when the budget is burning, and you don’t want to page anyone for a single request that hiccups.\nRow 1: SLO Status (stat panels) - Availability SLO (99.9% target) - Latency SLO (P99 \u003c 500ms target) - Error Budget Remaining (%) Row 2: Burn Rate (time series) - Error budget consumption rate - Projected budget exhaustion Row 3: SLI Breakdown (time series) - Availability over time - Latency percentiles over time The error budget query:\n# Error budget remaining (for 99.9% SLO over 30 days) 1 - ( (1 - (sum(rate(http_requests_total{status!~\"5..\"}[30d])) / sum(rate(http_requests_total[30d])))) / 0.001 # 0.1% error budget ) Kubernetes cluster dashboard A bird’s-eye view of cluster health. Resist the urge to put per-pod detail here. That belongs in the debugging dashboard, and mixing the two is how you end up back at thirty panels.\nRow 1: Cluster Status - Nodes Ready / Total - Pods Running / Scheduled - PVCs Bound / Total Row 2: Resource Utilization - CPU: Used / Requested / Allocatable - Memory: Used / Requested / Allocatable Row 3: Workload Health - Deployments Available / Total - StatefulSets Ready / Total - DaemonSets Ready / Total Row 4: Recent Events (table) - Warning events from last hour Debugging dashboard The one place where complexity is allowed. When you’re investigating an incident you want detail, variables, and the ability to slice by namespace and pod. The five-second rule doesn’t apply when you’re already neck-deep.\nRow 1: Service Selection (variables) - Namespace dropdown - Service dropdown - Pod dropdown Row 2: Request Flow - Incoming requests by endpoint - Outgoing requests by destination - Error breakdown by type Row 3: Resource Details - CPU per container (time series) - Memory per container (time series) - Network I/O (time series) Row 4: Logs Integration - Link to Loki with pre-filtered query - Recent error log count Panels worth stealing The individual panels I reach for again and again. Copy them, point them at your own metrics, done.\nRequest rate with status codes sum by (status_code) ( rate(http_requests_total{service=\"$service\"}[5m]) ) Stack the series and colour them by status. 2xx green, 3xx blue, 4xx yellow, 5xx red. When the graph goes red you don’t need the legend.\nLatency heatmap sum(rate(http_request_duration_seconds_bucket{service=\"$service\"}[5m])) by (le) Render it as a heatmap. You get the full latency distribution over time, and the hot spots show up where an average would have hidden them. A p99 spike that a line graph smooths away is a bright band here.\nResource usage vs limits # CPU usage percentage of limit sum(rate(container_cpu_usage_seconds_total{pod=~\"$pod\"}[5m])) / sum(kube_pod_container_resource_limits{resource=\"cpu\", pod=~\"$pod\"}) * 100 A gauge with thresholds at 0-70 green, 70-90 yellow, 90-100 red. Usage as a percentage of the limit rather than raw cores, so the same panel works for every service without retuning.\nPod restart indicator sum(increase(kube_pod_container_status_restarts_total{namespace=\"$namespace\"}[1h])) by (pod) A stat panel that only appears when the count is above zero, linked straight to the pod logs. A pod quietly restarting every few minutes is the kind of slow bleed I want to see before it turns into an outage.\nVariables, one dashboard for every service Variables are what stop you cloning the same dashboard fifty times. Define them once and the dashboard re-targets itself from a dropdown. These three cover most of what I need:\n# Namespace variable name: namespace query: label_values(kube_pod_info, namespace) multi: true include_all: true # Service variable (filtered by namespace) name: service query: label_values(kube_pod_info{namespace=~\"$namespace\"}, pod) multi: false # Time comparison name: comparison options: - 1h ago - 1d ago - 1w ago Then reference them in the queries:\n# Current value sum(rate(http_requests_total{namespace=\"$namespace\"}[5m])) # Comparison value (offset by selected time) sum(rate(http_requests_total{namespace=\"$namespace\"}[5m] offset $comparison)) That offset $comparison trick gives you a “vs last week” overlay almost for free. It’s the context rule from earlier, in one query.\nAnnotations, so you stop guessing what changed Annotations mark events directly on the graphs. Two sources are worth setting up:\n# Deployment annotations name: Deployments datasource: Prometheus query: changes(kube_deployment_status_observed_generation{deployment=\"$service\"}[5m]) \u003e 0 tags: deployment # Alert annotations name: Alerts datasource: Alertmanager filter: service=\"$service\" tags: alert Now a latency spike with a deployment marker under it answers “what changed?” before you’ve opened a terminal. Half my incidents turn out to be “we deployed something”, and the marker tells me that straight away.\nWiring dashboards together Links between dashboards are what make the levels work. The one-glance view hands you to the detail view, and you never hunt through folders.\nData links On a service panel, link straight to that service’s detail dashboard:\nURL: /d/service-detail?var-service=${__field.labels.service} Title: View ${__field.labels.service} details Panel links On the cluster overview, link down into the namespace views:\nURL: /d/namespace-view?var-namespace=${namespace} Title: Drill down to namespace Explore links And link to Loki , so going from a metric to the matching logs is one click:\nURL: /explore?left=[\"now-1h\",\"now\",\"Loki\",{\"expr\":\"{namespace=\\\"${namespace}\\\",pod=~\\\"${pod}\\\"}\"}] Title: View logs in Explore Keeping it organised Past a handful of dashboards, structure stops being optional. A little discipline here is the difference between finding the right dashboard in two seconds and giving up and querying Prometheus by hand.\nFolder structure ├── Overview │ ├── Platform Health │ └── SLO Summary ├── Services │ ├── API Gateway │ ├── Auth Service │ └── ... ├── Infrastructure │ ├── Kubernetes Cluster │ ├── Nodes │ └── Storage ├── Debugging │ ├── Service Debug │ └── Incident Investigation └── Archive └── (old dashboards, hidden) Naming convention [Category] - [Subject] - [Type] Examples: - Services - API Gateway - Overview - Services - API Gateway - Debug - Infrastructure - Kubernetes - Nodes - SLO - Platform - 30d Starring and the home dashboard Set the “is everything OK?” dashboard as the Grafana home, so it’s the first thing you see after logging in, and star the few you open daily. The faster the important ones are to reach, the less you fall back on old habits.\nDashboards as code Clicking dashboards together in the UI is fine until someone fat-fingers a panel, or the Grafana pod gets rescheduled and you find out the dashboard only lived in its memory. The fix is the same one I apply to everything else. Put it in Git and let ArgoCD deploy it. If I can’t rebuild a dashboard from source, I don’t own it.\nGrafana dashboard ConfigMap apiVersion: v1 kind: ConfigMap metadata: name: service-overview-dashboard labels: grafana_dashboard: \"1\" data: service-overview.json: | { \"title\": \"Service Overview\", \"uid\": \"service-overview\", \"panels\": [ ... ] } Drop in a ConfigMap with the grafana_dashboard: \"1\" label and the sidecar picks it up. The dashboard JSON now lives next to the service it monitors, and it gets reviewed in the same merge request that ships the service.\nGrafonnet (Jsonnet library) Hand-editing dashboard JSON gets old fast. Grafonnet generates it from code, so a shared panel is a function you call instead of JSON you copy and then forget to keep in sync:\nlocal grafana = import 'grafonnet/grafana.libsonnet'; local dashboard = grafana.dashboard; local prometheus = grafana.prometheus; dashboard.new( 'Service Overview', uid='service-overview', tags=['service', 'overview'], ) .addPanel( grafana.statPanel.new( 'Error Rate', datasource='Prometheus', ).addTarget( prometheus.target( 'sum(rate(http_requests_total{status=~\"5..\"}[5m])) / sum(rate(http_requests_total[5m])) * 100', ) ), gridPos={ x: 0, y: 0, w: 6, h: 4 }, ) What I run at home Here’s the whole thing in practice. On my homelab cluster I keep exactly five dashboards:\nHome. Everything OK? One glance. SLOs. Am I meeting my objectives, and how’s the error budget? Cluster. Kubernetes health from the top down. Service Debug. The variable-driven one, for when something’s wrong. Incident. The messy one that pulls metrics, logs and traces together. That’s the lot. I resist a sixth, actively. Every one of these gets opened weekly or more, and that’s the only test I apply. A dashboard that goes a month without being opened gets deleted, and so far I haven’t missed one.\nSet that next to the 47 from the top of this post. I’m not collecting less data, my observability stack hoovers up plenty. I’m reading five dashboards I trust instead of ignoring fifty.\nFighting dashboard rot Five stays five because I keep pruning. Left alone, any Grafana drifts back towards archaeology. A little process keeps it at five.\nReview on a schedule Once a month I go through everything and ask three things. Which dashboards got opened? Do the queries still point at services that exist? Do the thresholds still match how the system behaves? Metrics get renamed, SLOs get tightened, and a threshold that made sense a year ago can quietly stop meaning anything.\nSunset what nobody uses Not opened in 30 days, it moves to the Archive folder. Sixty days in Archive, it’s gone. Git has the history if I’m wrong.\nWrite down what each one is for Every dashboard gets a short description, so the next person knows why it exists. The next person is usually me, six months on, having forgotten everything.\nPurpose: Quick health check for production services Audience: On-call engineers When to use: Daily check, incident first response Links to: Service Debug dashboard for details Why I bother A dashboard is there to tell you something true about your system, fast enough that you act on it. A bad one is noise, and noise teaches people to ignore the monitoring altogether. That’s worse than no monitoring, because you think you’re covered.\nCollecting metrics is easy, and Grafana makes drawing graphs easy, so data was never the bottleneck. Reading it is. The five on my homelab are in Git, each one gets opened every week, and when one stops being opened it goes.","permalink":"https://tommeurs.nl/en/posts/grafana-dashboards-that-work/","title":"Grafana Dashboards That Actually Get Used"},{"content":" TL;DR Chaos engineering is breaking your own cluster on purpose, with the hypothesis written down first and a rollback ready. I run it on the homelab with Litmus: a weekly pod-delete, a monthly node drain, a game day every quarter. So far it has found four things the dashboard would never have shown me.\nMy dashboard is a wall of green. Pods running, replicas matched, CPU comfortable, no alerts firing. I look at it and get that small hit of “everything’s fine.” And mostly it is. The cluster’s been up for weeks and nothing has fallen over.\nThat wall is also the most dangerous thing in my homelab. It tells me nothing about what happens when something goes wrong. It only tells me that, right now, nothing has.\nSo picture the other version of that morning. A node dies at the one moment of the day when load matters. A DNS lookup starts timing out and takes every readiness probe down with it. A service I deployed eight months ago and forgot about eats memory until the kernel’s OOM killer steps in. None of that shows up on a healthy dashboard. It shows up while I’m asleep, and the person debugging it is me, badly rested and annoyed.\nSo how do I find out before production does? I break things myself. On a weekday afternoon, with a coffee and a rollback ready. That’s the whole pitch for chaos engineering, and I’d rather find my weak spots on my own schedule than let production pick the moment.\nWhat breaking things on purpose means A distributed system doesn’t fail in one place. You can read every line of code, draw the prettiest architecture diagram in the world, and still have no idea how the thing behaves when a node drops mid-request. That behaviour lives in the gaps between components. The only way to see it is to cause the failure and watch.\nThere’s a reason I keep coming back to this on the blog. I want to understand what I run. A green dashboard asks me to trust it, and I don’t fully trust anything I haven’t seen fail. After a chaos experiment I’ve watched it recover, instead of assuming it would.\nA clarification, because the name scares people off. Chaos engineering doesn’t mean randomly destroying things to see what happens, and it doesn’t mean yeeting fault injection into prod without a plan. Every experiment has three things attached to it:\nA hypothesis. “If one node fails, pods reschedule within five minutes.” You’re testing a belief you already hold. A blast radius you control. A known target, a set duration, a rollback you can hit straight away. Something you learn. Either the belief holds or you’ve got a bug you didn’t know about. I’ll take either. Netflix made this famous with Chaos Monkey, which killed random instances so engineers had no choice but to build services that survived it. The tooling has grown up a lot since then. The idea hasn’t changed. You cause the failure at a time you picked, instead of waiting for it.\nThe chaos engineering cycle Chaos Engineering Cycle\nDefine Hypothesis\nDesign Experiment\nRun in Controlled Environment\nObserve \u0026 Measure\nLearn \u0026 Improve\n1. Write down the hypothesis Write down what you believe is true before you touch anything. You’re committing to a prediction, so the result can surprise you:\n“If one node fails, pods will reschedule within 5 minutes” “If the database becomes slow, the API will degrade gracefully, not crash” “If DNS fails, cached responses will serve requests for 30 seconds” 2. Design the experiment Now scope it. Four questions:\nBlast radius. What can be affected? Duration. How long does the fault persist? Rollback. How do you stop it right now? Metrics. What tells you it passed or failed? People skip the rollback question. It’s the one that keeps an experiment from turning into an incident.\n3. Run it somewhere it can’t hurt Start small. Dev first, then staging, then production, and production only with safeguards. I skipped that order exactly once. I haven’t since.\n4. Observe and measure This is where your observability stack pays for itself. If you can’t see what happened, all you did was cause damage. Watch the obvious things. Did the right alerts fire? Did latency move? Did the error rate spike? How long did recovery take, next to what you guessed?\n5. Learn and fix Write down what you found, fix the weak spot, and update the runbook so the next person doesn’t start from scratch. The next person is usually me, six months on. Then design the next experiment.\nLitmus, chaos the Kubernetes way Litmus is a CNCF project that does chaos engineering the way Kubernetes does everything else. You declare an experiment as YAML and apply it. That’s why I picked it for the homelab. The experiment lives in Git next to the workload it attacks, so it’s reviewable and reproducible, and it isn’t a clever command someone ran once and never wrote down.\nInstalling Litmus # Add Litmus helm repo helm repo add litmuschaos https://litmuschaos.github.io/litmus-helm/ helm repo update # Install Litmus helm install litmus litmuschaos/litmus \\ --namespace litmus \\ --create-namespace \\ --set portal.frontend.service.type=ClusterIP Litmus architecture Litmus Components\nExecution Plane\nControl Plane\nLitmus Portal\n(UI/API)\nLitmus Server\nSubscriber\nChaos Runner\nChaos Experiments\nTarget Workload\nDeclaring an experiment with ChaosEngine apiVersion: litmuschaos.io/v1alpha1 kind: ChaosEngine metadata: name: nginx-chaos namespace: default spec: appinfo: appns: 'default' applabel: 'app=nginx' appkind: 'deployment' engineState: 'active' chaosServiceAccount: litmus-admin experiments: - name: pod-delete spec: components: env: - name: TOTAL_CHAOS_DURATION value: '30' - name: CHAOS_INTERVAL value: '10' - name: FORCE value: 'false' This kills nginx pods every 10 seconds for 30 seconds in total. Small, contained, easy to watch. Start here.\nThe experiments worth knowing You don’t need all of these on day one. It helps to know the menu though, because each one probes a different assumption you’re quietly making about your cluster.\nPod-level chaos Pod delete. Kills pods to test restart behaviour.\nexperiments: - name: pod-delete spec: components: env: - name: TOTAL_CHAOS_DURATION value: '60' - name: CHAOS_INTERVAL value: '10' - name: PODS_AFFECTED_PERC value: '50' # Kill 50% of pods Container kill. Kills one container inside a pod.\nexperiments: - name: container-kill spec: components: env: - name: TARGET_CONTAINER value: 'sidecar' - name: CHAOS_INTERVAL value: '10' Node-level chaos Node drain. Simulates node maintenance.\napiVersion: litmuschaos.io/v1alpha1 kind: ChaosEngine metadata: name: node-drain-chaos spec: engineState: 'active' auxiliaryAppInfo: '' chaosServiceAccount: litmus-admin experiments: - name: node-drain spec: components: env: - name: TOTAL_CHAOS_DURATION value: '60' - name: TARGET_NODE value: 'worker-1' Node CPU hog. Simulates CPU pressure.\nexperiments: - name: node-cpu-hog spec: components: env: - name: TOTAL_CHAOS_DURATION value: '60' - name: NODE_CPU_CORE value: '2' # Consume 2 cores Network chaos Pod network loss. Simulates a network partition.\nexperiments: - name: pod-network-loss spec: components: env: - name: NETWORK_INTERFACE value: 'eth0' - name: NETWORK_PACKET_LOSS_PERCENTAGE value: '100' # Total loss - name: TOTAL_CHAOS_DURATION value: '30' Pod network latency. Injects latency.\nexperiments: - name: pod-network-latency spec: components: env: - name: NETWORK_LATENCY value: '300' # 300ms latency - name: JITTER value: '100' # 100ms jitter Storage chaos Disk fill. Tests how the workload handles disk pressure.\nexperiments: - name: disk-fill spec: components: env: - name: FILL_PERCENTAGE value: '90' - name: TOTAL_CHAOS_DURATION value: '60' Running your first experiment Enough menu-reading. Here’s a full experiment against a throwaway app, start to finish, so you can copy it and watch something break.\n1. Deploy a test application apiVersion: apps/v1 kind: Deployment metadata: name: demo-app labels: app: demo spec: replicas: 3 selector: matchLabels: app: demo template: metadata: labels: app: demo spec: containers: - name: nginx image: nginx:alpine ports: - containerPort: 80 readinessProbe: httpGet: path: / port: 80 initialDelaySeconds: 5 periodSeconds: 5 2. Create a ServiceAccount for Litmus apiVersion: v1 kind: ServiceAccount metadata: name: litmus-admin namespace: default --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: litmus-admin rules: - apiGroups: [\"\"] resources: [\"pods\", \"pods/exec\", \"pods/log\", \"events\"] verbs: [\"create\", \"delete\", \"get\", \"list\", \"patch\", \"update\", \"watch\"] - apiGroups: [\"apps\"] resources: [\"deployments\", \"replicasets\"] verbs: [\"get\", \"list\", \"watch\"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: litmus-admin roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: litmus-admin subjects: - kind: ServiceAccount name: litmus-admin namespace: default 3. Run the pod-delete experiment apiVersion: litmuschaos.io/v1alpha1 kind: ChaosEngine metadata: name: demo-chaos namespace: default spec: appinfo: appns: 'default' applabel: 'app=demo' appkind: 'deployment' engineState: 'active' chaosServiceAccount: litmus-admin experiments: - name: pod-delete spec: components: env: - name: TOTAL_CHAOS_DURATION value: '30' - name: CHAOS_INTERVAL value: '10' - name: FORCE value: 'false' probe: - name: \"check-endpoint\" type: \"httpProbe\" httpProbe/inputs: url: \"http://demo-app.default.svc:80\" insecureSkipVerify: false method: get: criteria: \"==\" responseCode: \"200\" mode: \"Continuous\" runProperties: probeTimeout: 5 interval: 2 retry: 2 4. Observe the results # Watch chaos engine status kubectl get chaosengine demo-chaos -w # Check chaos result kubectl get chaosresult demo-chaos-pod-delete -o yaml # Watch pod behavior kubectl get pods -l app=demo -w Game days, the same idea aimed at the team Here’s the part the YAML can’t fix. A chaos experiment tells you whether the system survives. It says nothing about whether the humans behind it know what to do when the pager goes off. The first time anyone opens a runbook shouldn’t be during a real outage. A game day is how you avoid that.\nYou take the same controlled failure and point it at the team instead of the cluster.\nPlanning a game day Scope. Which systems, which failure scenarios. Team. The engineers who’d respond to the real thing. Runbooks. Written before the day, so there’s something to test. Success criteria. What does “handled well” look like? Time. A block in the calendar, not a gap between meetings. Game day structure 09:00 - Briefing: Explain scope, rules, how to abort 09:15 - Inject failure (team doesn't know which one) 09:15 - Team responds as if real incident 10:00 - Failure revealed, additional scenarios introduced 11:00 - Debrief: What worked, what didn't, what to improve Example scenarios Scenario What You Learn Kill primary database Failover speed, data integrity DNS outage Caching behavior, timeout handling Certificate expires Monitoring gaps, renewal process Cloud region unavailable Multi-region readiness Secrets manager down Application behavior without secrets Keeping it in Git The experiments are YAML, so they belong in version control with everything else. I keep mine managed with ArgoCD, defined but dormant, ready for when I want to run one:\n# chaos-experiments/pod-delete.yaml (in Git) apiVersion: litmuschaos.io/v1alpha1 kind: ChaosEngine metadata: name: scheduled-pod-delete namespace: chaos-testing spec: engineState: 'stop' # Manually activated # ... experiment config The flow is the one I use for everything else. Define the experiment in Git, let ArgoCD sync it, flip engineState to active when I want it to run, read the result, commit whatever fix or finding came out of it. The experiment and its history live next to the workload it tests.\nStaying safe while doing this Done carelessly, chaos engineering is an outage you caused yourself. The difference is discipline, so here’s mine.\nStart in non-production and stay there until you trust the experiment. Have the rollback ready before you start, not after things go sideways. Watch the experiment live instead of kicking it off and walking away. Tell anyone who might get paged that you’re about to do this. And write down what you found, because next month nobody will remember what it found.\nThe mistakes are just as predictable. Running during your real peak window when you weren’t trying to test peak behaviour. Injecting a failure without warning the on-call. Skipping the hypothesis, so you can’t tell signal from noise. Ignoring the finding because the cluster recovered “well enough.”\nMost important of all, decide your abort conditions up front. Error rate past a threshold, latency past a ceiling, real users complaining, or any behaviour you didn’t predict. Litmus can enforce part of this with a probe that kills the experiment for you:\n# Abort on high error rate probe: - name: \"error-rate-check\" type: \"promProbe\" promProbe/inputs: endpoint: \"http://prometheus:9090\" query: \"rate(http_requests_total{status=~'5..'}[1m])\" comparator: type: \"float\" criteria: \"\u003c\" value: \"0.1\" # Abort if \u003e 10% errors Why not everyone does this already If chaos engineering is this useful, why isn’t it everywhere? The obstacles are real, and if I don’t name them this reads like an easy sell. It isn’t one.\nThe big one is fear, and it’s reasonable fear. Deliberately breaking a system you’re responsible for feels insane the first time, especially if you’ve never seen it fail safely. “It works” is a strong argument against “let’s go break it.” Then there’s the time. Designing a real experiment, watching it, documenting it, fixing what it found, none of that is free, and it competes with shipping features. And there’s culture. On a team that assigns blame, causing a failure on purpose is a career risk, even when it’s the responsible thing to do.\nNone of those go away on their own. What worked for me was starting so small that there was nothing to be afraid of, a pod-delete against a throwaway nginx in a dev namespace.\nWhat I run in the homelab So here’s the version I run on my own cluster. Nothing heroic. A weekly pod-delete against random pods in non-critical namespaces, a monthly node drain to rehearse losing a machine, and roughly once a quarter a game day with a couple of stacked failures.\nThat small cadence has already paid for itself. What it taught me:\nLonghorn handles a node disappearing far more calmly than I expected. I used to hope that. Now I’ve seen it. Several of my workloads had a useless terminationGracePeriod and were getting hard-killed mid-request, and I’d never noticed in normal operation. My alerting fires later than it should, consistently. I’d rather learn that on a weekday afternoon than during an incident. Recovery takes longer than I guess. Always. Each of those four was a page I’d have got in the middle of the night, traded for an afternoon of mild discomfort. I’d take that trade again.\nThe dashboard is green again this morning. I trust it a bit more than I did, because I’ve watched it go red on my own schedule and come back.","permalink":"https://tommeurs.nl/en/posts/chaos-engineering-kubernetes/","title":"Chaos Engineering: Breaking Your Cluster to Make It Stronger"},{"content":" TL;DR Three refurbished mini-PCs, K3s with embedded etcd, kube-vip for one API address that survives a dead node, then MetalLB, Traefik and Longhorn on top. About €450 in hardware and an afternoon. Below are the commands, in the order I ran them.\nThree mini-PCs sit on a shelf in my house. Between them they run my GitLab, my monitoring stack, home automation, file sync, a password manager, and a pile of other things I’d rather not hand to someone else’s computer. The whole setup cost less than one month of the managed Kubernetes bill for the same thing, and I know what every layer does because I put it there.\nNote added 2026-09-23. The forge on that shelf is Forgejo now, not GitLab. Nothing about building the cluster changes, so substitute whichever forge you run wherever this post says GitLab.\nYou don’t need a cloud account or a rack for this. Three second-hand machines and an afternoon where nobody bothers you will do. This is the cluster I keep pointing at when I write about sovereign infrastructure, so here’s how it goes together, step by step.\nOne command first Before any hardware talk, the part that surprises people. A working Kubernetes cluster is one command.\ncurl -sfL https://get.k3s.io | sh - That’s a single-node K3s install. A real Kubernetes API, a real kubectl, workloads that run. If all you want is to poke at manifests on a spare laptop, you can stop reading here. You’ve already got something useful.\nK3s is why this is so painless. It’s the full Kubernetes API in one binary of about 70MB, with the bits most homelabs never touch stripped out and sane defaults for the rest. It runs on a Raspberry Pi, and it runs very well on a mini-PC. People call it “Kubernetes lite” sometimes, which sells it short. What you deploy on it is identical to what you’d deploy on a big managed cluster. You skip the enterprise overhead you weren’t going to touch anyway.\nThe rest of this post turns that one command into something I’d trust with data I care about. Three nodes, embedded etcd, a virtual IP that survives a node dying, and the storage and ingress to host things on.\nWhy refurbished mini-PCs You can build this on anything. I keep recommending the same thing because it keeps being right. Refurbished business mini-PCs are the best deal I know of for this, and for the money nothing else comes close.\nWhat Why HP EliteDesk 800 G3 Reliable, cheap, available everywhere Lenovo ThinkCentre M720q Great thermals, quiet Dell OptiPlex 3060 Micro Good expansion options Aim for roughly this per node:\nCPU: Intel i5 6th-8th gen (4-6 cores) RAM: 16-32GB (DDR4, upgradeable) Storage: 256GB+ NVMe/SSD Network: Gigabit Ethernet Expect to pay around €100-200 per node, so €300-600 for three.\nPeople always ask about a Raspberry Pi cluster instead, and Pis do work. I ran one for a while. But you’re fighting the platform the whole time. ARM images that don’t exist yet for the thing you want. SD cards that corrupt themselves when a node loses power. 8GB of RAM as a hard ceiling on a Pi 4. USB-attached storage that flakes out at the worst moment. A refurbished mini-PC gives you x86, an NVMe slot, proper RAM, and a machine that was built to run business workloads for years without complaint. I haven’t run a Pi as a node since.\nMy own cluster is three of the same box, which keeps things simple:\nNode 1: HP EliteDesk 800 G3 Mini i5-7500T, 32GB RAM, 512GB NVMe Role: Control plane + Worker Node 2: HP EliteDesk 800 G3 Mini i5-7500T, 32GB RAM, 512GB NVMe Role: Control plane + Worker Node 3: HP EliteDesk 800 G3 Mini i5-7500T, 16GB RAM, 256GB NVMe Role: Control plane + Worker That’s about €450 for 12 cores, 80GB of RAM and 1.2TB of storage.\nGetting the nodes ready Two things to sort out before K3s goes on. Addresses and the OS.\nPredictable addresses Every node needs an IP that doesn’t move. Set static IPs, or easier, hand out DHCP reservations from your router.\n192.168.1.10 k3s-node-1 192.168.1.11 k3s-node-2 192.168.1.12 k3s-node-3 192.168.1.100 k3s-api # VIP for API server That .100 is a virtual IP we’ll set up later, so the API has one stable address even when a node dies. Give the nodes DNS names too, in your local resolver, or in /etc/hosts if you’re in a hurry.\n192.168.1.10 k3s-node-1.home.lan 192.168.1.11 k3s-node-2.home.lan 192.168.1.12 k3s-node-3.home.lan 192.168.1.100 k3s.home.lan The OS To get going, Ubuntu Server 22.04 LTS minimal is the path of least resistance. Install it, update it, and turn swap off, because Kubernetes wants it off.\n# After install, update sudo apt update \u0026\u0026 sudo apt upgrade -y # Install useful tools sudo apt install -y curl wget vim htop # Disable swap (required for Kubernetes) sudo swapoff -a sudo sed -i '/ swap / s/^/#/' /etc/fstab If you want the immutable, locked-down version of this, Talos Linux is where I’d send you next. It’s the better foundation and it’s harder to start on, so I’d keep it for round two. Ubuntu gets you a cluster today.\nThree servers, one control plane This is the step from “Kubernetes on a laptop” to “Kubernetes I’d trust.” The flag that matters is --cluster-init. It turns on embedded etcd, so the control plane survives a node failure.\nOn node 1:\n# On node 1 curl -sfL https://get.k3s.io | sh -s - server \\ --cluster-init \\ --tls-san k3s.home.lan \\ --tls-san 192.168.1.100 \\ --disable traefik \\ --disable servicelb What those flags do.\n--cluster-init turns on embedded etcd, which is what makes the control plane HA. --tls-san adds the DNS name and the VIP to the API certificate as SANs, so it’s valid for the address we’re about to use. --disable traefik skips the bundled ingress. We’ll bring our own. --disable servicelb skips the bundled load balancer. MetalLB does this better. Grab the join token:\nsudo cat /var/lib/rancher/k3s/server/node-token Then bring up nodes 2 and 3, pointed at node 1 and with the same flags:\n# On node 2 and 3 curl -sfL https://get.k3s.io | sh -s - server \\ --server https://192.168.1.10:6443 \\ --token \u003cTOKEN_FROM_NODE_1\u003e \\ --tls-san k3s.home.lan \\ --tls-san 192.168.1.100 \\ --disable traefik \\ --disable servicelb Pull the kubeconfig, point it at your workstation, and check that all three nodes joined:\n# Get kubeconfig sudo cat /etc/rancher/k3s/k3s.yaml # Copy to your workstation, update server URL export KUBECONFIG=~/.kube/k3s-config # Check nodes kubectl get nodes NAME STATUS ROLES AGE VERSION k3s-node-1 Ready control-plane,etcd,master 5m v1.28.5+k3s1 k3s-node-2 Ready control-plane,etcd,master 3m v1.28.5+k3s1 k3s-node-3 Ready control-plane,etcd,master 2m v1.28.5+k3s1 Three control-plane nodes, each with an etcd member. Any one of them can fall over now and the cluster keeps running. That’s what the three boxes are for.\nOne address that survives a dead node There’s a gap left. Your kubeconfig points at one node’s IP, and if that’s the node that dies, kubectl stops working even though the cluster is fine. You want one API address that floats to whichever control-plane node is still alive.\nMy pick is kube-vip. It runs as a DaemonSet on the control-plane nodes and answers for the VIP (192.168.1.100) over ARP, moving it to a healthy node by itself.\n# On each control plane node kubectl apply -f https://kube-vip.io/manifests/rbac.yaml # Create kube-vip manifest cat \u003c\u003cEOF | sudo tee /var/lib/rancher/k3s/server/manifests/kube-vip.yaml apiVersion: apps/v1 kind: DaemonSet metadata: name: kube-vip namespace: kube-system spec: selector: matchLabels: app: kube-vip template: metadata: labels: app: kube-vip spec: hostNetwork: true tolerations: - key: node-role.kubernetes.io/control-plane operator: Exists effect: NoSchedule containers: - name: kube-vip image: ghcr.io/kube-vip/kube-vip:latest args: - manager env: - name: vip_interface value: eth0 - name: vip_address value: 192.168.1.100 - name: vip_arp value: \"true\" - name: lb_enable value: \"true\" EOF If you’d rather keep the VIP logic off the cluster, and your router runs HAProxy, you can balance the API there instead:\nfrontend k3s-api bind *:6443 default_backend k3s-servers backend k3s-servers balance roundrobin server node1 192.168.1.10:6443 check server node2 192.168.1.11:6443 check server node3 192.168.1.12:6443 check Both work. I run kube-vip, because it keeps everything inside the cluster and that’s one box fewer in the failure path. HAProxy on the router is easier to reason about if you already trust your router. Pick whichever matches what you run.\nLoad balancing, ingress and storage A bare cluster doesn’t host anything yet. Three pieces turn it into something you can deploy to.\nLoadBalancer services with MetalLB We turned the bundled servicelb off, so a LoadBalancer service has nothing answering it. MetalLB fills that gap by handing out IPs from a pool on your LAN.\nkubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.14.3/config/manifests/metallb-native.yaml Give it a range to hand out, well clear of your node and VIP addresses:\napiVersion: metallb.io/v1beta1 kind: IPAddressPool metadata: name: default namespace: metallb-system spec: addresses: - 192.168.1.200-192.168.1.250 --- apiVersion: metallb.io/v1beta1 kind: L2Advertisement metadata: name: default namespace: metallb-system Ingress We dropped the bundled Traefik so we could install it on our own terms. Here it is via Helm , taking a LoadBalancer IP from the MetalLB pool we just defined:\n# Traefik (via Helm) helm repo add traefik https://traefik.github.io/charts helm install traefik traefik/traefik \\ --namespace traefik \\ --create-namespace \\ --set service.type=LoadBalancer Storage with Longhorn Pods that keep data need real persistent volumes. Longhorn gives you block storage replicated across the three nodes, so a volume survives a node going down:\nhelm repo add longhorn https://charts.longhorn.io helm install longhorn longhorn/longhorn \\ --namespace longhorn-system \\ --create-namespace If you’re weighing storage options, I put Longhorn next to the heavier alternative in Longhorn vs Rook-Ceph.\nTalking to the cluster from your desk Last bit of setup. Get a kubeconfig onto your workstation, so you stop SSHing into node 1 for everything. The one K3s generates points at 127.0.0.1. Swap that for your VIP’s DNS name and the kubeconfig keeps working whichever node answers.\n# On node 1 sudo cat /etc/rancher/k3s/k3s.yaml # Save locally, edit server URL # Change: server: https://127.0.0.1:6443 # To: server: https://k3s.home.lan:6443 # Set permissions chmod 600 ~/.kube/k3s-config export KUBECONFIG=~/.kube/k3s-config Keeping it alive A homelab cluster you never maintain is a problem waiting for the day it breaks. A few habits keep it boring, and boring is what I want from infrastructure.\nUpgrades Upgrade one node at a time by rerunning the install with the same flags, or let the system-upgrade-controller do it for you:\n# On each node, one at a time curl -sfL https://get.k3s.io | sh -s - server \\ # same flags as initial install kubectl apply -f https://github.com/rancher/system-upgrade-controller/releases/latest/download/system-upgrade-controller.yaml Backups etcd is where the whole cluster state lives. Snapshot it. K3s does this on a schedule if you ask, so ask:\n# Manual snapshot k3s etcd-snapshot save --name manual-backup # Scheduled backups (add to k3s config) # /etc/rancher/k3s/config.yaml etcd-snapshot-schedule-cron: \"0 */6 * * *\" etcd-snapshot-retention: 5 Pulling a node for maintenance Need to open a box or reboot it? Drain it first, so workloads reschedule cleanly, then bring it back:\n# Drain node before maintenance kubectl drain k3s-node-2 --ignore-daemonsets --delete-emptydir-data # Do maintenance... # Uncordon when done kubectl uncordon k3s-node-2 When something’s broken A few failures come up often enough that I know them by heart. If kubectl can’t reach the server, check that K3s is running at all, that the firewall isn’t eating port 6443, and that your kubeconfig points where you think it does:\nsudo systemctl status k3s sudo ufw status A node stuck on NotReady usually means the kubelet, the container runtime or the disk is unhappy:\nsudo journalctl -u k3s -f sudo crictl ps df -h And if etcd itself gets corrupted, you’ll be reaching for those snapshots. The --cluster-reset path exists, but treat it as a last resort, because it tears the cluster down to a single member:\n# Check etcd health sudo k3s etcd-snapshot list # Force new cluster if etcd corrupted (DANGEROUS) # sudo k3s server --cluster-reset What it’s for That one command at the top is now three nodes sharing an HA control plane, a VIP that survives a dead node, load balancing, ingress, replicated storage and a backup schedule. Enough to host things on.\nHere’s what I run on mine:\nGitLab for code hosting Prometheus and Thanos for monitoring ArgoCD for GitOps Home Assistant for automation Nextcloud for file sync Vaultwarden for passwords All of it self-hosted, all of it in my house.\nThe money side is lopsided. The same thing in the cloud would cost me €150-300 a month. The shelf costs maybe €10-20 in electricity:\nCloud (3 nodes) Homelab (3 nodes) Monthly €150-300 €10-20 (electricity) Annual €1800-3600 €120-240 3 year €5400-10800 €360-720 + €450 hardware The hardware pays for itself in three to six months. After that it’s compute I own outright, for the price of the power.\nBut money was never the reason I did this. Running your own cluster teaches you how Kubernetes behaves. A managed one teaches you which command makes the dashboard go green. You find out what happens when a node dies, because one will, and you’ll be the one watching the VIP move. You learn capacity planning with your own RAM on the line. None of that comes with a managed cluster’s control panel.\nAnd you get sovereignty over the whole stack. Your data stays yours, and so does the responsibility for it. The cloud is fine for work that belongs to someone else. The things that matter to me sit on a shelf in my house, where I can reach out and touch them, and now and then pull the plug on one to watch the other two carry on.","permalink":"https://tommeurs.nl/en/posts/k3s-cluster-setup/","title":"K3s Cluster Setup on Refurbished Mini-PCs"},{"content":" TL;DR Longhorn and Rook-Ceph both give you replicated block storage that survives a node dying. Longhorn is one Helm install and a UI that shows you what a volume is doing. Rook-Ceph is Ceph with a Kubernetes operator around it, so you get block, object and filesystem storage, and the operations that come with them. I run Longhorn on three nodes. Below is why, and when I’d switch.\nRun your first stateful workload on a self-hosted cluster and you find out fast that nobody hands out PersistentVolumes for free. There’s no cloud provider storage class to lean on. You’ve got your nodes, their disks, and a Postgres pod that won’t schedule. An hour of reading later you’re left with two names that keep coming up. Longhorn and Rook-Ceph .\nI’ve run both in production, and I’ll say up front that I default to Longhorn on small clusters. The why comes later, but you should know it while you read. Both are CNCF projects, both replicate block storage across nodes so a dead node doesn’t take your data with it, and both are good software. Where they disagree is how much complexity you should have to take on.\nWhat you’re choosing between Longhorn is distributed block storage written for Kubernetes from day one. Each volume gets replicated across your nodes with plain Linux storage primitives, and there’s no separate storage system underneath that you’d have to learn first.\nRook-Ceph is a Kubernetes operator wrapped around Ceph. Ceph is a distributed storage system that’s years older than Kubernetes, runs petabytes at places like CERN, and brings its whole feature set with it. Block, object, filesystem, erasure coding, all of it. Rook teaches Kubernetes how to drive it.\nMost of what follows comes from that difference in age. Longhorn was born in your cluster. Ceph moved in from somewhere else and brought everything it owned. Some of that you’ll use. The rest you’ll still have to run.\nFour things that decide it A feature checklist won’t help you here, because both tools tick most of the boxes. What matters is what running it is like. I weigh four things. How much operational work I’m taking on. How it behaves when a node dies, because this is storage and that’s the whole job. How much of my RAM and CPU it wants. And how far it scales before it gets awkward. Performance matters too, but for most homelab and small-production workloads it isn’t what decides it, so I’ll get to it later.\nLonghorn: the one that lives in your cluster Longhorn Architecture\nNode 3\nNode 2\nNode 1\nLonghorn Engine\nReplica\nLonghorn Engine\nReplica\nReplica\nPersistentVolume\nHow Longhorn works The model fits in your head, and that’s most of the appeal. Every PVC gets its own Longhorn engine, running as a pod. That engine writes your data to replicas on the local disks of several nodes, and it writes synchronously, so nothing is acknowledged until every replica has it. The workload talks to its volume over iSCSI, which the engine exposes.\nWhen I want to know what a volume is doing, I open the Longhorn UI and look. Which replicas are healthy, where they live, whether a rebuild is running. No black box. That’s how I want to run infrastructure, as I wrote in Sovereign Infrastructure. I need to understand what I’m running, and Longhorn lets me.\nInstalling it helm repo add longhorn https://charts.longhorn.io helm repo update helm install longhorn longhorn/longhorn \\ --namespace longhorn-system \\ --create-namespace Basic configuration:\n# longhorn-values.yaml defaultSettings: defaultReplicaCount: 3 defaultDataPath: /var/lib/longhorn storageMinimalAvailablePercentage: 15 defaultLonghornStaticStorageClass: longhorn persistence: defaultClass: true defaultClassReplicaCount: 3 What Longhorn does well Helm install, and you have storage. No dedicated storage nodes, no pool topology to design first. The built-in web UI shows volumes, backup status and node health without you wiring anything up. Backups go straight to S3-compatible storage as incremental snapshots, which on my setup means I point it at my own MinIO and forget about it. And there’s no older system underneath being adapted, so nothing you meet is there for historical reasons.\nWhere Longhorn falls short The limits are easy to state. Longhorn is fine for the workloads most of us run, but it isn’t built for extreme IOPS, because every volume’s traffic goes through its own engine pod and that pod is the ceiling. It works well up to roughly 100 nodes and gets awkward past that. And every replica is a full copy, so three replicas means three times the raw capacity. There’s no erasure coding to soften that.\nRook-Ceph: the one that moved in Rook-Ceph Architecture\nAccess\nStorage\nManagement\nRook Operator\nCeph Monitors\nCeph Manager\nOSD\n(disk 1)\nOSD\n(disk 2)\nOSD\n(disk 3)\nOSD\n(disk 4)\nRBD\n(Block)\nRGW\n(Object)\nCephFS\n(Filesystem)\nPersistentVolume\nHow Rook-Ceph works Ceph’s model is clever, and the cleverness is exactly why there’s more to learn. Each disk becomes an OSD, an Object Storage Daemon. Data spreads across those OSDs by the CRUSH algorithm and placement rules you write, so Ceph decides where each piece lives based on your failure domains instead of round-robin. On top of that you get three ways in. Block via RBD, S3-compatible object storage via RGW, and a real filesystem via CephFS. Holding it all together is a quorum of monitor daemons that track cluster state.\nEvery one of those parts is something you can inspect, which I like. It’s also something you’ll have to understand the first time it misbehaves.\nInstalling it helm repo add rook-release https://charts.rook.io/release helm repo update # Install Rook operator helm install rook-ceph rook-release/rook-ceph \\ --namespace rook-ceph \\ --create-namespace # Create Ceph cluster kubectl apply -f ceph-cluster.yaml Cluster configuration:\n# ceph-cluster.yaml apiVersion: ceph.rook.io/v1 kind: CephCluster metadata: name: rook-ceph namespace: rook-ceph spec: cephVersion: image: quay.io/ceph/ceph:v18.2.0 mon: count: 3 allowMultiplePerNode: false mgr: count: 2 storage: useAllNodes: true useAllDevices: false deviceFilter: \"^sd[b-z]\" # Use sdb, sdc, etc. resources: mon: requests: cpu: 500m memory: 1Gi osd: requests: cpu: 500m memory: 2Gi What Ceph does well Here the extra weight pays for itself. Ceph handles petabytes, the scale CERN and Bloomberg run it at, so you won’t outgrow it. Block, object and filesystem from one system, plus erasure coding, snapshots and cross-cluster mirroring. There’s a lot to tune, so a team that knows Ceph can shape it to a workload. And erasure coding cuts the overhead. Instead of paying 3x for replication you can land closer to 1.5x, and at large capacity that’s real money.\nWhat Ceph costs you The same power costs the same effort. Many more moving parts, and monitors, managers and OSDs all want resources and attention. Before you’ve stored a byte you’re running three monitors, two managers and your OSDs, and the memory they take is not small. Ceph carries decades of features and configuration, so the learning curve is steep and doesn’t flatten quickly. For real performance you often end up dedicating nodes to OSDs, hardware you’ve set aside for storage and nothing else. I don’t count any of that as a flaw. It’s what Ceph charges for what it gives you, and the price only makes sense if you’ll use what you bought.\nSide by side Aspect Longhorn Rook-Ceph Complexity Low High Setup time 10 minutes 30+ minutes Resource overhead Low High Max scale ~100 nodes 1000+ nodes Storage types Block only Block, Object, Filesystem Performance Good Excellent (when tuned) Storage efficiency 3x (replication) 1.5x+ (erasure coding) Backup Built-in S3 External tools UI Excellent Ceph Dashboard Community Growing Mature The table is fine for a quick look, but the decision lives in the rows you’ll feel. For me that’s resource overhead and complexity, because I pay for those every day the cluster runs, and for setup time only once.\nWhen Longhorn is the right call Reach for Longhorn when your situation looks like this:\nSmall to medium clusters (under 100 nodes) Simplicity matters and you want storage that just works Limited ops capacity, a small team that can’t spend its time babysitting storage General workloads like databases and stateful apps with moderate I/O Homelab or edge where resources are tight # Typical Longhorn workload apiVersion: v1 kind: PersistentVolumeClaim metadata: name: postgres-data spec: accessModes: - ReadWriteOnce storageClassName: longhorn resources: requests: storage: 100Gi When Rook-Ceph is worth it Reach for Rook-Ceph when:\nLarge clusters (100+ nodes) Multiple storage types needed, block AND object AND filesystem from one system Performance critical and you need to tune for specific workloads Storage efficiency matters and erasure coding will save you real capacity Dedicated storage team, people who can learn Ceph and operate it well That last one is the point people skip. Ceph rewards a team that knows it and punishes one that doesn’t, and if nobody on the team owns the storage, the first person to learn Ceph properly will be doing it during an outage.\n# Rook-Ceph with erasure coding apiVersion: ceph.rook.io/v1 kind: CephBlockPool metadata: name: replicated-pool namespace: rook-ceph spec: failureDomain: host replicated: size: 3 --- apiVersion: ceph.rook.io/v1 kind: CephBlockPool metadata: name: erasure-coded-pool namespace: rook-ceph spec: failureDomain: host erasureCoded: dataChunks: 2 codingChunks: 1 Performance, the criterion I parked I parked this one earlier, and here’s why it rarely decides anything. For the workloads most people run, both are fast enough. The difference shows up at the edges.\nLonghorn under load # Tune replica count for performance vs durability defaultSettings: defaultReplicaCount: 2 # Faster than 3, less durable # Use dedicated disk path defaultDataPath: /mnt/fast-ssd/longhorn Longhorn is I/O bound by that per-volume engine pod. Push a high-IOPS workload through it and the engine is your bottleneck. That’s the trade you make for the simple architecture.\nRook-Ceph under load # Dedicated OSD nodes spec: placement: osd: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: storage-node operator: In values: - \"true\" # NVMe optimization storage: config: osdsPerDevice: \"1\" storeType: bluestore Ceph can saturate modern NVMe drives when it’s configured properly. “When it’s configured properly” is doing a lot of work in that sentence, and getting there is the work you signed up for.\nBackups Backups matter more to me than raw throughput, because storage you can’t restore from will lose data at some point and you won’t know when.\nLonghorn backups Built in. Configure an S3 target:\ndefaultSettings: backupTarget: s3://longhorn-backups@us-east-1/ backupTargetCredentialSecret: longhorn-s3-credentials Schedule backups per volume:\napiVersion: longhorn.io/v1beta1 kind: RecurringJob metadata: name: daily-backup spec: cron: \"0 2 * * *\" task: backup groups: - default retain: 7 Rook-Ceph backups There’s no equivalent built-in flow, so you reach for Velero with Ceph CSI snapshots:\nvelero install \\ --provider aws \\ --plugins velero/velero-plugin-for-csi \\ --features=EnableCSI Or native Ceph mirroring for disaster recovery between clusters, which is nice once you’re at the scale where a second cluster exists.\nThree nodes, so Longhorn I said at the top that I default to Longhorn. I run Longhorn in my homelab, and this is the configuration:\n# My Longhorn configuration defaultSettings: defaultReplicaCount: 2 # 3 nodes, 2 replicas defaultDataPath: /mnt/storage/longhorn backupTarget: s3://backups@minio/longhorn/ backupTargetCredentialSecret: minio-credentials storageMinimalAvailablePercentage: 20 persistence: defaultClass: true Three nodes is the deciding fact. That’s too small to carry Ceph’s overhead, where the monitors and managers alone would eat capacity I can’t spare. When a volume misbehaves I open the Longhorn UI and read what’s wrong. With Ceph I’d be paging through internals working out which daemon does what, and I’ve done enough of that already. Backups land in my own MinIO over S3, and on small nodes every spare MB stays spare.\nThe day I’m running 50-plus nodes, or need object storage next to block, I’ll switch to Ceph and pay the complexity gladly, because then I’d be using what I paid for. That day isn’t here. Your cluster might put it a lot closer. If you’ve got the team and the scale, pick Ceph and don’t let my three nodes talk you out of it.\nMigrating later if you outgrow Longhorn Starting on Longhorn and worried you’re painting yourself into a corner? You aren’t. The path out is boring, which is the best thing you can say about a migration:\nBack up the data from the Longhorn volume Deploy Rook-Ceph alongside it Restore into Ceph volumes Update workloads to use the new StorageClass Retire Longhorn once everything’s moved Both speak CSI, so your workloads see the same interface either way. You change a StorageClass and you’re done.\nPicking the complexity you can carry Storage is the part of Kubernetes that punishes mistakes hardest. Get it wrong and you lose data, which is the one failure you can’t roll back. Over-build it and you spend your weeks feeding complexity you never needed.\nMap it to scale and the answer usually falls out. Homelab and small clusters point at Longhorn. Medium production goes either way, depending on whether you’d use the extra Ceph features. Large scale points at Ceph. Both are solid software. What separates them is how much operational weight you want to carry, and only you know that for your own cluster.\nMine is three nodes and Longhorn, and it survives a node dying, which is the failure I built it for. When it outgrows that, the five steps above are the whole plan.","permalink":"https://tommeurs.nl/en/posts/longhorn-vs-rook-ceph/","title":"Longhorn vs Rook-Ceph: Kubernetes Storage Compared"},{"content":" TL;DR One GPG masterkey that only certifies, kept offline. Three aliases bound to it, so anyone can check they’re the same person. Ed25519 subkeys for signing and SSH, an ML-KEM plus X448 subkey for encryption so recorded ciphertext stays closed after a quantum computer shows up, and three backups that each assume the other two are gone. Needs GnuPG 2.5.\nA cryptographic signature still means exactly what it says. If the key is yours and the signature verifies, the content came from you. No vendor handed you that identity, no CA can pull it, no platform can suspend it. It exists because you generated the key, and it’s yours for as long as you hold the private half. Most of what gets called “online identity” is borrowed. A handle someone can ban, an email address the domain owner can take back the day they feel like it. A GPG signature sits outside all of that.\nFor years that was a property crypto nerds cared about and almost nobody else did. That’s changing, and faster than I expected. I used to assume the human behind an email or a commit was a human, and probably the one the headers claimed. I don’t anymore. AI-generated text, voice and video keep getting cheaper, and my inbox already has the results. PR messages that ape a maintainer’s style. A synthetic voice call from a “colleague” who needs a favour. Proving that you wrote the post or shipped the commit isn’t free any more. A signature is the one thing in that pile that can’t be faked without stealing your key first.\nThere’s a second pressure on the same setup, slower and much further out. If a big enough quantum computer ever shows up, everything you ever encrypted with classical public-key crypto becomes readable after the fact. The name for this is harvest-now-decrypt-later, and state-level adversaries are assumed to be doing it already. They record your encrypted traffic today and sit on it, planning to crack it in ten or twenty years when the hardware exists. You can’t un-send the ciphertext you already leaked. You can stop adding to it, and that means post-quantum encryption now, while the tooling is still a bit rough.\nSo here’s the question I care about. Will the GPG setup you have right now still hold up in a decade? Attribution is a problem today. Long-horizon confidentiality is a problem later. A key you generate today has to survive both.\nThis post is the setup I landed on after chewing on that. One offline masterkey, three identity aliases bound to it, post-quantum encryption via ML-KEM (Kyber), and three independent backup layers. It runs long, because every choice here is a trade-off and the reasoning is the part worth reading. New to GPG? Read GPG explained first. From here on I assume you’ve generated at least one key in your life.\nWhat we’re building [Masterkey - Ed25519 - Certify only] ← offline, in a vault, on paper │ ├── UID 1: Tom Herder \u003ctom@byteherder.com\u003e ├── UID 2: kapott \u003ckapott@pm.me\u003e ├── UID 3: Tom Meurs \u003ctom.meurs@gmail.com\u003e │ ├── [Sub 1 - Ed25519] [S] Sign → email and git ├── [Sub 2 - ML-KEM+X448] [E] Encrypt → quantum-safe └── [Sub 3 - Ed25519] [A] Auth → SSH login One root, three identities, three subkeys. Why carve it up like this?\nChoice Reason Masterkey = certify-only The master only signs other keys and identities. Never used for mail, never used for git. Keep it offline and your identity survives every laptop theft. Separate subkeys for Sign / Encrypt / Auth You can rotate or revoke each one independently. Laptop stolen? Revoke subkeys, keep identity. Multiple UIDs on one key All three aliases are cryptographically bound to the same master - proof they come from the same person. ML-KEM (Kyber) for encryption Defends against harvest-now-decrypt-later. Someone recording your ciphertext today cannot decrypt it in 2046 with a quantum computer. Ed25519 for signing and auth Signatures do not have a harvest problem - they are verified at the moment of signing. Full post-quantum signing (ML-DSA) exists in RFC 9580 but tooling support in 2026 is still thin (Thunderbird, GitHub, SSH servers). Ed25519 is the pragmatic choice. One trade-off I want to flag before you copy any commands. Putting kapott@pm.me and tom.meurs@gmail.com on the same key makes it publicly provable that “kapott” and “Tom Meurs” are the same human. Great if you want attribution across your aliases. Dangerous if kapott is supposed to stay pseudonymous, because once that public key hits a keyserver the link is permanent. Keyservers don’t forget, and you can’t ask them to. If you need a real pseudonym, give it its own key (scenario B at the end).\nPrerequisites You need GnuPG 2.5 or later. ML-KEM subkey generation doesn’t exist in 2.4.\n# Arch sudo pacman -S gnupg paperkey qrencode zbar diceware pwgen veracrypt # Debian/Ubuntu - enable the official GnuPG repo for 2.5.x curl -fsSL https://repos.gnupg.com/GPG-KEY-gnupg.asc \\ | sudo tee /etc/apt/keyrings/gnupg.asc \u003e /dev/null echo \"deb [signed-by=/etc/apt/keyrings/gnupg.asc] https://repos.gnupg.com/debian bookworm main\" \\ | sudo tee /etc/apt/sources.list.d/gnupg.list sudo apt update \u0026\u0026 sudo apt install -y gnupg2 paperkey qrencode zbar-tools veracrypt diceware pwgen Verify:\ngpg --version | head -2 # must be 2.5.x or later Generate the keys on a clean OS. An Ubuntu live-USB or Tails is the paranoid option, and paranoid is the right setting here. Nothing your daily laptop might be quietly running gets near the master. Work inside a dedicated GNUPGHOME so none of this leaks into your existing keyring:\nmkdir -p ~/gpg-build \u0026\u0026 chmod 700 ~/gpg-build export GNUPGHOME=~/gpg-build Harden GnuPG Drop a hardened config in $GNUPGHOME/gpg.conf before generating anything. It disables weak algorithms and opts into modern defaults:\ncat \u003e \"$GNUPGHOME/gpg.conf\" \u003c\u003c'EOF' keyid-format 0xlong with-fingerprint with-subkey-fingerprint list-options show-uid-validity no-emit-version no-comments no-greeting export-options export-minimal personal-cipher-preferences AES256 AES192 AES personal-digest-preferences SHA512 SHA384 SHA256 default-preference-list SHA512 SHA384 SHA256 AES256 AES192 AES ZLIB BZIP2 ZIP Uncompressed weak-digest SHA1 disable-cipher-algo 3DES disable-cipher-algo CAST5 disable-pubkey-algo RSA1024 require-cross-certification no-symkey-cache throw-keyids keyserver hkps://keys.openpgp.org EOF chmod 600 \"$GNUPGHOME\"/gpg.conf gpgconf --kill gpg-agent \u0026\u0026 gpgconf --launch gpg-agent A passphrase you can remember The passphrase is the last thing between an attacker and your master once they have a copy of the encrypted file. It has to survive offline brute-forcing by someone with hardware you can’t predict. Use Diceware, a random run of seven or more words pulled from a big list. Easier to keep in your head than %Xh9$!Lq, and stronger.\ndiceware -n 8 -d ' ' --no-caps # schemerig koning fiets bramboes tunnel ijverig mosterd kraaier Write it on paper and lock that paper somewhere physical. Lose the passphrase and every backup in this post is landfill.\nCreate the masterkey We want Ed25519, certify-only, ten years.\ngpg --expert --full-generate-key Answers: 11 (ECC with custom capabilities) → toggle s to remove Sign (Certify only remains) → q → 1 (Curve 25519) → 10y → your primary UID (I use my real name and my real mail address). GPG asks for the passphrase twice and you’re done.\nCapture the fingerprint right away so the rest of this post can use $KEYID:\nexport KEYID=$(gpg --list-secret-keys --with-colons | awk -F: '/^fpr/ {print $10; exit}') echo \"$KEYID\" \u003e ~/gpg-build/KEY_INFO.txt Add UIDs for each alias gpg --expert --edit-key \"$KEYID\" Inside the editor:\ngpg\u003e adduid # Tom Herder \u003ctom@byteherder.com\u003e gpg\u003e adduid # kapott \u003ckapott@pm.me\u003e gpg\u003e uid 1 # pick your primary gpg\u003e primary gpg\u003e save Every UID is now signed by the same master. That signature is the proof that they belong to one person.\nAdd the subkeys Still in --edit-key:\nSigning subkey (Ed25519): addkey → 10 (ECC sign only) → 1 → 2y.\nEncryption subkey - the post-quantum one: addkey → 16 (ECC and Kyber) → 4 (Kyber 1024 + X448) → 2y.\nThis is the hybrid piece, and the reason you needed GnuPG 2.5. The subkey welds ML-KEM-1024 (NIST FIPS 203, quantum-safe) onto X448 (classical ECC). To read your ciphertext an attacker has to break both. So you’re covered whichever side cracks first, whether classical crypto falls to a quantum computer or somebody finds a hole in ML-KEM itself.\nAuthentication subkey (Ed25519): addkey → 11 (ECC custom) → toggle s off and a on → q → 1 → 2y → save.\nVerify with gpg --list-secret-keys --with-subkey-fingerprint. You should see [C], [S], [E], [A] markers lining up with master, sign, encrypt, auth.\nWhile you’re still in here, generate a revocation certificate. It’s the kill switch for the day the masterkey is compromised, and you make it now, while the master is still in front of you:\ngpg --output \"$GNUPGHOME/revocation-$KEYID.asc\" --gen-revoke \"$KEYID\" Take the master offline This is the step that makes the rest worth the effort. Export everything, wipe the master secret out of the working keyring, then re-import only the subkey secrets.\ncd \"$GNUPGHOME\" gpg --export-secret-keys --armor --output MASTER-FULL.asc \"$KEYID\" gpg --export-secret-subkeys --armor --output SUBKEYS-ONLY.asc \"$KEYID\" gpg --export --armor --output PUBLIC.asc \"$KEYID\" gpg --export-ownertrust \u003e OWNERTRUST.txt gpg --delete-secret-keys \"$KEYID\" gpg --import SUBKEYS-ONLY.asc gpg --import-ownertrust OWNERTRUST.txt Check the result:\ngpg --list-secret-keys --with-subkey-fingerprint # sec# ed25519/0x... [C] ← the # means \"master secret not present\" # ssb ed25519/... [S] # ssb ky1024_cv448/.. [E] # ssb ed25519/... [A] That # is what all of the above was for. Your daily laptop now carries nothing but disposable subkeys, so revoking them after a theft costs you a few minutes instead of your identity.\nThree backup layers One backup is no backup. So I run three independent paths, and each one assumes the other two have already failed when I reach for it.\nHot - VeraCrypt volume on multiple clouds. Always reachable, survives a lost laptop.\nveracrypt --text --create cloud-backup/gpg-vault.vc \\ --volume-type=normal --size=128M \\ --encryption=AES-Twofish-Serpent --hash=Whirlpool \\ --filesystem=FAT --pim=0 --keyfiles=\"\" \\ --random-source=/dev/urandom sudo veracrypt --text --mount cloud-backup/gpg-vault.vc /tmp/vault --pim=0 --keyfiles=\"\" sudo cp MASTER-FULL.asc SUBKEYS-ONLY.asc PUBLIC.asc OWNERTRUST.txt \\ revocation-$KEYID.asc KEY_INFO.txt /tmp/vault/ sync \u0026\u0026 sudo veracrypt --text --dismount /tmp/vault rclone copy cloud-backup/gpg-vault.vc gdrive:SecureBackups/ rclone copy cloud-backup/gpg-vault.vc proton:SecureBackups/ Give the VeraCrypt volume a different passphrase than your GPG key, so one leak doesn’t knock over the other. The AES-Twofish-Serpent cascade with an Argon2-derived key holds up against a quantum computer too. Grover’s algorithm halves the effective symmetric key length, and you still walk away with roughly 128 bits of margin.\nCold - paperkey + QR codes. Survives cloud outage and account bans.\nPaperkey strips the redundant structure out of a secret key and keeps only the bytes that matter, which is what you want on archival paper. It handles Ed25519 cleanly. It can’t do Kyber yet, since that’s too new and there’s no compact encoding for it, so the master goes through paperkey while the full armored export goes through QR codes, split across several PNGs with high error correction.\n# Paperkey for the master paperkey --secret-key \u003c(gpg --export-secret-keys \"$KEYID\") \\ --output master.paperkey.txt # QR stack for the full bundle split -b 1500 -d -a 2 MASTER-FULL.asc part_ i=1; total=$(ls part_* | wc -l) for f in part_*; do qrencode -l H -s 10 -m 4 \\ -o \"qr-$(printf '%02d' $i)-of-$(printf '%02d' $total).png\" \u003c \"$f\" i=$((i+1)) done Test your restore before you trust it:\nfor f in qr-*.png; do zbarimg --raw --quiet \"$f\" \u003e\u003e combined.asc; done diff combined.asc MASTER-FULL.asc \u0026\u0026 echo \"restore works\" Print on archival paper with a laser printer, because inkjet ink fades inside a decade and outlasting that is the point. Stick the stack in a fire-resistant box, or split it across two locations if you’re feeling thorough.\nHardware - YubiKey (optional). This covers the ugly case where your backups leak but the physical token is still in your pocket. In 2026 YubiKey firmware still doesn’t do PQC subkeys, so only the Ed25519 sign and auth subkeys go on the token while the Kyber encrypt subkey stays in software. Yubico’s roadmap pencils in PQC for firmware 6 in 2027, so this gap should close.\nPer-employer identities Client and employer work gives you two scenarios, and it’s a choice worth making on purpose.\nScenario A - add a UID to the existing masterkey. Easy, and it proves cryptographically that the same person stands behind all your identities. The catch is that every colleague who imports your employer public key also gets a full list of your personal aliases. Fine for tom@byteherder.com, which is my public identity anyway. A real problem for kapott@pm.me.\nScenario B - separate key per employer. One more key to babysit, but the separation is clean. You can still cross-sign it with your main master when you want the attribution, and if you don’t, the identities stay cryptographically unrelated. This is my default whenever privacy per context matters.\nexport GNUPGHOME=~/gpg-dsm mkdir -p \"$GNUPGHOME\" \u0026\u0026 chmod 700 \"$GNUPGHOME\" gpg --expert --full-generate-key # repeat the §6–§8 flow Keep a separate VeraCrypt volume per identity (dsm-vault.vc, clientX-vault.vc). Same three-layer backup as before, with the blast radius walled off per client.\nDaily use Publish the public key. Keys.openpgp.org makes you verify each UID by email before it publishes anything, which doubles as a check that you typed the addresses right:\ngpg --keyserver hkps://keys.openpgp.org --send-keys \"$KEYID\" Git signing. Pin your commits to the signing subkey. The trailing ! forces GnuPG to use this specific subkey instead of picking one for you:\nSIGNKEY=$(gpg --list-secret-keys --with-colons \"$KEYID\" \\ | awk -F: '/^ssb/ \u0026\u0026 $12~/s/ {print $5; exit}') git config --global user.signingkey \"$SIGNKEY!\" git config --global commit.gpgsign true git config --global tag.gpgsign true Per-repo overrides let you sign employer commits with the employer key:\ngit config user.email tom.meurs@dsm.com git config user.signingkey \"DSM_SIGNKEY!\" SSH via gpg-agent. Your auth subkey becomes your SSH key:\ngpg --list-secret-keys --with-keygrip \"$KEYID\" # note the A-subkey keygrip echo \"\u003cKEYGRIP\u003e\" \u003e\u003e ~/.gnupg/sshcontrol # In your shell rc export GPG_TTY=$(tty) export SSH_AUTH_SOCK=$(gpgconf --list-dirs agent-ssh-socket) gpgconf --launch gpg-agent ssh-add -L # paste this line in GitHub → SSH keys Email. Once all three addresses point at the same master key, Thunderbird matches the UID to your From: header on its own. Mutt does the same once you set pgp_default_key = $KEYID.\nRotation Every eighteen months, comfortably before the two-year subkey expiry lands, I bring the master back for a few minutes and push the expiry out again. It’s the same VeraCrypt-mount dance as the backups:\nsudo veracrypt --text --mount ~/gpg-build/cloud-backup/gpg-vault.vc /tmp/vault export GNUPGHOME=$(mktemp -d) \u0026\u0026 chmod 700 \"$GNUPGHOME\" gpg --import /tmp/vault/MASTER-FULL.asc gpg --expert --edit-key \"$KEYID\" # key 1, key 2, key 3 → expire → 2y → save gpg --export-secret-subkeys --armor \"$KEYID\" \u003e ~/SUBKEYS-NEW.asc rm -rf \"$GNUPGHOME\" \u0026\u0026 unset GNUPGHOME sudo veracrypt --text --dismount /tmp/vault GNUPGHOME=~/.gnupg gpg --import ~/SUBKEYS-NEW.asc gpg --keyserver hkps://keys.openpgp.org --send-keys \"$KEYID\" If a subkey ever leaks, it’s the same flow with one extra step. revkey the compromised subkey, export, publish. The master never moved, so your identity is untouched.\nThe broader principle None of this is minimal, and I’m not going to pretend otherwise. Seven Diceware words, three backup layers, a recurring ritual, a couple of USB sticks living in different drawers. You could skip all of it, sign your commits with some SaaS “identity service”, and never read another word about keyservers.\nThe catch is that a SaaS identity belongs to the provider. They can revoke it or lose it in an incident. They can be served a subpoena and hand it over. The day they pivot their business model you migrate on their clock instead of yours. I went through this control-versus-convenience question in Sovereign Infrastructure. Owning the root costs me more time than a subscription would. What I get for it is an identity that doesn’t live inside a system I can’t open up and inspect.\nAttribution is the same argument, and it’s already live. When text, voice and video can all be faked on hardware anyone can buy, the honest default for any message is “could be anyone.” Signing your commits won’t stop someone churning out garbage in your name. It gives everyone who cares a way to check, and that check holds even when the forgery is otherwise perfect. The check only exists if you were already signing before the forgery turned up, so start now.\nQuantum-safety is the same argument stretched across decades. I’m not expecting a quantum computer to read my mail any time soon. I am expecting a patient adversary to record my encrypted traffic today and bet on the hardware showing up later. The only counter is to stop feeding them ciphertext that’ll one day be cheap to crack, which means post-quantum encryption now, rough tooling and all.\nMy master has been offline since the day I generated it. The subkeys on this laptop expire in two years, and at eighteen months I bring the master back to push that out. The rest is paper in a box and a VeraCrypt file on two clouds. That’s what it took to make a key I expect to still work in 2046.","permalink":"https://tommeurs.nl/en/posts/quantum-safe-gpg-identity/","title":"Quantum-safe GPG identity with multiple aliases"},{"content":" TL;DR Most default alert rules page on a number with no user behind it, and after a week of that you stop reading pages. I alert on error rate, P99 latency, a disk that fills within 24 hours and a certificate under seven days, about ten rules in total, each with a runbook. Alertmanager sends critical to the pager, warning to Slack during office hours, info nowhere. Everything that doesn’t need a human gets deleted.\nFor a while my alerting was fine. A handful of rules, pages were rare, and when one came in it meant something. Then the cluster grew, I bolted on the Prometheus Operator defaults, and “fine” turned into noise without me noticing.\nThe page that did it came in at 3 AM. “High CPU usage on node-worker-3.” I looked at the graph, saw the node had been sitting at 75% for ten minutes, and went back to sleep. Next night, same alert. A week later I’d stopped opening them at all.\nThat’s alert fatigue, and it’s the dangerous kind of broken because it looks like it’s working. The pages keep coming and you keep ignoring them. When everything pages, a real incident lands on the same pile, and the first person to notice is a user.\nSo which alerts deserve to wake someone up? That’s the only question in this post, and the tooling has little to do with the answer. I’ve run systems on both ends, drowning in pages and systems where a page is rare enough that I trust it the second it fires. The difference between the two is how you decide what gets to interrupt a person.\nIt’s also the value I keep coming back to on this blog, understanding over trust. An alert whose meaning you can’t explain is a black box wired to your pager. You can’t reason about it, so you learn to ignore it.\nWhy default alerts make it worse Most monitoring setups arrive with alerting rules pre-loaded. The Prometheus Operator ships hundreds of them, and they fire on everything:\nCPU \u003e 80% Memory \u003e 80% Disk \u003e 80% Pod restarts \u003e 0 Any 5xx error I understand why they ship. Four things still go wrong:\nNo context. Is 80% CPU bad? Depends on the workload. A batch job pinning a core is doing its job. No impact. Is a user affected right now? The alert has no idea, and neither do you, half awake. No action. What am I supposed to do about it? Usually nothing. Too sensitive. A short spike fires an alert that resolves itself before you’ve unlocked your phone. You end up with pages that don’t need a human, and humans who’ve learned not to trust pages. The two feed each other.\nHow I decide what gets to page me A better tool wouldn’t have fixed this. What fixed it was a few rules about what an alert is allowed to be.\n1. Alert on symptoms, not causes Bad: “Pod restarted” Good: “Service error rate \u003e 1%”\nNobody using your service cares whether a pod restarted. They care whether the thing works. So alert on what the user experiences, and go chase the cause after you’ve been woken up for a real reason. A pod that crash-loops while the service stays healthy, because the others picked up the load, is the self-healing I set the cluster up for. I don’t want a page for it.\n2. Alert only on what needs a hand If nobody has to do anything, it has no business paging anyone. Informational stuff goes on a dashboard or into a daily review. Not to your phone at night.\nThe test I use is one question. “If I get this alert, what do I do?”\nAnswer is “investigate and probably nothing” → not an alert Answer is “this specific remediation step” → fine, that’s an alert 3. Every alert needs a runbook When the thing fires, what do you do about it? If you can’t write that down ahead of time, the alert isn’t finished. This is the low-friction principle applied to the version of me that’s half asleep. That version shouldn’t have to reverse-engineer what the daytime version meant.\nannotations: runbook_url: https://wiki.example.com/runbooks/high-error-rate summary: \"Error rate exceeds SLO\" description: \"Service {{ $labels.service }} error rate is {{ $value }}%\" 4. Severity has to mean something Severity Response Example critical Wake someone up NOW Service down, data loss imminent warning Handle during business hours Disk filling, certificate expiring info Review in daily standup Unusual pattern, non-urgent The first time a critical fires that didn’t need anyone out of bed, you’ve taught yourself that critical doesn’t mean critical. A few of those and the whole scale is worthless. So I guard that label.\nTie alerts to an SLO instead of a threshold The alerts I trust most map onto a Service Level Objective. The SLO already says how much badness is acceptable, so the alert inherits that meaning instead of guessing at a threshold.\n# SLO: 99.9% availability # Error budget: 0.1% = 43 minutes/month groups: - name: slo-alerts rules: # Fast burn: consuming error budget quickly - alert: HighErrorRateFastBurn expr: | ( sum(rate(http_requests_total{status=~\"5..\"}[5m])) / sum(rate(http_requests_total[5m])) ) \u003e 0.01 for: 2m labels: severity: critical annotations: summary: \"Error rate exceeds 1% (fast burn)\" description: \"At this rate, monthly error budget exhausts in {{ $value | humanizeDuration }}\" # Slow burn: steadily consuming error budget - alert: HighErrorRateSlowBurn expr: | ( sum(rate(http_requests_total{status=~\"5..\"}[1h])) / sum(rate(http_requests_total[1h])) ) \u003e 0.001 for: 1h labels: severity: warning annotations: summary: \"Error rate elevated (slow burn)\" The two burn rates do different jobs:\nFast burn means you’re torching the error budget right now. Wake someone up. Slow burn means you’re drifting toward an SLO breach over hours. Deal with it during work time. The alerts I run Here’s the concrete stuff. Not the whole catalogue, just the categories that have earned a place on my pager.\nAvailability # Service is down - alert: ServiceDown expr: up{job=\"my-service\"} == 0 for: 1m labels: severity: critical annotations: summary: \"Service {{ $labels.job }} is down\" runbook_url: https://wiki/runbooks/service-down # High error rate (user impact) - alert: HighErrorRate expr: | sum by (service) (rate(http_requests_total{status=~\"5..\"}[5m])) / sum by (service) (rate(http_requests_total[5m])) \u003e 0.05 for: 5m labels: severity: critical annotations: summary: \"{{ $labels.service }} error rate above 5%\" Latency # P99 latency too high - alert: HighLatency expr: | histogram_quantile(0.99, sum by (le, service) (rate(http_request_duration_seconds_bucket[5m])) ) \u003e 2 for: 10m labels: severity: warning annotations: summary: \"{{ $labels.service }} P99 latency above 2s\" Capacity These two are predictive, which is the point. I’d rather get a warning during office hours that a disk fills tomorrow than a critical page at midnight because it already did.\n# Disk filling (predictive) - alert: DiskFillingUp expr: | predict_linear(node_filesystem_avail_bytes{fstype!=\"tmpfs\"}[6h], 24*60*60) \u003c 0 for: 1h labels: severity: warning annotations: summary: \"Disk {{ $labels.mountpoint }} will fill within 24h\" # Certificate expiring - alert: CertificateExpiringSoon expr: | certmanager_certificate_expiration_timestamp_seconds - time() \u003c 7*24*60*60 for: 1h labels: severity: warning annotations: summary: \"Certificate {{ $labels.name }} expires in less than 7 days\" What I deliberately don’t alert on This list matters as much as the one above. Every rule in it is something a default config will happily set up for you, and every one of them is noise.\n# DON'T: Generic resource usage - alert: HighCPU expr: node_cpu_seconds_total \u003e 0.8 # Bad: no context # DON'T: Expected behavior - alert: PodRestart expr: increase(kube_pod_container_status_restarts_total[1h]) \u003e 0 # Bad: restarts are normal # DON'T: Things that auto-resolve - alert: PodPending expr: kube_pod_status_phase{phase=\"Pending\"} == 1 for: 1m # Bad: too short, scheduler will handle it Routing it all through Alertmanager Deciding what’s worth a page is half the work. The other half is getting the page to the right place at the right time. Critical goes to the pager, warning goes to Slack and only during work hours, info goes to a log and nowhere else.\n# alertmanager.yaml global: resolve_timeout: 5m route: receiver: 'default' group_by: ['alertname', 'severity'] group_wait: 30s group_interval: 5m repeat_interval: 4h routes: # Critical: page immediately - match: severity: critical receiver: 'pagerduty' continue: true # Warning: Slack during work hours - match: severity: warning receiver: 'slack-ops' active_time_intervals: - work_hours # Info: just log - match: severity: info receiver: 'null' receivers: - name: 'pagerduty' pagerduty_configs: - service_key: '\u003ckey\u003e' severity: '{{ .CommonLabels.severity }}' - name: 'slack-ops' slack_configs: - api_url: '\u003cwebhook\u003e' channel: '#ops-alerts' title: '{{ .CommonAnnotations.summary }}' text: '{{ .CommonAnnotations.description }}' - name: 'null' time_intervals: - name: work_hours time_intervals: - weekdays: ['monday:friday'] times: - start_time: '09:00' end_time: '18:00' Silencing and inhibition Two more ways to keep the volume down. Silence what you already know about, and suppress the alerts that are downstream symptoms of a bigger one.\nSilences for maintenance # Create silence during maintenance amtool silence add alertname=~\".+\" --duration=2h --comment=\"Planned maintenance\" # Silence specific service amtool silence add service=\"api\" --duration=1h --comment=\"Deploying v2.3.0\" Inhibition rules When one alert implies a pile of others, suppress the pile. If the whole cluster is down, I don’t need forty separate pages telling me each service on it is down too.\ninhibit_rules: # If cluster is down, don't alert on individual services - source_match: alertname: 'ClusterDown' target_match_re: alertname: '.+' equal: ['cluster'] # If node is down, don't alert on pods on that node - source_match: alertname: 'NodeDown' target_match_re: alertname: 'Pod.+' equal: ['node'] Surfacing alert state in Grafana Not every alert should page, but you still want to see them. The ones that don’t wake you up live on a dashboard. That’s where info and recently resolved alerts go to get reviewed instead of ignored.\n# Prometheus datasource query ALERTS{alertstate=\"firing\"} Create an alerts overview panel showing:\nCurrently firing alerts Recent alert history Alert trends over time Link from alerts to the relevant dashboard:\nannotations: dashboard_url: https://grafana/d/abc123?var-service={{ $labels.service }} The human side of on-call All the config in the world doesn’t help if the rotation around it is a mess. A few things I’ve found worth writing down.\nRotation structure Primary → First responder, handles all pages Secondary → Backup if primary doesn't respond in 10 min Escalation → Management, for major incidents Response time SLAs Severity Acknowledge Start Work Critical 5 minutes 15 minutes Warning 4 hours 8 hours Info Next standup When convenient Post-incident review Every critical alert is worth a quick look afterwards, even when the fix was easy:\nWas the alert necessary? Did it give enough context? Was the runbook helpful? Could the underlying thing be prevented? If the same alert keeps firing and the answer is always “do nothing”, the alert is broken. Fix it or delete it.\nWhat my setup looks like # Core SLO alerts only groups: - name: slo rules: # Availability: error rate - alert: HighErrorRate expr: error_rate \u003e 0.01 for: 5m severity: critical # Latency: P99 - alert: HighLatency expr: p99_latency \u003e 2 for: 10m severity: warning # Capacity: predictive - alert: DiskFillingUp expr: disk_will_fill_24h severity: warning # Security: cert expiry - alert: CertExpiring expr: cert_expires_7d severity: warning # That's it. ~10 alert rules total. About ten rules, and every one of them earns its keep:\nSLO-based, so each alert is tied to something a user would feel. Minimal, because a rule I can’t justify is a rule I delete. Clear severity. Critical means get out of bed, warning means it waits until morning. Runbook on everything, so half-asleep me has a next step that was written down in advance. Digging out if you’re already buried If you’re reading this from inside a pile of noisy alerts, you don’t fix it by adding more. You fix it by deleting. Roughly the order I’d go in:\nAudit what you have. How many alerts fired last month, and how many of those led to anyone doing anything? Delete the dead ones. If a rule hasn’t fired in 90 days, you probably don’t need it, and if it has fired but nobody acted, you definitely don’t. Raise the thresholds that always auto-resolve. If an alert clears itself before you respond, it was set too tight. Add a for duration so brief spikes can’t page you. Merge the near-duplicates. One “high error rate” beats forty per-endpoint variations of the same thing. Every page needs a human. If it doesn’t, take it off the pager.\nWhy I bother with all this A noisy alerting setup looks healthy while it rots your reliability. The engineer on call learns the pages are mostly junk, starts skimming them, and one night skims past the one that mattered. That’s the failure I’m designing against, and no tool fixes it, because the tool did exactly what it was told.\nSo I keep the volume low enough that every page still gets read, and I only page on things a user would feel.\nThe observability stack with Loki and Tempo produces far more data than a person can watch. Alerting decides which slice of it gets to interrupt someone. Let too much through and you’re back where I started, ignoring your own phone at 3 AM.\nThe CPU rule that woke me up was the first one I deleted. What’s left is the ten rules above, each with a runbook I wrote while awake.","permalink":"https://tommeurs.nl/en/posts/alerting-that-works/","title":"Alerting That Works: From Alert Fatigue to Actionable Notifications"},{"content":" TL;DR cert-manager watches Certificate resources, fetches the certificate from Let’s Encrypt, your own CA or Vault, stores it in a Secret and renews it 30 days before it expires. I run one DNS-01 ClusterIssuer through Cloudflare and one wildcard certificate, copy that Secret into the namespaces that need it, and let two Prometheus alerts tell me when a renewal is stuck. Nothing left to remember.\nFor years my certificates renewed the way most people’s do. A calendar reminder, a manual certbot run, and the hope that I’d get to it before the thing expired. It worked, right up to the morning a service started throwing cert errors at me and I had no idea why. The renewal cron had been failing for weeks. Quietly.\nThat’s the part of manual TLS nobody warns you about. The failure doesn’t announce itself. The cert just expires, usually at a bad moment, and you find out because a browser is yelling at someone. Meanwhile the knowledge of how renewal works lives in one person’s head, and teams skip HTTPS on internal services because wiring it up by hand is annoying enough to put off.\nI wanted the cluster to own its certificates so I wouldn’t have to. cert-manager does that. You declare which certificates you need, it handles issuance, renewal and the Kubernetes Secrets they live in, and you stop looking at expiry dates.\nIt’s one of the first things I install in a new cluster, right after the CNI .\nHow cert-manager works External\nKubernetes Cluster\nwatches\ncreates\nACME protocol\nDNS challenge\nmounts\ncert-manager\nCertificate\nResource\nTLS Secret\nIngress\nLet's Encrypt\nACME Server\nDNS Provider\nThe loop is simple once you see it:\nYou create a Certificate resource cert-manager requests a certificate from the issuer (Let’s Encrypt, Vault, etc.) cert-manager completes the challenge (HTTP-01 or DNS-01) to prove you control the domain cert-manager stores the certificate in a Kubernetes Secret Your Ingress or Gateway mounts that Secret for TLS Renewal happens on its own, 30 days before expiry. No calendar.\nInstallation helm repo add jetstack https://charts.jetstack.io helm repo update helm install cert-manager jetstack/cert-manager \\ --namespace cert-manager \\ --create-namespace \\ --set installCRDs=true The --set installCRDs=true matters. Skip it and cert-manager’s custom resources never get registered, and the errors you get later don’t point at that. If you run everything through GitOps with ArgoCD like I do, the Application looks like this:\napiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: cert-manager namespace: argocd spec: project: default source: repoURL: https://charts.jetstack.io chart: cert-manager targetRevision: v1.14.0 helm: values: | installCRDs: true prometheus: servicemonitor: enabled: true destination: server: https://kubernetes.default.svc namespace: cert-manager syncPolicy: syncOptions: - CreateNamespace=true Issuers An Issuer tells cert-manager where certificates come from. There are two scopes, and the difference trips people up:\nIssuer: works in one namespace ClusterIssuer: works across all namespaces I default to ClusterIssuer unless I’ve got a real reason to keep a certificate authority inside one namespace. One issuer instead of a copy per namespace.\nLet’s Encrypt (ACME) The setup most people reach for first, free certificates from Let’s Encrypt:\napiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: letsencrypt-prod spec: acme: server: https://acme-v02.api.letsencrypt.org/directory email: admin@example.com privateKeySecretRef: name: letsencrypt-prod-account-key solvers: - http01: ingress: class: nginx Start with the staging issuer. Let’s Encrypt’s rate limits are strict, and if you’re iterating on config you’ll hit them. I learned that the slow way, locked out for a week after fat-fingering a solver config a few times too many. Staging hands you untrusted certs through the exact same flow, so you can break things there first:\napiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: letsencrypt-staging spec: acme: server: https://acme-staging-v02.api.letsencrypt.org/directory email: admin@example.com privateKeySecretRef: name: letsencrypt-staging-account-key solvers: - http01: ingress: class: nginx HTTP-01 vs DNS-01 challenges The challenge is how Let’s Encrypt checks that you control the domain before it hands you a cert. There are two ways to prove it, and the one you pick shapes everything downstream.\nHTTP-01 has Let’s Encrypt fetch a file from /.well-known/acme-challenge/ on your domain:\nsolvers: - http01: ingress: class: nginx It works for anything reachable on port 80 from the public internet. Quick to set up, nothing extra to configure. That reachability is also the catch, because Let’s Encrypt has to be able to fetch the file, and an internal service can’t hand it one.\nDNS-01 has Let’s Encrypt look for a DNS TXT record instead:\nsolvers: - dns01: cloudflare: email: admin@example.com apiTokenSecretRef: name: cloudflare-api-token key: api-token DNS-01 is the one I end up using most, because it covers what HTTP-01 can’t:\nWildcard certificates (*.example.com), which HTTP-01 flat out cannot issue Internal services that never touch the public internet Split-horizon DNS setups Cloudflare DNS-01 setup # API Token Secret apiVersion: v1 kind: Secret metadata: name: cloudflare-api-token namespace: cert-manager type: Opaque stringData: api-token: \"your-cloudflare-api-token\" --- apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: letsencrypt-cloudflare spec: acme: server: https://acme-v02.api.letsencrypt.org/directory email: admin@example.com privateKeySecretRef: name: letsencrypt-cloudflare-account solvers: - dns01: cloudflare: apiTokenSecretRef: name: cloudflare-api-token key: api-token The Cloudflare API token needs Zone:DNS:Edit, scoped to the zones you use. Don’t reach for the global API key here. A scoped token that leaks costs you the DNS of one zone. The global key costs you the account.\nPrivate CA Public certificates are fine until you have services that should never be reachable from the public internet at all. For those you run your own CA, and the whole trust chain stays inside infrastructure you control:\n# Create CA key pair apiVersion: v1 kind: Secret metadata: name: ca-key-pair namespace: cert-manager type: kubernetes.io/tls data: tls.crt: \u003cbase64-encoded-ca-cert\u003e tls.key: \u003cbase64-encoded-ca-key\u003e --- apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: internal-ca spec: ca: secretName: ca-key-pair Or let cert-manager create a self-signed CA:\napiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: selfsigned-issuer spec: selfSigned: {} --- apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: my-ca namespace: cert-manager spec: isCA: true commonName: my-ca secretName: my-ca-secret privateKey: algorithm: ECDSA size: 256 issuerRef: name: selfsigned-issuer kind: ClusterIssuer --- apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: my-ca-issuer spec: ca: secretName: my-ca-secret HashiCorp Vault If you already run Vault, its PKI engine makes a solid issuer. Short-lived certs, one audit log, and one place to revoke from:\napiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: vault-issuer spec: vault: server: https://vault.vault:8200 path: pki/sign/my-role auth: kubernetes: role: cert-manager mountPath: /v1/auth/kubernetes secretRef: name: cert-manager-vault-token key: token Requesting certificates Once an issuer exists, there are two ways to get a certificate out of it. Ask for one explicitly, or let cert-manager infer it from an Ingress. I use both, depending on how much control I want over the details.\nCertificate resource The explicit route. You spell out what you want:\napiVersion: cert-manager.io/v1 kind: Certificate metadata: name: api-example-com namespace: production spec: secretName: api-example-com-tls issuerRef: name: letsencrypt-prod kind: ClusterIssuer dnsNames: - api.example.com - api-v2.example.com duration: 2160h # 90 days renewBefore: 720h # Renew 30 days before expiry Ingress annotation (automatic) The lazy route, in the good sense. One annotation on an Ingress and cert-manager does the rest:\napiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: api annotations: cert-manager.io/cluster-issuer: letsencrypt-prod spec: tls: - hosts: - api.example.com secretName: api-example-com-tls rules: - host: api.example.com http: paths: - path: / pathType: Prefix backend: service: name: api port: number: 80 cert-manager spots the annotation, generates the Certificate resource for you and fills the Secret. For most app deployments that’s all you need, and the TLS config sits next to the routing config, which is where I’d go looking for it anyway.\nGateway API Moving to the Gateway API instead of Ingress? Same idea, different resource:\napiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: main annotations: cert-manager.io/cluster-issuer: letsencrypt-prod spec: gatewayClassName: cilium listeners: - name: https port: 443 protocol: HTTPS hostname: \"*.example.com\" tls: mode: Terminate certificateRefs: - name: wildcard-example-com-tls Wildcard certificates A wildcard cert covers every subdomain under one name, so you get one certificate instead of a fresh one per service. That alone keeps you well clear of Let’s Encrypt’s rate limits. The price is DNS-01, since HTTP-01 can’t validate a wildcard:\napiVersion: cert-manager.io/v1 kind: Certificate metadata: name: wildcard-example-com namespace: production spec: secretName: wildcard-example-com-tls issuerRef: name: letsencrypt-cloudflare kind: ClusterIssuer dnsNames: - \"*.example.com\" - \"example.com\" List both *.example.com and example.com. The wildcard doesn’t cover the apex domain, and forgetting that line is how you end up serving a broken cert on your root domain.\nCross-namespace Secrets Here’s a wrinkle that catches people. A certificate Secret lives in exactly one namespace, and you often need the same cert in several. A Secret in cert-manager can’t be mounted by a pod in production. So how do you share one cert across namespaces without minting a separate one each time? Two ways.\nReflector (external controller) kubernetes-reflector copies a Secret into the namespaces you list:\napiVersion: cert-manager.io/v1 kind: Certificate metadata: name: shared-cert namespace: cert-manager annotations: reflector.v1.k8s.emberstack.com/reflection-allowed: \"true\" reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces: \"production,staging\" spec: secretName: shared-cert-tls # ... trust-manager For CA bundles specifically, cert-manager ships its own answer. trust-manager pushes a trust bundle into ConfigMaps across namespaces, which is what you want when the thing you’re sharing is the CA cert rather than a leaf certificate:\nhelm install trust-manager jetstack/trust-manager \\ --namespace cert-manager apiVersion: trust.cert-manager.io/v1alpha1 kind: Bundle metadata: name: my-ca-bundle spec: sources: - secret: name: my-ca-secret key: ca.crt target: configMap: key: ca-bundle.crt namespaceSelector: matchLabels: trust-bundle: enabled Monitoring and troubleshooting Automation is great until it fails without telling you, which is the exact trap I opened this post with. So cert-manager has to be observable, and a stuck renewal has to become an alert instead of an outage. Here’s how I poke at it when something looks off.\nCheck certificate status kubectl get certificates -A kubectl describe certificate api-example-com -n production Check certificate requests kubectl get certificaterequests -A kubectl describe certificaterequest api-example-com-xyz -n production Check orders and challenges kubectl get orders -A kubectl get challenges -A # Debug a stuck challenge kubectl describe challenge api-example-com-xyz-123 -n production When a cert misbehaves, you walk down the chain. Certificate, then CertificateRequest, then Order, then Challenge. The error is almost always in one of those describe outputs, and it usually surfaces at the Challenge.\nPrometheus metrics Manual checks are for debugging. For the safety net you want metrics. cert-manager exports them, and these are the two alerts I rely on:\n# Alert on expiring certificates - alert: CertificateExpiringSoon expr: certmanager_certificate_expiration_timestamp_seconds - time() \u003c 604800 for: 1h labels: severity: warning annotations: summary: \"Certificate {{ $labels.name }} expires in less than 7 days\" # Alert on failed renewals - alert: CertificateRenewalFailed expr: certmanager_certificate_ready_status{condition=\"False\"} == 1 for: 1h labels: severity: critical annotations: summary: \"Certificate {{ $labels.name }} is not ready\" That first alert is the one that would’ve saved me in my certbot-cron days. Seven days’ warning is plenty of room to fix the real problem before anything user-facing breaks.\nCommon issues Challenge stuck pending:\nHTTP-01: Ingress not routing to cert-manager solver DNS-01: API credentials wrong or missing permissions Rate limited:\nToo many certificates requested from Let’s Encrypt Use staging issuer for testing Consolidate into wildcard certificates Secret not updating:\nCheck Certificate resource status Check CertificateRequest and Order resources Look at cert-manager logs: kubectl logs -n cert-manager deploy/cert-manager Best practices A handful of habits that came out of running this for a while:\n1. Use ClusterIssuers Unless you need namespace isolation, ClusterIssuers cut down duplication:\napiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: letsencrypt-prod # Available to all namespaces 2. Separate staging and production Always have both issuers:\n# letsencrypt-staging for testing # letsencrypt-prod for production 3. Monitor expiration Even with automation, monitor as a safety net:\ncertmanager_certificate_expiration_timestamp_seconds - time() \u003c 604800 4. Use DNS-01 for internal services If your services aren’t publicly reachable, HTTP-01 won’t work.\n5. Standardize Secret names A convention like ${service}-tls makes it predictable:\nsecretName: api-example-com-tls My production setup Here’s what I run. One DNS-01 ClusterIssuer and one wildcard certificate, and almost everything flows from those two resources:\n# ClusterIssuer for Let's Encrypt with Cloudflare DNS apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: letsencrypt-prod spec: acme: server: https://acme-v02.api.letsencrypt.org/directory email: certs@example.com privateKeySecretRef: name: letsencrypt-prod solvers: - dns01: cloudflare: apiTokenSecretRef: name: cloudflare-api-token key: api-token selector: dnsZones: - \"example.com\" --- # Wildcard certificate for all services apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: wildcard-example-com namespace: cert-manager spec: secretName: wildcard-example-com-tls issuerRef: name: letsencrypt-prod kind: ClusterIssuer dnsNames: - \"*.example.com\" - \"example.com\" Why these four:\nDNS-01 via Cloudflare handles everything, wildcards included, so I never have to think about whether a service is publicly reachable A wildcard certificate means one cert for every subdomain and effectively zero rate limit worry A central namespace keeps the certificate in cert-manager and reflects it out to where it’s needed Prometheus alerts so I find out about a problem days before a user does Why this matters TLS stopped being optional a while ago. Browsers warn on plain HTTP, APIs reject it, and internal traffic needs encryption if you take zero trust at all seriously.\nThe deeper reason I care is friction. Anything that depends on me remembering to do a thing will fail at some point, because at some point I’ll forget. Manual certificate renewal is that kind of system with a calendar reminder taped to it. With cert-manager I define a certificate once and the cluster renews it, the same way for every service and every team.\nThe certbot cron is gone. The wildcard above renews itself 30 days before it expires, and if a renewal ever sticks, the alert reaches me seven days before a browser would.","permalink":"https://tommeurs.nl/en/posts/cert-manager-kubernetes-tls/","title":"cert-manager: Automatic TLS Certificates in Kubernetes"},{"content":" TL;DR Cilium is the CNI I run in every cluster. It replaces kube-proxy and its iptables chains with eBPF programs in the kernel, enforces network policies on identity and on HTTP paths, and Hubble shows me every flow and every drop. WireGuard between the pods, BGP to the switch, no sidecars. My production values are at the bottom.\nThe first time a service stopped resolving in one of my clusters, I spent an evening reading iptables chains. Hundreds of rules, generated by kube-proxy, evaluated top to bottom. I never found the problem. I restarted the node and it went away.\nThat bothered me more than the outage did. I was running something I couldn’t read.\nIt’s why I moved to Cilium. It uses eBPF to push the networking logic into the Linux kernel and skips iptables entirely. It’s faster, you can see what your traffic does, and a network policy stops being a guess.\nThis is what I run in my clusters now. I’ll start with the smallest piece that works and add to it, so you can stop wherever you have enough.\nWhat eBPF is Start here, because the rest only makes sense once this clicks. eBPF, extended Berkeley Packet Filter, lets you run sandboxed programs inside the Linux kernel without patching the kernel source or loading a module.\nKernel Space\nUser Space\nloads\nApplication\nCilium Agent\neBPF Programs\nNetwork Stack\nKernel Hooks\n(XDP, TC, Socket)\nTraditional networking leans on iptables, a chain of rules walked one by one until something matches. An eBPF program gets compiled and runs in the kernel, and it looks things up in maps instead of walking a list. The difference shows up when you scale it.\nIptables with 10,000 services means a linear scan that gets slower with every service you add. Cilium with 10,000 services does one constant-time map lookup. Same workload, very different behaviour under load. And when something is off, I can dump the maps instead of guessing which of a few hundred rules misfired.\nInstalling Cilium The smallest version that works. Swap your existing CNI for Cilium and check that it came up.\n# Install Cilium CLI CILIUM_CLI_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt) curl -L --fail --remote-name-all https://github.com/cilium/cilium-cli/releases/download/${CILIUM_CLI_VERSION}/cilium-linux-amd64.tar.gz sudo tar xzvfC cilium-linux-amd64.tar.gz /usr/local/bin rm cilium-linux-amd64.tar.gz # Install Cilium in cluster cilium install --version 1.15.0 # Verify installation cilium status --wait The CLI is fine for a first look, but I don’t run anything by hand twice. Here’s the same install as an ArgoCD Application, so the network layer lives in Git like everything else in the cluster:\napiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: cilium namespace: argocd spec: project: default source: repoURL: https://helm.cilium.io/ chart: cilium targetRevision: 1.15.0 helm: values: | kubeProxyReplacement: true k8sServiceHost: \"10.0.0.1\" k8sServicePort: \"6443\" hubble: enabled: true relay: enabled: true ui: enabled: true operator: replicas: 2 ipam: mode: kubernetes destination: server: https://kubernetes.default.svc namespace: kube-system kube-proxy replacement It’s the first thing I turn on after the install. Cilium can replace kube-proxy entirely and implement Kubernetes Services in eBPF. It’s how I got rid of the iptables chains this post started with:\n# Cilium values kubeProxyReplacement: true k8sServiceHost: \"10.0.0.1\" # API server IP k8sServicePort: \"6443\" What you get:\nNo iptables rules for services Direct server return, which is faster Socket-level load balancing, where the decision happens once at connect() instead of on every packet Maglev hashing, so backend selection stays consistent Check that it’s on:\nkubectl -n kube-system exec ds/cilium -- cilium status | grep KubeProxyReplacement # KubeProxyReplacement: True [eth0 10.0.0.10 (Direct Routing)] Network policies Now the part that changes how you think about security. Cilium speaks the standard Kubernetes NetworkPolicy, and it adds its own CiliumNetworkPolicy on top with a lot more reach. If you’ve never written a policy, start with the standard one.\nKubernetes NetworkPolicy apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: api-ingress namespace: production spec: podSelector: matchLabels: app: api policyTypes: - Ingress ingress: - from: - podSelector: matchLabels: app: frontend ports: - protocol: TCP port: 8080 CiliumNetworkPolicy, the extended one Once that makes sense, the native policy is where it gets interesting:\napiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: api-policy namespace: production spec: endpointSelector: matchLabels: app: api ingress: - fromEndpoints: - matchLabels: app: frontend toPorts: - ports: - port: \"8080\" protocol: TCP rules: http: - method: \"GET\" path: \"/api/v1/.*\" - method: \"POST\" path: \"/api/v1/orders\" Look at the rules.http block. Cilium enforces L7 policy there, so you allow specific HTTP methods and paths and reject the rest, without a service mesh. A compromised frontend can still call GET /api/v1/.... It can’t start calling an admin endpoint you never meant to expose. I can read that policy and tell you what a breached pod reaches, which I couldn’t with iptables.\nDNS-based policies Allow traffic to an external service by name:\napiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: allow-external-api spec: endpointSelector: matchLabels: app: backend egress: - toFQDNs: - matchName: \"api.stripe.com\" - matchPattern: \"*.amazonaws.com\" toPorts: - ports: - port: \"443\" Cilium intercepts the DNS queries and updates the policy when the IPs behind those names change. You write the rule against a name a human understands instead of a CIDR that rotates out from under you.\nCluster-wide policies The same thing across all namespaces:\napiVersion: cilium.io/v2 kind: CiliumClusterwideNetworkPolicy metadata: name: deny-external-by-default spec: endpointSelector: {} egress: - toEntities: - cluster - host - toFQDNs: - matchPattern: \"*.internal.company.com\" egressDeny: - toEntities: - world This denies internet access by default and lets only internal traffic through, unless you open something on purpose. Default-deny egress feels paranoid right up to the day a dependency you forgot about tries to phone home and you catch it in the logs.\nHubble, the observability layer Remember the iptables evening at the top? Hubble is what I wanted that evening. It’s Cilium’s observability layer, and it shows what the network is doing instead of leaving you to infer it from rule order.\n# Enable Hubble hubble: enabled: true relay: enabled: true ui: enabled: true metrics: enabled: - dns - drop - tcp - flow - icmp - http Hubble CLI # Install Hubble CLI export HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt) curl -L --fail --remote-name-all https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-amd64.tar.gz sudo tar xzvfC hubble-linux-amd64.tar.gz /usr/local/bin # Port-forward to Hubble Relay cilium hubble port-forward \u0026 # Observe flows hubble observe --namespace production # Filter by pod hubble observe --pod production/api-xyz123 # Filter by verdict hubble observe --verdict DROPPED # Filter by HTTP hubble observe --protocol http --http-status 500 Hubble UI Here’s the visual flow map:\ncilium hubble ui # Opens browser to http://localhost:12000 The UI shows:\nthe service dependency graph traffic flows, live policy verdicts, allowed or dropped HTTP request and response details Hubble metrics Export to Prometheus:\nhubble: metrics: serviceMonitor: enabled: true enabled: - dns:query;ignoreAAAA - drop - tcp - flow - icmp - http Useful ones:\nhubble_flows_processed_total, total observed flows hubble_drop_total, dropped packets by reason hubble_http_requests_total, HTTP requests by method and status hubble_dns_queries_total, DNS queries by type and response Service mesh without sidecars Here Cilium replaces a whole category of tooling. You get service mesh features without a sidecar proxy next to every pod:\n# Enable L7 proxy l7Proxy: true # Enable mutual TLS authentication: mutual: spiffe: enabled: true trustDomain: cluster.local What that gives you:\nmTLS between services, on SPIFFE identities L7 load balancing with retries Traffic management, canary and header-based routing Observability, L7 metrics and distributed tracing The part I care about is the missing sidecar. The eBPF dataplane handles all of it, which saves the memory and the latency you’d otherwise pay for an Envoy next to every pod. And it’s one fewer container to look at when a request goes missing.\napiVersion: cilium.io/v2 kind: CiliumEnvoyConfig metadata: name: api-routing spec: services: - name: api namespace: production backendServices: - name: api-v1 namespace: production - name: api-v2 namespace: production resources: - \"@type\": type.googleapis.com/envoy.config.listener.v3.Listener # ... Envoy configuration for traffic splitting BGP for bare metal If you self-host on real hardware like I do, there’s no cloud handing you a load balancer. Cilium speaks BGP, so your own switch can route to pods directly:\n# Enable BGP bgpControlPlane: enabled: true apiVersion: cilium.io/v2alpha1 kind: CiliumBGPPeeringPolicy metadata: name: rack-bgp spec: nodeSelector: matchLabels: rack: rack-1 virtualRouters: - localASN: 65001 exportPodCIDR: true neighbors: - peerAddress: \"10.0.0.1/32\" peerASN: 65000 Your pods get routable IPs, advertised over BGP to your network gear.\nBandwidth limits Cilium can cap bandwidth per pod:\napiVersion: v1 kind: Pod metadata: annotations: kubernetes.io/ingress-bandwidth: \"10M\" kubernetes.io/egress-bandwidth: \"10M\" spec: containers: - name: app image: my-app:latest This runs in eBPF instead of as tc rules, so it’s cheaper and the limit holds under load.\nEncryption Encrypt every pod-to-pod flow with one block of config:\n# WireGuard encryption (recommended) encryption: enabled: true type: wireguard # Or IPsec encryption: enabled: true type: ipsec I run WireGuard. It’s faster and simpler, and the kernel module is already there. IPsec only if a compliance certification demands it by name.\nCheck it:\nkubectl -n kube-system exec ds/cilium -- cilium status | grep Encryption # Encryption: Wireguard [NodeEncryption: Disabled, cilium_wg0 (Pubkey: xxx, Port: 51871, Peers: 2)] Multi-cluster with Cluster Mesh Connect several Cilium clusters:\n# Enable Cluster Mesh on each cluster cilium clustermesh enable # Connect clusters cilium clustermesh connect --destination-context cluster2 What that gives you:\nGlobal services, the same service reachable from any cluster Cross-cluster network policies, so you can allow traffic from cluster1 into cluster2 pods Shared identities, so one policy means the same thing in every cluster apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: allow-from-cluster1 spec: endpointSelector: matchLabels: app: api ingress: - fromEndpoints: - matchLabels: io.cilium.k8s.policy.cluster: cluster1 app: frontend Troubleshooting Endpoint status kubectl -n kube-system exec ds/cilium -- cilium endpoint list Policy verdicts kubectl -n kube-system exec ds/cilium -- cilium policy get Dropped traffic hubble observe --verdict DROPPED BPF maps kubectl -n kube-system exec ds/cilium -- cilium bpf lb list kubectl -n kube-system exec ds/cilium -- cilium bpf endpoint list What I run in production Here’s the whole config, the values I run. Next to the bare cilium install at the top, this is everything the post turned on:\ncilium: kubeProxyReplacement: true k8sServiceHost: \"10.0.0.1\" k8sServicePort: \"6443\" ipam: mode: kubernetes # Enable Hubble observability hubble: enabled: true relay: enabled: true ui: enabled: true metrics: enabled: - dns:query - drop - tcp - flow - http # WireGuard encryption encryption: enabled: true type: wireguard # L7 policies l7Proxy: true # Operator HA operator: replicas: 2 # Resource limits resources: limits: cpu: 1000m memory: 1Gi requests: cpu: 100m memory: 256Mi Why each piece is there:\nkube-proxy replacement, because it’s simpler and faster, and it removes the layer I couldn’t read WireGuard encryption, because pod-to-pod traffic should be encrypted and this costs me almost nothing Hubble metrics, exported to Prometheus so I get an alert instead of a surprise L7 proxy, for HTTP-aware policies without a separate mesh Why Cilium To be fair, the traditional CNI plugins work. Calico , Flannel , Weave, they all route packets and plenty of clusters run on them without complaint. My problem is what they’re built on. Iptables is a tool from 1998, and I end up reading it by hand when things go sideways.\nCilium fits how I want to run infrastructure. The dataplane is eBPF in the kernel, and I can dump its maps. Policies are identity-based, so they describe what a workload is rather than which IP it has today. Hubble shows me flows instead of leaving me to infer them from rule order. And service mesh, BGP and encryption come from one tool instead of four.\nAs I wrote in Sovereign Infrastructure, I want to understand what I’m running, and the network path is the worst place for a black box. Cilium catches L7 attacks that L3/L4 rules miss and runs faster while doing it. But the reason I switched is the outage I opened with. The next time a service stops resolving, I run hubble observe --verdict DROPPED and read the answer, instead of restarting a node and hoping.","permalink":"https://tommeurs.nl/en/posts/cilium-ebpf-networking/","title":"Cilium: eBPF Networking for Kubernetes"},{"content":" TL;DR Tempo is where my traces go. The OpenTelemetry Collector receives spans from every service, tail sampling keeps every error and every slow request, and Tempo writes the rest to MinIO with three days of retention. From a span in Grafana I click through to the Loki lines with the same trace ID. My production values are at the bottom.\nYour metrics say something is slow. Your logs say errors happened. Fine. Which request failed, where did the latency come from, and which service in the chain ate the timeout? Neither of them can tell you.\nThe first time this bit me, a checkout flow had started timing out under load. Ten services in the path, every one of them green on its own dashboard, and no way to follow one request from the front door to the place it died. I suspected the payment service because it felt like the payment service.\nDistributed tracing closes that gap. It follows one request through every service it touches and shows you where it stalled. I want to understand what I run instead of guessing at it, and tracing is what turns “I think it’s the payment service” into “here’s the span, here’s the slow query”. So here’s the setup, one Helm chart at a time, ending with the values I run.\nMetrics, logs and traces Complete Observability\nMetrics\n(Prometheus/Thanos)\nWHAT is happening\nLogs\n(Loki)\nWHY it happened\nTraces\n(Tempo)\nWHERE it happened\nGrafana\nMetrics tell you the error rate and the latency. Logs tell you the error message and what was going on around it. Traces tell you which service, which call, and which path the request took.\nAny one of them on its own leaves you half blind. Put all three in the same Grafana and you can follow a request from the alert to the query that caused it.\nA trace is a tree of spans A trace is a tree of spans, one tree per request:\nTrace: order-12345\nAPI Gateway\n250ms\nOrder Service\n180ms\nInventory Check\n45ms\nPayment Service\n120ms\nBank API\n95ms\nNotification\n15ms\nEach box is a span. A span has a name, like HTTP GET /orders, a duration, a parent span, a status of success or error, and whatever attributes you attached to it, user_id and order_id being the usual ones.\nThe trace ID is what holds the tree together. Every span from the same request carries it, however many services that request crossed.\nWhy I run Tempo I run Grafana Tempo because it fits how the rest of my stack already works. It writes to object storage and keeps no index beyond the trace ID, so it stays cheap at high volume and there’s no cluster to babysit. And Grafana treats it as a native data source, which is what makes the click from a span to a log line work later on.\nThat’s the decision to understand before you install anything. Tempo indexes trace IDs and nothing else, the same trick Loki pulls for logs. No index on spans, no index on attributes. Storage stays cheap, and the price is that you can’t ask it for “every trace with user_id=123” out of the box.\nSo how do you find a trace ID? Through the other two. Metrics point you at the bad five minutes, a log line carries the trace ID, and you jump into Tempo for the whole path. Sounds like a chore. Once the data sources are wired together it’s one click in Grafana, and that wiring is further down.\nHow the pieces fit OpenTelemetry\nApplications\nOTLP\nOTLP\nOTLP\ntraces\nmetrics\nlogs\nService A\n(instrumented)\nService B\n(instrumented)\nService C\n(instrumented)\nOTel Collector\nTempo\nPrometheus\nLoki\nObject Storage\nGrafana\nYour applications carry an OpenTelemetry SDK that emits spans. The OTel Collector sits in the middle, receives the telemetry, processes it and fans it out to the right backend. Tempo parks the traces in object storage. Grafana is where you look at them, next to the metrics and logs from the same request.\nTwo Helm charts, then the instrumentation.\nInstalling Tempo The smallest Tempo that receives traces, via Helm :\nhelm repo add grafana https://grafana.github.io/helm-charts helm repo update helm install tempo grafana/tempo \\ --namespace monitoring \\ --values tempo-values.yaml A single binary with traces on a local volume:\n# tempo-values.yaml tempo: storage: trace: backend: local local: path: /var/tempo/traces receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 persistence: enabled: true size: 50Gi That’s enough to start receiving traces, and enough to lose all of them when the pod’s volume dies. Use it for a first look. Before you rely on it during an incident, the next values file swaps the local disk for object storage and splits Tempo into its components.\n# tempo-values.yaml tempo: storage: trace: backend: s3 s3: bucket: tempo-traces endpoint: minio.storage:9000 access_key: ${MINIO_ACCESS_KEY} secret_key: ${MINIO_SECRET_KEY} insecure: true receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 # Retention compactor: compaction: block_retention: 48h # Distributed mode for scale distributor: replicas: 2 ingester: replicas: 3 querier: replicas: 2 compactor: replicas: 1 I point it at my own MinIO rather than a hosted bucket, so the trace data stays on hardware I own. Same reasoning as everything else in the stack.\nInstalling the OpenTelemetry Collector Every span, metric and log line goes through the OTel Collector before it lands anywhere:\nhelm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts helm install otel-collector open-telemetry/opentelemetry-collector \\ --namespace monitoring \\ --values otel-collector-values.yaml The collector’s config:\n# otel-collector-values.yaml mode: deployment replicaCount: 2 config: receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 http: endpoint: 0.0.0.0:4318 processors: batch: timeout: 1s send_batch_size: 1024 # Add Kubernetes metadata k8sattributes: auth_type: serviceAccount extract: metadata: - k8s.namespace.name - k8s.pod.name - k8s.deployment.name # Sample to reduce volume (adjust rate as needed) probabilistic_sampler: sampling_percentage: 10 exporters: otlp/tempo: endpoint: tempo.monitoring:4317 tls: insecure: true prometheus: endpoint: 0.0.0.0:8889 namespace: otel service: pipelines: traces: receivers: [otlp] processors: [k8sattributes, batch] exporters: [otlp/tempo] metrics: receivers: [otlp] processors: [batch] exporters: [prometheus] Instrumenting your applications Tempo and the collector don’t do anything until your apps emit spans. There are two ways in, and you’ll probably use both.\nAuto-instrumentation For a lot of languages OpenTelemetry hooks into the framework for you and you write zero tracing code. This is where I start with any service I didn’t write myself.\nJava:\napiVersion: apps/v1 kind: Deployment spec: template: spec: containers: - name: app image: my-java-app:latest env: - name: JAVA_TOOL_OPTIONS value: \"-javaagent:/otel/opentelemetry-javaagent.jar\" - name: OTEL_SERVICE_NAME value: \"order-service\" - name: OTEL_EXPORTER_OTLP_ENDPOINT value: \"http://otel-collector.monitoring:4317\" volumeMounts: - name: otel-agent mountPath: /otel initContainers: - name: otel-agent image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest command: [cp, /javaagent.jar, /otel/opentelemetry-javaagent.jar] volumeMounts: - name: otel-agent mountPath: /otel volumes: - name: otel-agent emptyDir: {} Python:\nFROM python:3.11 RUN pip install opentelemetry-distro opentelemetry-exporter-otlp RUN opentelemetry-bootstrap -a install CMD [\"opentelemetry-instrument\", \"python\", \"app.py\"] Node.js:\n// tracing.js - require this first const { NodeSDK } = require('@opentelemetry/sdk-node'); const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc'); const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node'); const sdk = new NodeSDK({ traceExporter: new OTLPTraceExporter({ url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || 'http://otel-collector:4317', }), instrumentations: [getNodeAutoInstrumentations()], serviceName: process.env.OTEL_SERVICE_NAME || 'my-service', }); sdk.start(); Auto-instrumentation gets you HTTP, gRPC and database calls for free. It can’t see your business logic. When you want a span around “process this order” with the order ID attached, you write that span yourself.\nManual instrumentation For your own spans and the attributes you’ll search on:\n// Go example import ( \"go.opentelemetry.io/otel\" \"go.opentelemetry.io/otel/attribute\" ) func ProcessOrder(ctx context.Context, orderID string) error { tracer := otel.Tracer(\"order-service\") ctx, span := tracer.Start(ctx, \"process-order\") defer span.End() // Add attributes span.SetAttributes( attribute.String(\"order.id\", orderID), attribute.String(\"order.type\", \"standard\"), ) // Create child span for sub-operation ctx, childSpan := tracer.Start(ctx, \"validate-inventory\") err := validateInventory(ctx, orderID) childSpan.End() if err != nil { span.RecordError(err) return err } return nil } Context propagation Here’s the part that breaks most first attempts. A trace stays one trace only if the trace ID travels with the request to the next service. Drop it on one hop and you get two traces that don’t know about each other.\nHTTP headers (automatic with instrumentation):\ntraceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01 tracestate: vendor=value gRPC metadata (also automatic with instrumentation)\nAuto-instrumentation handles this for you. If you build HTTP calls by hand, you inject and extract the context yourself:\n// Inject context into outgoing request req, _ := http.NewRequestWithContext(ctx, \"GET\", url, nil) otel.GetTextMapPropagator().Inject(ctx, propagation.HeaderCarrier(req.Header)) // Extract context from incoming request ctx := otel.GetTextMapPropagator().Extract(r.Context(), propagation.HeaderCarrier(r.Header)) Grafana integration This is the part that makes tracing worth having. With the right data source config, Grafana jumps from a trace to the matching logs and metrics without you copying a trace ID anywhere.\nAdd Tempo as a data source:\napiVersion: v1 kind: ConfigMap metadata: name: grafana-datasources data: tempo.yaml: | apiVersion: 1 datasources: - name: Tempo type: tempo url: http://tempo.monitoring:3100 access: proxy jsonData: tracesToLogs: datasourceUid: loki tags: ['app', 'namespace'] tracesToMetrics: datasourceUid: prometheus tags: ['service.name'] serviceMap: datasourceUid: prometheus nodeGraph: enabled: true search: hide: false lokiSearch: datasourceUid: loki Finding a trace In Grafana Explore:\nSelect Tempo data source Choose “Search” tab Filter by service name, duration, status Click a trace to see the waterfall From a trace to its logs With tracesToLogs configured, a span links to the log lines it produced:\nOpen a trace Click a span Click “Logs for this span” See Loki logs with the same trace ID That click is what the data source wiring buys you. No grep across pods, no guessing which one logged the error.\nFrom a trace to its metrics The reverse works too. From a slow trace you pivot to the latency histogram and error rate of that service, and you can tell whether you’re looking at one unlucky request or a trend.\nService graph Tempo can build a service dependency graph from the traces themselves:\n# Enable metrics generator in Tempo tempo: metricsGenerator: enabled: true remoteWriteUrl: http://prometheus.monitoring:9090/api/v1/write That produces metrics like:\ntraces_service_graph_request_total traces_service_graph_request_failed_total traces_service_graph_request_server_seconds Grafana renders those as a map of traffic and error rates between services, built from real requests rather than from an architecture diagram nobody updated.\nSampling Once real traffic hits this, the next problem is volume. Storing every trace at production scale gets expensive fast, so you sample. There are two ways to do it, and the difference matters.\nHead sampling # OTel Collector processors: probabilistic_sampler: sampling_percentage: 10 # Keep 10% of traces Dead simple, with one flaw. The decision is made before the request finishes, so a coin flip can throw away the exact trace that errored. The traces you’d want during an incident are the ones this drops.\nTail sampling processors: tail_sampling: decision_wait: 10s policies: # Always keep errors - name: errors type: status_code status_code: status_codes: [ERROR] # Always keep slow traces - name: slow type: latency latency: threshold_ms: 1000 # Sample 5% of everything else - name: probabilistic type: probabilistic probabilistic: sampling_percentage: 5 Tail sampling waits until the request is done and decides then. Every error kept, everything slow kept, a 5% net over the boring successes. It’s what I run, because the trace I go looking for during an incident is exactly the kind head sampling drops.\nWhat I run in production Here’s roughly what I run. Object storage on MinIO, tail sampling that keeps errors and slow requests, and the metrics generator on so the service graph comes with it:\n# Tempo with object storage tempo: storage: trace: backend: s3 s3: bucket: tempo-traces endpoint: minio.storage:9000 compactor: compaction: block_retention: 72h # 3 days of traces metricsGenerator: enabled: true remoteWriteUrl: http://prometheus:9090/api/v1/write # OTel Collector with tail sampling otel-collector: config: processors: tail_sampling: policies: - name: errors type: status_code status_codes: [ERROR] - name: slow type: latency threshold_ms: 500 - name: sample-rest type: probabilistic sampling_percentage: 5 The 5% keeps the volume manageable, and the metrics generator gives me the dependency map without extra work. Retention is 72 hours, and that sounds short until you notice that nearly every trace you go looking for is from the last hour. Anything that lasts longer than three days shows up in the metrics, and metrics are cheap to keep around for ages.\nDebugging with a trace Back to the checkout that was timing out. With tracing in place, the same problem reads like this:\nAlert fires: High latency on checkout service Check metrics: P99 latency spiked at 14:32 Find traces: Search Tempo for checkout-service, duration \u003e 1s, time range 14:30-14:35 Analyze trace: See that payment-service call took 4.2s Drill into span: See db.statement attribute showing slow query Check logs: Jump to Loki logs for that span, see connection pool exhaustion Fix: Increase connection pool size Steps 3 to 6 used to be guesswork. Now they’re clicks in Grafana.\nWhy this matters Microservices split work across teams nicely and make debugging miserable. One request touches ten services, and when it breaks, the logs show you the error text and the metrics show you the symptom. The path the request took, the thing that tells you which service to open, lives in the traces.\nPut Prometheus and Thanos for metrics, Loki for logs and Tempo for traces in one Grafana, correlated, on hardware you own. That last part is what I care about most. No vendor sees my request paths, no SaaS bill grows with my trace volume, and when a request fails I can open the box and read what happened to it.\nThe next time something times out, I search Tempo for the service, sort by duration, open the longest trace and read which span ate the time. Then I click through to its logs.","permalink":"https://tommeurs.nl/en/posts/tempo-distributed-tracing/","title":"Distributed Tracing with Tempo and OpenTelemetry"},{"content":" TL;DR Loki indexes labels and stores the log text as compressed chunks, so a month of logs fits on commodity hardware. The trade is full-text search. Below is the single-binary install I’d hand anyone first, the same chart on MinIO with three replicas, and the knobs I didn’t touch until the volume was real.\nYou’ve got Prometheus for metrics, so you can already see what’s happening across your clusters. Metrics tell you request latency spiked at 14:32. They don’t tell you the payment service threw a null pointer because someone shipped a config change with a typo. For that you need the log line.\nThe default answer for years was Elasticsearch. It’s powerful and flexible, and it indexes every token in every log line. That full-text index is great until you look at the bill. You pay for it in CPU at ingest, in RAM to keep the index hot, and in storage that grows faster than the logs themselves. I ran an ELK stack in a previous job and spent more time tuning JVM heap sizes than reading logs.\nLoki indexes a small set of labels and stores the log text itself as compressed chunks. Same trick that makes Prometheus cheap for metrics, now pointed at logs. You give up arbitrary full-text search and get back a system you can afford to run on commodity hardware.\nWhy Loki, and what it costs you Grafana Labs made four choices that explain the rest of this post. Only the labels get indexed, and the log lines are compressed and written to object storage. The labels come straight from Kubernetes metadata, so there’s no mapping config to maintain. The dashboards, alerting and Explore view are the same Grafana you already use for metrics. And there’s no JVM to babysit and no shard rebalancing to wake up for.\nThe cost is that you can’t grep across all your logs without first narrowing by label. You have to know which namespace, app or pod you care about before you start filtering content. Do I miss that? Only when I don’t know which app logged the thing, and that’s rarer than I expected. Kubernetes debugging starts from “the checkout pods are unhappy” and drills in, and that’s a label selector. If your job is open-ended log analytics where you don’t know what you’re hunting for, that constraint will bite, and I come back to it at the end.\nArchitecture Kubernetes Cluster\nNodes\nPromtail\n(DaemonSet)\nPromtail\nPromtail\nPod Logs\n(/var/log/pods)\nLoki\nObject Storage\n(chunks)\nGrafana\nThree moving parts.\nPromtail runs on each node as a DaemonSet, tails the log files under /var/log/pods, attaches labels, and ships the lines to Loki.\nLoki takes those lines, indexes them by label, compresses the bodies into chunks, and writes the chunks to object storage.\nGrafana queries Loki with LogQL and shows the results in Explore or on a dashboard.\nIf you’ve run Prometheus you’ve seen this shape before, collectors feeding a server feeding Grafana, and Grafana Labs copied it on purpose.\nOne process on one disk Start with the Grafana Helm charts.\nhelm repo add grafana https://grafana.github.io/helm-charts helm repo update # Install Loki with simple scalable deployment helm install loki grafana/loki \\ --namespace monitoring \\ --create-namespace \\ --values loki-values.yaml For a homelab or a small cluster, run Loki as a single binary backed by the local filesystem. No object storage, no replication, one process writing chunks to a PVC. This is the version I’d start anyone on, because you see logs in Grafana in about five minutes and can decide whether you like the label model before you wire up MinIO .\n# loki-values.yaml loki: auth_enabled: false commonConfig: replication_factor: 1 storage: type: filesystem schemaConfig: configs: - from: 2024-01-01 store: tsdb object_store: filesystem schema: v13 index: prefix: index_ period: 24h singleBinary: replicas: 1 persistence: size: 50Gi # Disable components not needed for single binary backend: replicas: 0 read: replicas: 0 write: replicas: 0 The single binary is enough to learn LogQL on. What it doesn’t survive is the node dying, because the chunks live on that node’s PVC. Once that matters, move the chunks into object storage. Here’s the same chart pointed at S3-compatible storage with a replication factor of three:\n# loki-values.yaml loki: auth_enabled: false commonConfig: replication_factor: 3 storage: type: s3 s3: endpoint: minio.storage:9000 bucketnames: loki-chunks access_key_id: ${MINIO_ACCESS_KEY} secret_access_key: ${MINIO_SECRET_KEY} insecure: true schemaConfig: configs: - from: 2024-01-01 store: tsdb object_store: s3 schema: v13 index: prefix: index_ period: 24h # Scalable deployment backend: replicas: 3 read: replicas: 3 write: replicas: 3 Getting logs in with Promtail Loki doesn’t read anything by itself. Promtail runs on every node and forwards whatever lands in the pod log directory.\nhelm install promtail grafana/promtail \\ --namespace monitoring \\ --set config.clients[0].url=http://loki:3100/loki/api/v1/push A more realistic Promtail config does some work on the way in. It parses the CRI log format, drops the noisy filename label, and pulls a few fields out of nginx access logs into labels you can query on.\n# promtail-values.yaml config: clients: - url: http://loki:3100/loki/api/v1/push snippets: # Add Kubernetes metadata as labels pipelineStages: - cri: {} - labeldrop: - filename - match: selector: '{app=\"nginx\"}' stages: - regex: expression: '^(?P\u003cremote_addr\u003e[\\d\\.]+) - (?P\u003cremote_user\u003e\\S+) \\[(?P\u003ctime_local\u003e[^\\]]+)\\] \"(?P\u003crequest\u003e[^\"]+)\" (?P\u003cstatus\u003e\\d+)' - labels: status: # DaemonSet tolerations for all nodes tolerations: - operator: Exists Labels are the index If you understand one thing about Loki, make it this. Labels are the entire index. They decide how logs get stored and how fast a query returns. A bad label set is how a system that’s supposed to be light on RAM ends up falling over.\nPromtail hands you these Kubernetes labels for free:\nLabel Source Example namespace Pod namespace default, monitoring pod Pod name nginx-abc123 container Container name nginx, sidecar node_name Node worker-1 app Pod label nginx job Scrape config kubernetes-pods Now the mistake I made too. Every unique combination of label values creates a separate stream, and Loki keeps an index entry per stream. Add a label with thousands of distinct values and you’ve created thousands of tiny streams. This is high cardinality, and it’s the fastest way to make Loki miserable. A request_id label generates a new stream for every request. That’s millions of streams a day, an index that no longer fits in memory, and queries that crawl.\nSo a label should be a dimension you’d group or filter by, with a handful of possible values. Anything unbounded stays in the log body, where you filter it with LogQL at query time instead of indexing it.\nGood labels (bounded, you filter on them):\nnamespace, app, environment, team Bad labels (unbounded, they belong in the log line):\nrequest_id, user_id, trace_id, timestamp Querying with LogQL LogQL is how you read your logs back. If you know PromQL, you already know most of it. Pick a stream with label selectors, then pipe it through filters and parsers.\nBasic queries # All logs from a namespace {namespace=\"production\"} # Specific app {app=\"frontend\", namespace=\"production\"} # Multiple containers {container=~\"nginx|envoy\"} # Exclude a namespace {namespace!=\"kube-system\"} Filtering content # Lines containing \"error\" {app=\"frontend\"} |= \"error\" # Lines NOT containing \"health\" {app=\"frontend\"} != \"health\" # Regex match {app=\"frontend\"} |~ \"status=(4|5)[0-9]{2}\" # Case insensitive {app=\"frontend\"} |~ \"(?i)error\" Parsing and extracting # Parse JSON logs {app=\"api\"} | json # Extract specific field {app=\"api\"} | json | status_code \u003e= 500 # Parse with pattern {app=\"nginx\"} | pattern `\u003cip\u003e - - [\u003c_\u003e] \"\u003cmethod\u003e \u003cpath\u003e \u003c_\u003e\" \u003cstatus\u003e` # Use extracted fields {app=\"nginx\"} | pattern `\u003c_\u003e - - [\u003c_\u003e] \"\u003cmethod\u003e \u003cpath\u003e \u003c_\u003e\" \u003cstatus\u003e` | status \u003e= 400 This is where keeping request_id out of the labels stops costing you anything. The {...} selector hits the index and narrows you to a few streams. Everything after the pipe runs over just those lines, not your whole log volume, and | json pulls the request id back out at query time.\nAggregations, or metrics from logs # Count errors per app sum by (app) (count_over_time({namespace=\"production\"} |= \"error\" [5m])) # Rate of requests sum(rate({app=\"nginx\"} | pattern `\u003c_\u003e \"\u003cmethod\u003e \u003cpath\u003e \u003c_\u003e\" \u003cstatus\u003e` [1m])) by (status) # Bytes per namespace sum by (namespace) (bytes_over_time({job=\"kubernetes-pods\"}[1h])) Wiring it into Grafana The reason I picked Loki over running a separate log stack is that the logs land in the same Grafana I already use for metrics. One data source to add, and the rest of the workflow is identical.\napiVersion: v1 kind: ConfigMap metadata: name: grafana-datasources namespace: monitoring data: loki.yaml: | apiVersion: 1 datasources: - name: Loki type: loki url: http://loki:3100 access: proxy jsonData: maxLines: 1000 Explore Explore is where I spend most of my Loki time when something’s on fire. Pick the Loki data source, build the selector in the label browser, add a content filter, and click a line to see the lines around it.\nDashboard panels Add logs to your dashboards:\n{ \"type\": \"logs\", \"datasource\": \"Loki\", \"targets\": [ { \"expr\": \"{namespace=\\\"production\\\", app=\\\"frontend\\\"} |= \\\"error\\\"\", \"refId\": \"A\" } ], \"options\": { \"showTime\": true, \"showLabels\": false, \"wrapLogMessage\": true } } Metrics and logs in one dashboard # Prometheus panel showing error rate sum(rate(http_requests_total{status=~\"5..\"}[5m])) by (app) # Loki panel showing error logs {app=\"$app\"} |= \"error\" The $app template variable feeds both panels. You spot a 5xx spike in the metric, glance down, and the matching error lines are already there. No tab-switching, no copy-pasting timestamps into a different tool. That view is what I wanted from a log system when I stopped tuning heap sizes.\nAlerting on a log line Once you’re querying logs, you can alert on them too. Sometimes the signal you care about only exists in a log line, not in a metric, and you don’t want to wait for someone to notice. Loki gives you two ways to do this, Grafana alert rules or Loki’s own ruler component.\n# Grafana alert rule apiVersion: 1 groups: - name: LogAlerts rules: - alert: HighErrorRate expr: | sum(count_over_time({namespace=\"production\"} |= \"error\" [5m])) \u003e 100 for: 5m labels: severity: warning annotations: summary: \"High error rate in production logs\" The ruler runs the same LogQL inside Loki itself, so the alert fires even if Grafana is down. Handy for the things you can’t miss, like a payment service logging CRITICAL:\n# loki-rules.yaml groups: - name: errors rules: - alert: CriticalError expr: | count_over_time({app=\"payment-service\"} |= \"CRITICAL\" [1m]) \u003e 0 for: 0m labels: severity: critical annotations: summary: \"Critical error in payment service\" Retention and storage Every chunk you keep is disk you pay for. Loki’s compactor handles retention and deletes chunks past your cutoff. Thirty days covers almost every “what happened last week” question without the bucket growing without bound.\nloki: limits_config: retention_period: 30d compactor: working_directory: /var/loki/compactor retention_enabled: true retention_delete_delay: 2h retention_delete_worker_count: 150 If you run multi-tenancy, you can set retention per tenant. Keep production around longer, let dev logs expire fast:\nloki: limits_config: retention_period: 30d # Default overrides: production: retention_period: 90d # Keep production logs longer development: retention_period: 7d # Dev logs expire faster Performance tuning You can run Loki for a long time without touching any of this. These are the knobs you reach for once you’re at real volume and something feels slow.\nChunk size Bigger chunks mean fewer index entries and better compression, at the cost of higher latency on small queries. It’s a throughput-versus-latency dial:\nloki: ingester: chunk_target_size: 1572864 # 1.5MB chunk_idle_period: 30m max_chunk_age: 2h Query limits Someone will eventually run an unbounded query over six months of logs and wonder why the cluster got slow. Limits stop that before it starts:\nloki: limits_config: max_query_length: 721h # Max time range max_query_parallelism: 32 # Concurrent sub-queries max_entries_limit_per_query: 5000 Caching Repeated queries over the same time range get a lot faster with a memcached in front of the chunk and results paths:\nloki: memcached: chunk_cache: enabled: true host: memcached.monitoring results_cache: enabled: true host: memcached.monitoring Structured logging pays off here Loki rewards you for logging in JSON. A structured line means | json pulls out every field as something you can filter on, no regex archaeology required.\n{ \"level\": \"error\", \"message\": \"Failed to process order\", \"order_id\": \"12345\", \"error\": \"payment declined\", \"duration_ms\": 234 } Now a query like this works, pulling level and duration_ms out of the line at query time:\n{app=\"order-service\"} | json | level=\"error\" | duration_ms \u003e 1000 Most runtimes have a one-liner for JSON output:\n# Spring Boot logging.pattern.console: '{\"timestamp\":\"%d\",\"level\":\"%p\",\"logger\":\"%c\",\"message\":\"%m\"}%n' # Node.js with winston const logger = winston.createLogger({ format: winston.format.json(), }); # Go with zap logger, _ := zap.NewProduction() The values I run Here’s everything from above in one config. Object storage on MinIO, three replicas, thirty-day retention, ingest rate limits, and Promtail on every node including the control plane.\n# Loki with object storage loki: auth_enabled: false commonConfig: replication_factor: 3 storage: type: s3 s3: endpoint: minio.storage:9000 bucketnames: loki-data limits_config: retention_period: 30d ingestion_rate_mb: 10 ingestion_burst_size_mb: 20 # Three-replica deployment backend: replicas: 3 persistence: size: 50Gi read: replicas: 3 write: replicas: 3 persistence: size: 50Gi # Promtail on all nodes promtail: tolerations: - operator: Exists config: clients: - url: http://loki:3100/loki/api/v1/push The bucket is on MinIO rather than a cloud provider for the reason in Sovereign Infrastructure, which is that my logs shouldn’t depend on a provider I can’t inspect or walk away from. Retention is 30 days because that answers the questions I ask without paying to hoard a year of noise. Three replicas so a node falling over doesn’t take my logs with it. And Promtail on the control plane too, because the logs you need most during an incident are often from the components you forgot to collect.\nLoki vs Elasticsearch Aspect Loki Elasticsearch Indexing Labels only Full-text Storage cost Lower Higher Query flexibility Label-first Full-text search Operations Simpler Complex Memory usage Lower Higher (JVM) Grafana integration Native Good Elasticsearch does one thing Loki can’t, which is search you didn’t plan for. The two systems optimise for different questions, so the question to ask is which one you ask more often.\nLoki fits when you query by known dimensions like namespace, app and pod, when cost matters, when you want fewer moving parts to operate, and when you’re already in Grafana. Elasticsearch fits when you need full-text search across all logs, when you don’t know what you’re looking for, and when log analytics is a primary use case rather than a debugging aid.\nWhy I run this one I can afford to keep a month of logs on hardware I own, the labels match how I already think about my cluster, and it lives in the same Grafana I open every day. When something breaks, the answer is one LogQL query in the panel under the metric that alerted.\nPair it with Prometheus and Thanos for metrics and traces on top, and you’ve got all three signals on infrastructure you can read the config of. The spike at 14:32 from the top of this post gets a |= \"error\" panel under its graph, and the typo’d config key is in one of those lines.","permalink":"https://tommeurs.nl/en/posts/loki-kubernetes-logging/","title":"Loki for Kubernetes Logging: The Prometheus-Like Approach"},{"content":" TL;DR Thanos Receive turns the metrics flow around. Each edge Prometheus pushes samples over remote_write to one central endpoint, buffers in its WAL when the link drops, and never sees an object storage credential. Below is the Receive StatefulSet, the hashring, the edge config, two ways to lock the endpoint down, and the hybrid I run with sidecars on the core clusters.\nIn my previous post on Prometheus and Thanos, I set up the sidecar architecture. Thanos Sidecar runs next to Prometheus, uploads TSDB blocks to object storage, and exposes the data to the Querier over gRPC. For clusters in the same datacenter with a fat, stable link to the central infrastructure, I’ve got no complaints. Everything pulls, and everything can reach everything else.\nThen I started putting Prometheus on clusters at the edge.\nThose clusters drop off the network for hours when a site loses its uplink. Some sit behind firewalls that allow outbound HTTPS and nothing else. And I went from a handful of clusters to a few dozen, with more on the way. The sidecar model was fighting me at every one of them.\nThe fix is to stop pulling and start pushing. Thanos has a component for that, called Receive, and this post is about running it.\nWhere the sidecar model runs out of road The sidecar setup makes four assumptions. Every one of them held in the datacenter.\nThanos Sidecar runs on every Prometheus instance Every cluster gets direct object storage access to upload its own blocks The network stays up long enough for those block uploads to finish gRPC reaches back from the central Querier to each cluster for live queries They stop holding at the edges of the topology. An edge site where the internet link comes and goes. An air-gapped site that can only talk outbound, never inbound. Hundreds of small clusters where one sidecar each adds up to real overhead. A multi-tenant platform where you don’t even own all the clusters pushing data at you.\nAssumption 2 bit me first. Handing object storage credentials to every edge site, including sites I don’t fully control, is a blast radius I don’t want. Assumption 3 bit me second. A factory site that loses its uplink for an afternoon stops uploading blocks, and there’s no story for catching up afterwards. The blocks sit there until someone notices.\nSo I wanted metrics from clusters I can’t reliably reach, and I wanted my storage keys on none of them. Does Thanos have a component for that? It does.\nThanos Receive flips the direction Receive turns the arrows around. Instead of sidecars pushing blocks out to object storage and the Querier reaching in, each Prometheus pushes its samples to one central Receive, and Receive owns all the storage plumbing.\nCentral Cluster\nEdge Clusters\nSite C (Retail)\nSite B (Warehouse)\nSite A (Factory)\nremote_write\nremote_write\nremote_write\nPrometheus\nPrometheus\nPrometheus\nThanos Receive\n(HA cluster)\nThanos Querier\nStore Gateway\nCompactor\nObject Storage\nGrafana\nThe part that makes this cheap to adopt is that Prometheus already has remote_write built in. You don’t install anything new at the edge. It’s a standard Prometheus feature that streams samples to any compatible endpoint, and Receive implements that endpoint. The edge side of this is a few lines of config.\nWhat push buys you at the edge 1. The edge deployment shrinks to almost nothing No sidecar, no storage credentials at the edge, no gRPC ports to expose inbound. The edge cluster runs Prometheus with a remote_write URL.\n# The ENTIRE edge configuration global: external_labels: cluster: factory-site-a region: europe-west remote_write: - url: https://thanos-receive.central.example.com/api/v1/receive bearer_token_file: /etc/prometheus/token That’s the whole edge config. The edge cluster has no idea object storage exists, never mind which bucket or which provider. It scrapes locally and ships samples upstream. This is the sovereignty angle too. A site I don’t fully control gets a write-only token to one endpoint instead of a key to my storage.\n2. It survives losing the network Drop the link and Prometheus keeps scraping, buffering the unsent samples locally. Bring the link back and it drains the backlog. The Write-Ahead Log (WAL) does this without you wiring up anything.\nremote_write: - url: https://thanos-receive.central.example.com/api/v1/receive queue_config: capacity: 100000 # Buffer size max_shards: 50 # Parallel send workers min_shards: 1 max_samples_per_send: 5000 batch_send_deadline: 30s min_backoff: 1s max_backoff: 5m # Exponential backoff on failure I’ve watched a factory cluster sit dark for four hours and then quietly backfill the whole gap once its uplink came back. Nobody touched anything. The WAL had held the samples the entire time.\n3. Outbound-only works with the firewall Edge sites tend to have paranoid firewalls, and honestly they should. Inbound connections get blocked. VPNs are a maintenance tax I’d rather not pay across dozens of sites. Outbound HTTPS to one known hostname, though, is almost always fine. Push fits these networks as they are, so I don’t have to argue with anyone about opening a port.\n4. Less running at the edge Dropping the sidecar means less memory on the edge nodes, one fewer container to patch and watch, no gRPC connections to keep alive, and fewer ways for things to break. On a refurbished mini PC at a remote site, every bit of that counts.\nSetting it up The Receive deployment on the central cluster Receive is stateful. It writes incoming samples to local disk first, then uploads blocks to object storage on its own schedule, so it needs persistent volumes. Run it as a StatefulSet:\napiVersion: apps/v1 kind: StatefulSet metadata: name: thanos-receive namespace: monitoring spec: serviceName: thanos-receive replicas: 3 selector: matchLabels: app: thanos-receive template: metadata: labels: app: thanos-receive spec: containers: - name: thanos-receive image: quay.io/thanos/thanos:v0.34.0 args: - receive - --http-address=0.0.0.0:10902 - --grpc-address=0.0.0.0:10901 - --remote-write.address=0.0.0.0:19291 - --tsdb.path=/var/thanos/receive - --objstore.config-file=/etc/thanos/objstore.yaml - --label=receive_replica=\"$(POD_NAME)\" - --tsdb.retention=6h # Hashring for HA distribution - --receive.hashrings-file=/etc/thanos/hashring.json - --receive.local-endpoint=$(POD_NAME).thanos-receive.monitoring.svc:10901 ports: - name: http containerPort: 10902 - name: grpc containerPort: 10901 - name: remote-write containerPort: 19291 volumeMounts: - name: data mountPath: /var/thanos/receive - name: objstore-config mountPath: /etc/thanos env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name volumes: - name: objstore-config secret: secretName: thanos-objstore-secret volumeClaimTemplates: - metadata: name: data spec: storageClassName: longhorn accessModes: [\"ReadWriteOnce\"] resources: requests: storage: 50Gi The hashring keeps replicas honest Run more than one Receive replica and you need a hashring to decide which replica owns which incoming series:\n[ { \"endpoints\": [ \"thanos-receive-0.thanos-receive.monitoring.svc:10901\", \"thanos-receive-1.thanos-receive.monitoring.svc:10901\", \"thanos-receive-2.thanos-receive.monitoring.svc:10901\" ] } ] Store this in a ConfigMap:\napiVersion: v1 kind: ConfigMap metadata: name: thanos-receive-hashring namespace: monitoring data: hashring.json: | [ { \"endpoints\": [ \"thanos-receive-0.thanos-receive.monitoring.svc:10901\", \"thanos-receive-1.thanos-receive.monitoring.svc:10901\", \"thanos-receive-2.thanos-receive.monitoring.svc:10901\" ] } ] The hashring pins every time series to a consistent Receive instance, so samples for one series always land on the same replica. Without it, several replicas taking writes at once would give you duplicated or split data.\nExposing Receive to the outside Edge clusters need to reach Receive from off-cluster, so put it behind a load balancer and an Ingress:\napiVersion: v1 kind: Service metadata: name: thanos-receive-lb namespace: monitoring spec: type: LoadBalancer ports: - name: remote-write port: 443 targetPort: 19291 selector: app: thanos-receive --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: thanos-receive namespace: monitoring annotations: cert-manager.io/cluster-issuer: letsencrypt nginx.ingress.kubernetes.io/backend-protocol: \"HTTP\" spec: tls: - hosts: - thanos-receive.example.com secretName: thanos-receive-tls rules: - host: thanos-receive.example.com http: paths: - path: /api/v1/receive pathType: Prefix backend: service: name: thanos-receive port: number: 19291 Pointing the edge Prometheus at Receive On each edge cluster, this is all the Prometheus operator config you need to start pushing:\napiVersion: monitoring.coreos.com/v1 kind: Prometheus metadata: name: prometheus namespace: monitoring spec: replicas: 1 retention: 24h # Local retention for edge queries externalLabels: cluster: factory-site-a region: europe-west environment: production remoteWrite: - url: https://thanos-receive.example.com/api/v1/receive bearerTokenSecret: name: thanos-remote-write-token key: token queueConfig: capacity: 50000 maxShards: 20 minShards: 1 maxSamplesPerSend: 2000 batchSendDeadline: 30s minBackoff: 1s maxBackoff: 5m writeRelabelConfigs: # Drop high-cardinality metrics you don't need centrally - sourceLabels: [__name__] regex: 'go_.*' action: drop One thing I learned the annoying way. Get your externalLabels right from the start, because they’re how you tell which site a metric came from once everything is pooled centrally. Forget them and you’ll be staring at a Grafana panel unable to say whether a spike is the factory or the warehouse. I have done that staring.\nLocking it down A Receive endpoint open to the internet with no auth is a free metrics-injection service for anyone who finds it. Don’t ship that. I’ve used two options.\nBearer token, when simple is enough Generate a token, hand it to the edge clusters, and have Receive check it:\n# Generate token openssl rand -hex 32 \u003e receive-token.txt # Create secret in central cluster kubectl create secret generic thanos-receive-auth \\ --from-file=token=receive-token.txt \\ -n monitoring # Create secret in edge cluster kubectl create secret generic thanos-remote-write-token \\ --from-file=token=receive-token.txt \\ -n monitoring Configure Receive to validate tokens:\n# Add to Receive container args - --receive.tenant-header=THANOS-TENANT - --receive.default-tenant-id=anonymous mTLS, when a shared token isn’t good enough For sites where a single shared secret feels too fragile, mutual TLS gives each cluster its own identity:\n# Receive args - --grpc-server-tls-cert=/etc/tls/server.crt - --grpc-server-tls-key=/etc/tls/server.key - --grpc-server-tls-client-ca=/etc/tls/ca.crt # Edge Prometheus remoteWrite remoteWrite: - url: https://thanos-receive.example.com/api/v1/receive tlsConfig: certFile: /etc/prometheus/tls/client.crt keyFile: /etc/prometheus/tls/client.key caFile: /etc/prometheus/tls/ca.crt Keeping tenants apart If different teams or customers push into the same Receive, a tenant header keeps their data separated:\n# Edge cluster A remoteWrite: - url: https://thanos-receive.example.com/api/v1/receive headers: THANOS-TENANT: tenant-a # Edge cluster B remoteWrite: - url: https://thanos-receive.example.com/api/v1/receive headers: THANOS-TENANT: tenant-b Receive stores the data with the tenant label, so a query can select one tenant:\n# Query only tenant-a data http_requests_total{tenant_id=\"tenant-a\"} Every sample carries its tenant label, so one Receive cluster can serve several tenants without their data bleeding into each other’s queries.\nSurviving the flaky link The whole reason I went down this road was the edge connectivity problem, so this is the part I care about most. How graceful is the degradation? Graceful enough that I stopped worrying about the edge links.\nTuning the WAL buffer Prometheus parks unsent samples in its Write-Ahead Log and replays them when the endpoint comes back. The buffer size is yours to tune for how bad your worst outage gets:\nremoteWrite: - url: https://thanos-receive.example.com/api/v1/receive queueConfig: capacity: 100000 # Samples to buffer maxShards: 30 # Parallel senders when catching up maxSamplesPerSend: 5000 batchSendDeadline: 60s minBackoff: 1s maxBackoff: 10m # Don't hammer when down For a typical edge cluster scraping every 30s, a 100k sample buffer covers roughly 3-4 hours of metrics. Your mileage depends on scrape interval and how many targets each site has, so size it against your own worst-case outage rather than mine.\nWatch the remote write from the edge On the edge cluster, alert when remote write starts failing so you know the buffer is filling:\n- alert: PrometheusRemoteWriteFailing expr: | rate(prometheus_remote_storage_failed_samples_total[5m]) \u003e 0 for: 15m labels: severity: warning annotations: summary: \"Prometheus remote write failing\" description: \"Edge cluster {{ $labels.cluster }} failing to send metrics\" Watch for silent edges from the centre The edge-side alert only fires if the edge can still reach you. The case that scares me is a site that goes completely dark, so catch that one centrally:\n- alert: EdgeClusterMissing expr: | absent_over_time(up{job=\"prometheus\", cluster=~\"edge-.*\"}[1h]) for: 30m labels: severity: critical annotations: summary: \"Edge cluster not reporting\" description: \"No metrics from {{ $labels.cluster }} for 1 hour\" Sidecar or Receive, per cluster Receive and the sidecar solve different networks, and the table shows where they differ:\nAspect Sidecar Receive Edge/offline support Poor Excellent Real-time queries Yes (via gRPC) Yes (via Receive gRPC) Resource at edge Higher Lower Object storage access Required at edge Central only Network direction Outbound + inbound Outbound only Complexity at edge Higher Lower Block upload latency 2h (block size) Real-time Sidecar fits when all your clusters have stable connectivity, you want minimal central infrastructure, and real-time queries across every cluster matter. Receive fits when edge clusters have intermittent connectivity, you’re scaling to many (50+) clusters, edge resources are constrained, or the firewall rules are outbound-only.\nYou don’t have to choose globally. The healthiest setup I’ve run is a hybrid, with sidecars on the core clusters that have stable links, Receive for everything at the edge, and both feeding the same Querier and the same object storage. The Querier doesn’t care how the data arrived.\nHow I run this That hybrid is what’s in production for me:\n# Central infrastructure cluster: Sidecar # (stable connectivity, needs real-time queries) prometheus: thanos: enabled: true objectStorageConfig: name: thanos-objstore # Edge manufacturing sites: Remote Write # (intermittent connectivity, resource constrained) prometheus: remoteWrite: - url: https://thanos-receive.central:443/api/v1/receive bearerTokenSecret: name: edge-token queueConfig: capacity: 150000 # 6+ hours buffer maxBackoff: 15m # All feed the same Thanos Query thanos: query: stores: - dnssrv+_grpc._tcp.prometheus-operated.monitoring.svc # Sidecars - dnssrv+_grpc._tcp.thanos-receive.monitoring.svc # Receive - dnssrv+_grpc._tcp.thanos-store.monitoring.svc # Historical Edge sites still lose connectivity for hours. They reconnect, the metrics backfill on their own, and the central Grafana dashboards show the full picture no matter which path the data took to get there.\nWhat this solved The original problem was that I couldn’t see clusters I couldn’t reliably reach, and I didn’t want to hand storage credentials to sites I don’t fully trust. Receive answers both. Edge sites ride out connectivity loss on their local WAL, they hold nothing but a write-only token to a single endpoint, and all the storage complexity stays in one place I do control.\nThe deeper reason I like it lines up with how I think about infrastructure generally. Push the simple thing to the edge and keep the complicated thing central, and each edge site becomes something I can reason about end to end. The firewall rules match what the security team would have asked for anyway. When a site goes dark now, the WAL fills, the central alert fires after an hour, and the gap backfills when the link returns. Nobody opens an incident for it.\nPull works when the network stays up, and push works when it doesn’t, so I run both. The four-hour factory outage from earlier in this post is what convinced me. Nobody touched anything, and the graph has no hole in it.","permalink":"https://tommeurs.nl/en/posts/thanos-remote-write-push-architecture/","title":"Thanos Remote Write: Push-Based Metrics for Edge and Multi-Cluster"},{"content":" TL;DR Talos and NixOS boot from a file that says what the machine is. That file is what an ISO 27001, SOC 2 or NIS2 auditor asks for, so the evidence collection shrinks to a git log, a diff with no output and an inventory dump. Below is the mapping from config to control, the scripts, and the part that stays hard work.\nAn ISO 27001 audit starts weeks before the auditor shows up, with someone collecting screenshots. Configuration panels, firewall rules, a dashboard that shows which patches went out. Then the Word documents that describe what the systems are supposed to do. Then the change tickets, dug out of the ticketing system, each one pointing at a “server maintenance” nobody can reconstruct six months later. Everyone treats this as the price of being in business. I did too, for years.\nThen I’d been running Talos and NixOS for a while, and one of the auditor’s questions started to sound different. The whole job is one question, asked twenty ways. Prove the system is configured the way you claim. On a declarative system that has a short answer. Here’s the file it boots from.\nThat’s what this post is about. I picked these systems for sovereignty and for resilience, and the properties that made me pick them, immutable images, reproducible builds and a Git history I can trust, are the properties an auditor is fishing for. I didn’t plan that. It was there when I looked.\nWhat auditors ask Strip the framework-specific vocabulary and ISO 27001, SOC 2, NIS2 and PCI-DSS circle the same four questions.\nWhat is your system configuration? Asset management, configuration management. How do you control changes? Change management, access control. Can you prove it? Audit trails, evidence. Is it consistent? No drift, no undocumented changes. In a traditional shop every one of these gets answered with paper. A Word document that describes what should be configured. A screenshot that proves it looked that way on one specific Tuesday. A change ticket that points at “server updates” which may or may not have happened the way the ticket says.\nNone of those documents is wrong on the day it’s written. The document says automatic updates are on, and they were, until the reorg. The screenshot was accurate when it was taken and says nothing about today. What’s in the evidence folder is a memory of the system, and nobody checks the memory against the machine until the next audit.\nWhen the config is the documentation Run Talos or NixOS and there’s no separate document to drift, because the configuration is the system. The machine refuses to boot in any state other than the one you declared, so the file and the running node can’t quietly disagree.\nTalos # This isn't documentation about the system. # This IS the system. machine: type: controlplane network: hostname: prod-control-01 install: disk: /dev/sda image: ghcr.io/siderolabs/installer:v1.6.0 sysctls: net.ipv4.ip_forward: \"1\" kubelet: extraArgs: rotate-server-certificates: \"true\" cluster: clusterName: production apiServer: admissionControl: - name: PodSecurity configuration: defaults: enforce: \"restricted\" When the auditor asks how the Kubernetes nodes are configured, you hand them this. Not an approximation, and not last quarter’s snapshot. The exact state, and the node is enforcing it right now.\nNixOS { # Security hardening - this is both config AND documentation security.sudo.wheelNeedsPassword = true; security.auditd.enable = true; security.audit.rules = [ \"-w /etc/passwd -p wa -k identity\" \"-w /etc/group -p wa -k identity\" ]; # Firewall configuration networking.firewall = { enable = true; allowedTCPPorts = [ 22 443 6443 ]; logRefusedConnections = true; }; # Automatic security updates system.autoUpgrade = { enable = true; allowReboot = false; dates = \"04:00\"; }; } Every control sits in code, and the auditor knows it’s live because NixOS won’t build a generation that doesn’t match it. There’s no “yes it’s configured, but we disabled it temporarily and forgot”. The build includes the control or the build fails.\nMapping declared state to the controls “Trust me, it’s all in Git” doesn’t pass an audit on its own. Somebody has to connect each piece of config to the control it satisfies, and that somebody is you. Here’s how I do it for four controls I get asked about.\nISO 27001: A.12.1.2 - Change Management The requirement says changes to systems shall be controlled. The traditional answer is change tickets, an approval workflow and documentation written by hand.\nOn a declarative system every change is a Git commit, a pull request needs a review before it merges, and merging to main is what deploys. The Git history is the change log.\n# Your change management evidence git log --oneline --since=\"2025-01-01\" -- infrastructure/ 2d8d00e Add NixOS vs Talos comparison e49f5bd Add Talos Linux blog post defb991 Switch image generation to DALL-E ... Every commit carries who made it, when, what changed, who approved it in the PR review, and why in the message. A ticketing system gives you a worse version of the same record, and it’s decoupled from the change itself.\nISO 27001: A.12.6.1 - Management of Technical Vulnerabilities The requirement says information about technical vulnerabilities shall be obtained and appropriate measures taken. With Talos the OS version is one line:\nmachine: install: image: ghcr.io/siderolabs/installer:v1.7.0 # Specific version Updating means changing that line and deploying. The Git history shows which version ran on any date, and if a new version turns out to carry a vulnerability, the rollback is a revert.\nNIS2: Article 21 - Cybersecurity Risk Management The requirement asks for appropriate technical and organisational measures to manage risk. Here’s what three of those measures look like in NixOS, network segmentation, encryption at rest and access control:\n{ # Network segmentation networking.vlans = { management = { id = 10; interface = \"eth0\"; }; production = { id = 20; interface = \"eth0\"; }; }; # Encryption at rest boot.initrd.luks.devices.\"encrypted\".device = \"/dev/sda2\"; # Access control services.openssh = { enable = true; settings = { PermitRootLogin = \"no\"; PasswordAuthentication = false; }; }; } Each control is visible, and the build enforces it.\nSOC 2: CC6.1 - Logical Access Controls The requirement says the entity implements logical access security software, infrastructure and architectures. This is the one I enjoy most, because the Talos answer to logical access control is to not have a login. There’s no SSH. All management goes through the API, every API call is authenticated with mTLS , and what you may do once you’re in is Kubernetes RBAC.\n# Talos machine config machine: features: rbac: true # Kubernetes RBAC enabled # No SSH configuration because SSH doesn't exist The auditor can check on the node itself that there’s no SSH daemon to break into. There’s nothing else to inspect, and no policy document that says please don’t.\nThe audit that runs itself Because everything lives in code, the evidence collection that used to eat weeks is three scripts on a schedule.\nConfiguration drift report #!/bin/bash # For Talos: compare declared vs actual for node in $(talosctl get members -o json | jq -r '.[] | .hostname'); do echo \"=== $node ===\" talosctl get machineconfig -n $node -o yaml \u003e /tmp/actual.yaml diff infrastructure/talos/$node.yaml /tmp/actual.yaml done No output means no drift, and the empty diff goes into the evidence folder as it is.\nNixOS system closure # Generate a complete list of all software and versions nix-store -q --requisites /run/current-system | \\ xargs -I {} nix-store -q --binding name {} 2\u003e/dev/null | \\ sort -u \u003e system-inventory.txt One command and you’ve got the complete software inventory, every package and library with the version that’s running. Auditors who are used to chasing that by hand light up when you hand them a file.\nGit-based audit trail # Generate change report for audit period git log --pretty=format:\"%h|%an|%ad|%s\" \\ --date=short \\ --since=\"2025-01-01\" \\ --until=\"2025-12-31\" \\ -- infrastructure/ \u003e annual-changes.csv Import it into Excel. Done.\nCompliant between audits There’s a step further. Pair the declarative systems with GitOps and the cluster corrects itself back to the declared state all year, whether or not anyone is coming to look.\n# ArgoCD Application for infrastructure apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: infrastructure spec: source: repoURL: https://github.com/company/infrastructure path: kubernetes/ destination: server: https://kubernetes.default.svc syncPolicy: automated: prune: true selfHeal: true # Automatically fix drift Someone patches a node by hand during an incident? ArgoCD reverts it on the next sync. So “were you compliant on March 14th” has an answer in the Git log and the sync history for March 14th, and I don’t have to remember.\nDoing the boring parts well None of this works on its own. Declarative infrastructure makes good compliance evidence possible, and then four habits decide whether you get it.\n1. Write meaningful commit messages # Bad git commit -m \"Update config\" # Good git commit -m \"Enable audit logging per ISO 27001 A.12.4.1 - Added auditd configuration for file access monitoring - Configured log retention for 90 days - Addresses finding from Q3 security review Ticket: SEC-1234\" Your Git history is going to be read by an auditor someday. Write it for that reader.\n2. Tag compliance-related config { # ISO 27001 A.12.4.1 - Event Logging security.auditd.enable = true; # ISO 27001 A.13.1.1 - Network Controls networking.firewall.enable = true; # NIS2 Art.21(2)(d) - Supply Chain Security nix.settings.require-sigs = true; } A comment that links a line of config to a control turns the audit from a scavenger hunt into a grep.\n3. Pin every version # Talos machine config machine: install: image: ghcr.io/siderolabs/installer:v1.6.0 # Not :latest Auditors want to know what ran on a specific date. Pinned versions plus Git history answer that for free. :latest throws the answer away.\n4. Automate evidence collection #!/bin/bash # Quarterly compliance report generator REPORT_DIR=\"compliance-reports/$(date +%Y-Q%q)\" mkdir -p $REPORT_DIR # System inventory nix-store -q --requisites /run/current-system \u003e $REPORT_DIR/inventory.txt # Configuration snapshots cp -r /etc/nixos $REPORT_DIR/nixos-config/ # Change log git log --since=\"3 months ago\" -- infrastructure/ \u003e $REPORT_DIR/changes.txt # Drift check ./scripts/check-drift.sh \u003e $REPORT_DIR/drift-report.txt echo \"Report generated in $REPORT_DIR\" Run it every quarter, archive the output, and the folder is ready when the auditor shows up.\nBe honest about the cost I’m not going to pretend this is free. Talos and NixOS both have a steeper learning curve than installing Ubuntu and clicking through a hardening guide. NixOS in particular will fight you for the first month while you learn the language. You trade a forgiving setup for a strict one, and the strictness lands on day one.\nThen there’s the mapping work. The system gives you enforced, version-controlled state, and somebody still has to sit down and connect each piece of config to the control it satisfies. The auditor won’t do that translation for you, and neither will the tooling. On day one it can feel like you’ve added a compliance task rather than removed one.\nThe difference is when you pay. This cost is paid once, and every audit after that reads the same repository. The screenshot-and-Word-document approach pays every cycle, and the bill grows with the estate.\nTry it on one cluster You don’t have to convert the whole estate to find out whether this holds. Pick one cluster. Stand it up on Talos, put the config in Git, point ArgoCD at it with selfHeal on. Run it for a quarter and see what your evidence folder looks like when the audit comes around, next to the team that’s still building a slide deck out of screenshots.\nSo the first paragraph of this post is still true. The screenshots, the Word documents and the tickets are what an audit asks for, and on a traditional setup they’re the only way to answer. What I’d missed for years is that they’re all stand-ins for one file, the one that says what the machine is. Talos and NixOS boot from that file. Hand it over, and what’s left is the mapping work.","permalink":"https://tommeurs.nl/en/posts/declarative-infrastructure-compliance-certification/","title":"Declarative Infrastructure as Compliance Documentation: Talos, NixOS, and Audit-Ready Systems"},{"content":" TL;DR KVM and QEMU are the engine inside Proxmox and most of the cloud you rent. On NixOS, with libvirt on top, that engine is one block of config, and live migration, memory ballooning, GPU passthrough, SR-IOV and CPU pinning are each a command or an XML snippet away. What Proxmox adds is the dashboard and the delegation, and I’m honest below about how much that’s worth.\nAsk around how to run a “real” hypervisor at home and you get the same shortlist. VMware, Hyper-V, or at minimum Proxmox. Something with a web UI, a clustering tab and a marketing page full of enterprise features. The shortlist is so well worn that running VMs without one of those products feels like cutting corners. I don’t blame anyone for that. The marketing is good.\nHere’s what the marketing leaves out. The part doing the work in every one of those products is a Linux kernel module that’s been production-grade for over a decade. KVM with libvirt does live migration, memory ballooning, CPU pinning, GPU passthrough, SR-IOV and nested virtualization. The features on the glossy page are kernel features, and the web UI sits in front of them.\nI run NixOS as my hypervisor. No Proxmox, no web UI, a declarative Nix config and virsh. This post walks through what that gives me, feature by feature, and then the part where I’d be lying if I said the CLI was always enough.\nKVM, QEMU and libvirt Three names come up in every command below, so a short round of introductions. KVM is the kernel module. It turns the Linux kernel into a type-1 hypervisor with hardware acceleration, so a guest runs at close to native speed. QEMU is the userspace program that gives the guest its virtual hardware, the disks and the network cards. libvirt is the management layer over both, with a stable API across virtualization backends and the virsh CLI on top.\nIt’s the same stack from the VMs on your laptop up to the cloud. AWS’s Nitro hypervisor is built on KVM. Google Cloud runs KVM. Most of the cloud you pay for is this stack with a billing system bolted on.\nThe NixOS config Here’s my hypervisor config, the whole thing:\n{ config, pkgs, ... }: { # Enable virtualization virtualisation.libvirtd = { enable = true; qemu = { package = pkgs.qemu_kvm; runAsRoot = true; swtpm.enable = true; # TPM emulation ovmf = { enable = true; # UEFI support packages = [ pkgs.OVMFFull.fd ]; }; }; }; # Add yourself to the libvirt group users.users.tom = { extraGroups = [ \"libvirtd\" ]; }; # Useful tools environment.systemPackages = with pkgs; [ virt-manager # GUI if you want it virt-viewer # For console access libguestfs # VM image manipulation win-virtio # Windows VirtIO drivers ]; # Bridge networking (optional, for bridged VMs) networking.bridges.br0.interfaces = [ \"enp1s0\" ]; networking.interfaces.br0.useDHCP = true; } Apply it and the machine is a hypervisor. There’s no installer, no setup wizard and no account to create on first boot, because the config is the install. Remove the block and rebuild, and the hypervisor is gone the same way it arrived.\nCreating VMs with virsh Let’s make one. First a disk:\n# Create a 50GB qcow2 disk qemu-img create -f qcow2 /var/lib/libvirt/images/ubuntu-server.qcow2 50G Then define the VM. You can write the libvirt XML by hand, or let virt-install write it for you:\nvirt-install \\ --name ubuntu-server \\ --memory 4096 \\ --vcpus 2 \\ --disk /var/lib/libvirt/images/ubuntu-server.qcow2 \\ --cdrom /var/lib/libvirt/images/ubuntu-24.04-server.iso \\ --os-variant ubuntu24.04 \\ --network bridge=br0 \\ --graphics vnc That’s a running VM. The day-to-day is five commands:\nvirsh list --all # List all VMs virsh start ubuntu-server # Start VM virsh shutdown ubuntu-server # Graceful shutdown virsh destroy ubuntu-server # Force stop virsh console ubuntu-server # Serial console Live migration This is the feature people are most sure needs a commercial product. Moving a running VM to another host with no downtime sounds like something with a licence key attached.\nIt’s a virsh subcommand. Live migration copies the VM’s memory to the target host while the guest keeps running, then does the final switchover in milliseconds. The guest doesn’t notice.\nYou need three things in place:\nShared storage (NFS, Ceph , GlusterFS) or the same disk path on both hosts The same CPU architecture, or a compatible CPU model libvirt running on both hosts # From the source host virsh migrate --live ubuntu-server qemu+ssh://target-host/system # With options for better performance virsh migrate --live --persistent --undefinesource \\ --copy-storage-all \\ ubuntu-server qemu+ssh://target-host/system One command, and the VM is running on the other host. If you don’t have shared storage, --copy-storage-all moves the disk along with the memory.\nIf you want the target host picked for you, that’s a loop around the same command:\n#!/bin/bash # Simple load balancer - migrate VM to least loaded host HOSTS=(\"host1\" \"host2\" \"host3\") VM=\"ubuntu-server\" # Find host with lowest load TARGET=$(for h in \"${HOSTS[@]}\"; do echo \"$(ssh $h 'cat /proc/loadavg | cut -d\" \" -f1') $h\" done | sort -n | head -1 | cut -d\" \" -f2) virsh migrate --live $VM qemu+ssh://$TARGET/system That’s a load balancer in a few lines of bash. Proxmox’s HA tab drives the same QEMU migration.\nMemory ballooning Ballooning lets you change a VM’s memory while it runs. The guest has a balloon driver that hands pages back to the host or asks for more, no reboot needed.\nEnable it in the VM definition:\n\u003cmemballoon model='virtio'\u003e \u003caddress type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/\u003e \u003c/memballoon\u003e Then resize at runtime:\n# Set current memory to 2GB (VM has 4GB max) virsh setmem ubuntu-server 2G --live # Give it back 4GB virsh setmem ubuntu-server 4G --live This is what makes overcommitting safe. Hand out 64GB across VMs on a 32GB host and let the balloon drivers sort out who needs what.\nGPU passthrough The one homelab people obsess over. Hand a physical GPU to a VM and the guest gets near-native graphics, which is what a gaming VM, a machine learning box or a hardware transcoder wants.\nFirst IOMMU, in the NixOS config:\n{ boot.kernelParams = [ \"intel_iommu=on\" # or amd_iommu=on for AMD \"iommu=pt\" ]; # Isolate the GPU from the host boot.extraModprobeConfig = '' options vfio-pci ids=10de:2204,10de:1aef ''; boot.initrd.kernelModules = [ \"vfio_pci\" \"vfio\" \"vfio_iommu_type1\" ]; } Find the GPU’s PCI IDs:\nlspci -nn | grep -i nvidia # 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation ... [10de:2204] # 01:00.1 Audio device [0403]: NVIDIA Corporation ... [10de:1aef] Then hand both functions, video and audio, to the VM:\n\u003chostdev mode='subsystem' type='pci' managed='yes'\u003e \u003csource\u003e \u003caddress domain='0x0000' bus='0x01' slot='0x00' function='0x0'/\u003e \u003c/source\u003e \u003c/hostdev\u003e \u003chostdev mode='subsystem' type='pci' managed='yes'\u003e \u003csource\u003e \u003caddress domain='0x0000' bus='0x01' slot='0x00' function='0x1'/\u003e \u003c/source\u003e \u003c/hostdev\u003e The VM talks to the GPU directly. Install the vendor drivers in the guest and you have full hardware acceleration with no host in the path.\nNetwork card passthrough and SR-IOV Same principle for network cards. A 10GbE or 25GbE card can go straight to a VM:\n# Find the NIC lspci -nn | grep -i ethernet # Pass it through like the GPU SR-IOV (Single Root I/O Virtualization) is the more interesting option. It splits one physical NIC into several virtual functions, each of which you can hand to a different VM:\n# Enable 4 virtual functions on the NIC echo 4 \u003e /sys/class/net/enp5s0f0/device/sriov_numvfs # List them lspci | grep \"Virtual Function\" Each VF has its own PCI address. The VM gets hardware-speed networking without virtio in between.\nCPU pinning For latency-sensitive workloads you want the VM’s vCPUs nailed to specific host cores, so the scheduler stops moving them around:\n\u003cvcpu placement='static'\u003e4\u003c/vcpu\u003e \u003ccputune\u003e \u003cvcpupin vcpu='0' cpuset='4'/\u003e \u003cvcpupin vcpu='1' cpuset='5'/\u003e \u003cvcpupin vcpu='2' cpuset='6'/\u003e \u003cvcpupin vcpu='3' cpuset='7'/\u003e \u003cemulatorpin cpuset='0-1'/\u003e \u003c/cputune\u003e That pins the VM’s 4 vCPUs to host cores 4-7 and keeps the emulator threads on cores 0-1. Add isolcpus to the kernel parameters and the latency gets close to bare metal.\nVMs in the same config as the host This is where NixOS pulls ahead of a stock Linux box with libvirt on it. The VMs themselves can live in the config:\n{ virtualisation.libvirtd.enable = true; # Define VMs as Nix expressions systemd.services.\"libvirt-guest-ubuntu\" = { after = [ \"libvirtd.service\" ]; requires = [ \"libvirtd.service\" ]; wantedBy = [ \"multi-user.target\" ]; serviceConfig = { Type = \"oneshot\"; RemainAfterExit = \"yes\"; }; script = '' ${pkgs.libvirt}/bin/virsh define /etc/libvirt/qemu/ubuntu-server.xml || true ${pkgs.libvirt}/bin/virsh start ubuntu-server || true ''; preStop = '' ${pkgs.libvirt}/bin/virsh shutdown ubuntu-server || true sleep 10 ${pkgs.libvirt}/bin/virsh destroy ubuntu-server || true ''; }; } Or with NixVirt, which is cleaner:\n{ virtualisation.libvirt.connections.\"qemu:///system\" = { domains = [{ definition = ./vms/ubuntu-server.xml; active = true; }]; }; } The VMs are now in the same file as the host. Versioned, reproducible, and gone the moment you remove them from the file. There’s no stale definition somewhere that nobody remembers creating, because the file is the only place a definition can be.\nWhere the CLI hurts So far this reads like the simpler stack wins. I’ve run it for years and I’m not going to pretend that’s the whole picture. Every feature is there. The day-to-day around the features is rougher than a good UI, and I’d rather say so than have you find out.\nFour things get tedious. virt-install has a flag for everything, so I’m in the man page every time I make a VM. I keep wrapper scripts, they work, and they encode assumptions only I know about.\nMonitoring means scripts that poll virsh for status and resource usage, or a Grafana dashboard with a libvirt exporter, which is a weekend of its own. Storage pools, volumes and snapshots are all doable from virsh, but the commands are long and I fat-finger them. And which VM sits on which bridge with which IP lives in my scripts or my head. There’s no screen for it.\nThree things scripts do badly. A quick overview, the “how is everything doing” glance, is virsh list plus virsh domstats plus a parser, and that’s no match for a dashboard. Console access is fine locally with virt-viewer, and a web console is nicer when I’m remote and half awake. Delegation, letting someone manage two specific VMs without handing them the host, is awkward with raw libvirt.\nWhy Proxmox exists Which brings me to Proxmox. The same KVM, QEMU and libvirt underneath, wrapped in a web UI that does everything in the previous section without a script.\nDo I think you need it? For a homelab with one admin, no. NixOS plus virsh plus a few scripts has been fine for me for years. Where the UI earns its place is onboarding, where a new person reads the infrastructure off a screen instead of out of my scripts. Incidents, where clicking beats recalling virsh syntax under stress. Visibility, with every VM, its status and its network in one view.\nProxmox also ships clustering, Ceph integration and backup management, and rebuilding those from scripts is real work.\nWhat Proxmox sells is the operations layer. The hypervisor underneath it is the one you already have.\nBack to the shortlist The shortlist was right about the dashboard and wrong about the hypervisor. VMware, Hyper-V and Proxmox run the same kernel module as my NixOS box, and NixOS adds declarative config and version control that none of them have on that axis. What those three sell is the dashboard, the delegation and the ergonomics during an incident.\nWhich one you want depends on which of those two you’d rather build yourself. If you want to understand virtualization down to the kernel module, start with NixOS, KVM and virsh. Nothing is hidden, and every “enterprise” feature is a command you typed. As I argued in Sovereign Infrastructure, a black box you can’t inspect is its own kind of risk, and there’s no black box in this stack.\nIf you run production with a team, the Proxmox UI is buying you an operations layer, and that’s a fair thing to buy. For my homelab, understanding the machine is half the reason it exists, so the config at the top of this post is my hypervisor. Declarative and versioned, with no vendor between me and the kernel. It’s still the whole install.","permalink":"https://tommeurs.nl/en/posts/nixos-hypervisor-kvm-qemu/","title":"NixOS as a Hypervisor: KVM and QEMU Can Do Everything"},{"content":" TL;DR Talos and NixOS are both declarative and both can be immutable, and that’s about where the resemblance stops. Talos runs Kubernetes and nothing else, with no shell to drift in. NixOS is a full Linux where the whole machine is one Nix config. I run Talos on my cluster nodes and NixOS on the machines around them, and this post is how I got to that split.\nI’ve written about Talos Linux as the immutable Kubernetes OS, and I’ve compared Arch vs NixOS for workstations. After both, the same question kept arriving in my inbox. Why not NixOS on the Kubernetes nodes too?\nFair question. On paper the two look like siblings. NixOS and Talos are both declarative. Both can be immutable. Both put the configuration of a machine under version control. So why would you pick one over the other for a cluster?\nMy bias first. I run a homelab where I want to understand and own every layer, and I’ve got a low tolerance for drift. I’ve run both of these in production, and I like Nix as a language, which not everyone does. So I come to this with a preference already formed. I’ve tried to give both a fair run below anyway.\nI’ll compare them on the things that moved my own decision: how you configure them, what an attacker finds on the box, what else the node can run, how upgrades and rollbacks go, how Kubernetes gets onto the node, and what debugging looks like when something breaks. The feature lists are on both project homepages already.\nWhere the two disagree on purpose Talos Linux runs Kubernetes and nothing else. Every design decision in it bends toward that one job.\nNixOS is a general-purpose Linux that happens to be good at Kubernetes. It can do anything a traditional distribution does, except that the whole machine is described in one declarative config.\nEvery trade-off below comes from that one difference in intent.\nConfiguration Talos: YAML Talos configuration is plain YAML:\nmachine: type: controlplane network: hostname: node-01 interfaces: - interface: eth0 dhcp: true install: disk: /dev/sda cluster: clusterName: production controlPlane: endpoint: https://k8s.example.com:6443 You apply it with talosctl apply-config and the node converges to that state. There’s nothing else to learn. The schema is documented and the fields do what they say.\nNixOS: the Nix language NixOS configuration is written in Nix, which is a functional language with ideas of its own:\n{ config, pkgs, ... }: { networking.hostName = \"node-01\"; services.k3s = { enable = true; role = \"server\"; extraFlags = toString [ \"--cluster-init\" \"--disable traefik\" ]; }; # You can do anything here services.prometheus.exporters.node.enable = true; virtualisation.docker.enable = true; system.stateVersion = \"24.05\"; } You apply it with nixos-rebuild switch and the system converges to what you declared. Same idea, different vocabulary.\nTalos’s YAML is limited and you’ll learn it in an afternoon. Nix is a lot more powerful, and it took me weeks before it stopped feeling alien. I say that as someone who ended up liking it.\nSo the question is scope. If the node only needs Kubernetes, the YAML is enough and the Nix learning curve buys you nothing. If the node needs Kubernetes plus a pile of other things, that same curve is what pays for them.\nSecurity Talos: nothing to find Talos gets its security by deleting things:\nNo SSH daemon No shell No package manager Read-only root filesystem Minimal userspace (just containerd , kubelet, and essentials) An attacker who compromises a container and escapes to the host finds almost nothing there. No tools to pivot with, no obvious way to persist, no package manager to pull malware down with. That’s blast radius reduction as an OS design, and I care about it more than I care about most features.\nNixOS: hardened when you say so NixOS can be hardened a long way, but every step is opt-in:\n{ # Disable SSH password auth services.openssh = { enable = true; settings.PasswordAuthentication = false; }; # Firewall networking.firewall = { enable = true; allowedTCPPorts = [ 6443 ]; }; # Read-only store (always true) # But /etc, /var, /home are still mutable } The /nix/store is always read-only, which is good. /etc, /var and /home stay mutable unless you do something about them. You can, and it’s a thing you have to remember to do.\nSo Talos is hardened on delivery and NixOS is hardened after you’ve put the work in. For a node whose only job is Kubernetes, the paranoid defaults are the ones I want, and I’d rather not even have the option to skip them.\nWhat else the node can run This is where the two pull apart hardest.\nNixOS: anything A NixOS node can run Kubernetes and, next to it, anything else you’d put on a Linux box. The Prometheus node exporter, a log shipper or monitoring agents as systemd services instead of a pod. A bare-metal database or a legacy app that has no business in a container. Any of the 80,000+ packages in nixpkgs. The debugging tools you want on the node itself, one config line away.\nHere’s a PostgreSQL next to Kubernetes, for testing:\n{ services.k3s.enable = true; services.postgresql = { enable = true; package = pkgs.postgresql_15; }; } Talos: nothing else, on purpose Talos’s restrictions are the product. You can’t make an undocumented change, because there’s no shell to make it in. Every change goes through the config, then the API, then apply. The node is its config and there’s no second place to look. And what doesn’t exist on the box can’t be exploited.\nIf your nodes only ever run Kubernetes workloads, those walls stop you making mistakes. If your nodes need to do more, the same walls are a cage, and NixOS is the way out of it.\nUpgrades and rollback Talos: two partitions talosctl upgrade --nodes 192.168.1.10 \\ --image ghcr.io/siderolabs/installer:v1.7.0 Talos writes the new OS to an alternate partition and reboots into it. If the new image doesn’t boot, it falls back to the previous one by itself. The upgrade either lands completely or it doesn’t land, and you’re not watching a console either way.\nNixOS: generations nixos-rebuild switch --flake .#node-01 Every configuration change gives you a new generation, and each generation is bootable from GRUB. Going back is picking an earlier one:\nnixos-rebuild switch --rollback # Or select from boot menu NixOS keeps several generations by default, so you can drop back to the configuration from weeks ago.\nBoth rollback stories are good, which is rarer than it sounds. Talos does it for you when a boot fails. NixOS keeps as many generations as you like and lets you pick.\nGetting Kubernetes onto the node Talos: it’s already there Talos ships with everything a cluster needs. The Kubernetes components are built in, the CNI config lives in the machine config, etcd is managed for you, certificates rotate automatically, and an upgrade drains the node, upgrades it and uncordons it again. You don’t install Kubernetes on Talos. It comes with the OS.\nNixOS: pick one NixOS gives you options:\n# Option 1: K3s (lightweight) services.k3s = { enable = true; role = \"server\"; }; # Option 2: Full Kubernetes via kubeadm services.kubernetes = { roles = [\"master\" \"node\"]; masterAddress = \"k8s.example.com\"; }; # Option 3: Just the kubelet services.kubernetes.kubelet.enable = true; You decide how Kubernetes gets onto the box. More options, and every option is a decision you now own.\nTalos means fewer decisions and defaults that were chosen for Kubernetes specifically. NixOS lets you wire it up however suits you, and then you’re the one who has to stand behind the wiring.\nDebugging Talos: through the API # View logs talosctl logs kubelet --nodes 192.168.1.10 # Get system info talosctl get members # Read files talosctl read /etc/hosts # Dashboard (read-only TUI) talosctl dashboard Everything goes through the API. No SSH, no shell. There’s a debug container for emergencies, and it’s inconvenient on purpose so you don’t reach for it out of habit.\nNixOS: SSH in # SSH in ssh root@192.168.1.10 # Use any tool htop journalctl -u k3s kubectl get nodes vim /etc/nixos/configuration.nix It’s a Linux box. Every tool you know already works. That’s comfortable, and it’s also how drift starts, because every SSH session is a chance to fix something that never makes it back into Git. I’ve done it.\nTalos makes you debug through the API, the logs and the metrics. NixOS lets you debug however you like, including the ways that leave the node different from its config. Whether that freedom helps or hurts depends on how disciplined you are on a bad day.\nWhat I’d choose After running both, this is how I decide.\nChoose Talos when:\nKubernetes is the only workload on these nodes you want the smallest attack surface you can get preventing drift matters more to you than flexibility you’re fine managing machines through an API the team already works declaratively Choose NixOS when:\nthe same nodes need non-Kubernetes workloads you want a full Linux with the declarative benefits you need packages that don’t exist as Talos extensions the team is already invested in Nix you want to customise past what Talos allows My setup I run both, in different roles. The homelab Kubernetes nodes are Talos. They only run Kubernetes, and I want the security and the simplicity of a cluster with nowhere to hide. The auxiliary servers are NixOS. They run Postgres, backup services and the monitoring infrastructure I keep outside Kubernetes on purpose.\nThat gives me cluster nodes that can’t drift, and NixOS machines for everything that doesn’t belong in a pod. Each one sits where its limits are useful.\nThe hybrid approach Some teams run NixOS on the control plane nodes, where they want debugging access, and Talos on the workers, where they want the lockdown. It works. It also means maintaining two shapes of node for as long as the cluster exists.\nThere’s another path I’ve seen work. Start on NixOS because it’s familiar, then move to Talos once the team is used to declarative operations. NixOS teaches the mental model gently. Talos enforces it.\nThe KubeVirt option There’s a third approach worth putting on the table. KubeVirt, with NixOS running as VMs inside Kubernetes. Talos, or any other minimal OS, goes on the bare metal, and the NixOS workloads become VMs that Kubernetes manages.\nThe stack looks like this:\nKubeVirt Stack\nArgoCD (GitOps)\nKubeVirt VMs (NixOS, other OSes)\nKubernetes (K3s/Talos)\nBare Metal (Ansible provisioned)\nWhy this works Your VM definitions become Kubernetes manifests:\napiVersion: kubevirt.io/v1 kind: VirtualMachine metadata: name: nixos-workload-01 spec: running: true template: spec: domain: resources: requests: memory: 4Gi cpu: 2 devices: disks: - name: nixos-disk disk: bus: virtio volumes: - name: nixos-disk persistentVolumeClaim: claimName: nixos-workload-01-pvc With ArgoCD driving it, this VM definition lives in Git. The NixOS config inside the VM lives in Git too. Top to bottom, the whole thing is declarative and has a history.\nThe full declarative stack Combine that with:\nAnsible for the bare metal provisioning, installing Talos or K3s on the physical nodes ArgoCD for everything in Kubernetes, KubeVirt VMs included NixOS flakes for the VM configurations, pulled from Git when the VM boots Then your entire infrastructure, from bare metal to VMs to Kubernetes workloads, is defined in Git. Change a VM’s NixOS config, commit, ArgoCD syncs, the VM rebuilds. Full audit trail, full reproducibility, and exactly the sovereignty story I keep coming back to.\nWhen this makes sense This earns its complexity when:\nyou have mixed workloads, some containerised and some that need a full VM you want one control plane, Kubernetes, for all of it you have workloads that can’t be containerised but still want them under GitOps you’re already running KubeVirt for other reasons The cost is real. You’re now running a virtualisation layer inside Kubernetes, with everything that implies. But if you need VMs anyway, managing them through the same GitOps pipeline as the rest of your estate is a strong place to be.\nExample: a database on a NixOS VM Say you want a PostgreSQL server with kernel tuning that won’t fit in a container. Instead of maintaining a separate NixOS server by hand:\nDefine the VM in a KubeVirt manifest, in Git The VM boots NixOS and pulls its config from a flake, in Git ArgoCD manages the VM lifecycle Backups, monitoring and networking all run through Kubernetes The database gets NixOS’s flexibility, you get Kubernetes’s scheduling, and everything is in Git.\nWhich constraints do you want NixOS and Talos both point at the same kind of infrastructure: declarative, reproducible, version-controlled. What differs is which constraints they put on you.\nTalos boxes you into Kubernetes-only, API-only, minimal-surface operations. That’s what you want when you’d rather the system enforce the discipline than rely on yours.\nNixOS holds you to a declarative configuration and leaves you the full run of a Linux box. That’s what you want when you need the range and you trust the team to use it without drifting.\nFor pure Kubernetes nodes I run Talos. For everything around them I run NixOS. Both projects ship regularly and both communities are active, so if you’re picking today, pick by what else the node has to run. That was the question that settled it for me.","permalink":"https://tommeurs.nl/en/posts/nixos-vs-talos-kubernetes-nodes/","title":"NixOS vs Talos for Kubernetes Nodes: Two Flavors of Immutable Infrastructure"},{"content":" TL;DR Talos Linux is a Linux that runs Kubernetes and nothing else. No SSH, no shell, no package manager. Every change goes through an API and a YAML file in Git. I’ve run my nodes on it for three years, and this post is how I stopped missing the shell.\nThe first time I tried to SSH into a Talos node, I got nothing. No shell, no connection, not even a login prompt. I sat there for a minute wondering how I was supposed to debug a machine I couldn’t get into.\nThat was three years ago. I haven’t run Kubernetes on anything else since.\nBefore Talos, my nodes looked the way most Kubernetes nodes look. A general-purpose distro, hardened with a CIS benchmark script, an SSH key added, a config management tool to keep drift in check, and then two years of SSH-ing in to fix the things the tool didn’t catch. Every node had its own history, and only some of it was written down.\nWhat Talos leaves out Talos Linux is a Linux distribution built to run Kubernetes and nothing else. “Distribution” is generous, because Talos throws away most of what makes a Linux box feel like one. There’s no SSH, so you can’t shell into a node. There’s no package manager, so no apt-get, no yum, nothing to install. There’s no shell at all, no bash, no sh. The root filesystem is read-only. Everything you’d do to the machine goes through a gRPC API.\nMy first reaction was “that’s insane, how do you get anything done”. Yours probably is too. The answer took me a while, and it’s the rest of this post.\nThe fix that lived on one disk Years before Talos, production went down in the middle of the night. I SSH’d into the bad node, did some panicked debugging, found the problem, and fixed it by hand-editing a config file and bouncing a service. Back to bed.\nThat fix lived in exactly one place, on that node’s disk. It wasn’t in Git and it wasn’t documented anywhere. Months later we replaced the node and the problem came straight back, because nobody remembered the hand edit that had been keeping it alive.\nThat’s the cost of a mutable node, and it compounds. Every SSH session is a chance for drift. Every quick fix is a debt you pay later, usually during the next incident, and without the context you had the first time.\nNow picture a node you can’t drift by hand, because there’s no way in. That’s what Talos is. You can’t SSH in and tinker. Every change goes through the API, so every change can be tracked, versioned and reproduced. My hand edit would have been a one-line diff in Git, reviewed, and the replacement node would have had it from day one.\nWhen I explain this, the pushback comes fast. What about debugging? What about emergencies?\nDebugging without a shell Fair. Talos ships talosctl, a CLI that talks to the Talos API, and it covers the things you’d normally open a shell for:\n# Get system logs talosctl logs kubelet # Stream kernel messages talosctl dmesg -f # List running processes talosctl processes # Get node health talosctl health # Read any file from the node talosctl read /etc/hosts # Get full machine configuration talosctl get machineconfig Everything you’d normally do over SSH is here, with one difference that matters. The API has access control and audit logging, and it can be scoped to specific operations. SSH is an all-or-nothing root shell, which is exactly what an attacker wants as well.\nFor the moments where you want to sit and watch a machine, there’s a dashboard:\ntalosctl dashboard That’s a read-only TUI with system stats, logs and node status. For real emergencies there’s a debug container, a privileged container with full host access. It’s awkward to use, and that’s deliberate, so you don’t reach for it out of habit.\nOne YAML file per node A Talos node is configured with a single YAML machine config. A trimmed-down example:\nmachine: type: controlplane token: \u003cbootstrap-token\u003e ca: crt: \u003cca-certificate\u003e key: \u003cca-key\u003e network: hostname: node-01 interfaces: - interface: eth0 dhcp: true install: disk: /dev/sda image: ghcr.io/siderolabs/installer:v1.6.0 cluster: clusterName: homelab controlPlane: endpoint: https://192.168.1.100:6443 That file describes everything about the node. Network settings, disk layout, Kubernetes version, kernel arguments, system extensions, CNI configuration.\nYou apply it with one command:\ntalosctl apply-config --nodes 192.168.1.10 --file controlplane.yaml Talos doesn’t so much configure the node as make the node converge to the file. Change the file, apply again, and the node moves to the new state. It’s the same declarative model Kubernetes uses for pods, one layer further down.\nGitOps one layer down Compare that with where OS configuration usually lives. Ansible playbooks that have drifted from what’s on the box. A config management tool nobody fully trusts. A layer of tribal knowledge built up in SSH sessions that left no trace. The source of truth is “ask the person who set it up”, and that person leaves eventually.\nBecause a Talos node is one YAML file, it drops straight into the same GitOps workflow as everything else. My repo looks like this:\ninfrastructure/ ├── talos/ │ ├── controlplane-1.yaml │ ├── controlplane-2.yaml │ ├── controlplane-3.yaml │ ├── worker-1.yaml │ └── worker-2.yaml ├── kubernetes/ │ └── ... (manifests managed by ArgoCD) └── ... When I need to change something on a node, say a kernel argument or a network setting, I edit the YAML, commit, and apply. The commit is the documentation. It’s reviewable in a PR and reproducible on the next node I build. That’s the whole reason I do it this way.\nWhat an attacker finds on the box Take away the shell and the package manager and a whole class of attacks stops working. No package manager, so no curl | bash installing malware. No SSH, so nothing to brute-force. A read-only filesystem, so no swapping out system binaries. A minimal userspace, so there’s less to attack in the first place.\nThe node runs only what Kubernetes needs: containerd , kubelet, and a few supporting services. No cron, no systemd-resolved, none of the background daemons a general-purpose distro ships because you might want them someday.\nSo say an attacker pops a container and escapes to the host. Where do they land? A read-only filesystem with no shell, no package manager, and nowhere to persist. It lines up with zero trust principles, in that the node doesn’t even trust itself to stay configured correctly and keeps enforcing the declared state.\nUpgrades with a rollback built in Upgrades are the part I like most:\ntalosctl upgrade --nodes 192.168.1.10 \\ --image ghcr.io/siderolabs/installer:v1.7.0 Talos pulls the new OS image, writes it to a second partition, and reboots into it. If the new version fails to come up, the node rolls back to the previous one on its own.\nThat’s a transactional upgrade. No half-applied state, no Saturday spent on a kernel that panicked halfway through the upgrade. It completes, or it rolls back whole.\nFor Kubernetes version bumps you update the machine config with the new version and apply it. Talos does the draining and cordoning if you use the Talos-aware upgrade path.\nWhen not to use Talos This doesn’t fit everyone. Stick with a traditional distro if:\nyou need specific packages that aren’t available as Talos extensions you’re still learning Linux and the abstraction hides too much of it your team isn’t ready for API-driven management you run non-Kubernetes workloads on the same nodes Talos does Kubernetes and nothing else. If that’s not your main use case, you’ll spend your time fighting the design instead of using it.\nDebugging also has a steeper curve up front. When something breaks you can’t vim /etc/whatever your way out. You have to learn the Talos tools and concepts first. Long term that’s a benefit, because it forces better habits, but it’s a real cost during adoption and I’d rather say so.\nTrying it in Docker The fastest way to try Talos is with Docker:\n# Create a local cluster talosctl cluster create --name demo # Get kubeconfig talosctl kubeconfig --nodes 127.0.0.1 # You're now running Talos locally kubectl get nodes For something closer to real, start with a small homelab cluster. My homelab runs on refurbished mini-PCs, which is a cheap way to experiment with Talos before you bet production on it. The Talos documentation is good, with guides for bare metal, AWS, GCP, Azure, VMware and more.\nWhat the toolchain assumes So why doesn’t everyone run nodes this way? Because the whole industry assumes the opposite. Traditional infrastructure management is built around the idea that at some point you’ll need to get onto the machine and fix it by hand. SSH keys, bastion hosts, jump boxes, console access. The entire toolchain exists to give you a way in.\nTalos removes the way in. The bet is that you never need one, because the node describes itself and converges on its own. If something’s wrong, you fix the file and reapply. That is rigid, and the rigidity is what stops the drift.\nIn practice it’s less dramatic than it sounds. Every incident I’ve had on Talos went the same way. Read the logs through the API, spot the misconfiguration, fix the file, apply. The fix is a commit, so it documents itself. Nobody has to ask afterwards whether someone SSH’d in and changed something, because nobody can.\nThree years in, I don’t miss SSH. The API gives me everything a shell gave me, plus access control, an audit trail, and a node I can rebuild from a file. Talos has a strong opinion about how a node should be run. Mine turned out to match it.","permalink":"https://tommeurs.nl/en/posts/talos-linux-immutable-kubernetes/","title":"Talos Linux: The Immutable Kubernetes OS That Changed How I Think About Nodes"},{"content":"Most of us ship software the same way now. Pick a framework, pull in a few hundred dependencies, and the service idles at a gigabyte of RAM before it does its first useful piece of work. Nobody profiles it. If it’s slow, you scale up. Compute is cheap, your time isn’t, and the bill goes to someone else. I don’t say that as a complaint. I’ve done it, and I’ll do it again on the next deadline.\nI used to live in a different world. I’d spend hours, sometimes whole weekends, squeezing a program onto hardware that “couldn’t possibly” run it. Getting it to run anyway was the best feeling I knew. The demoscene was my inspiration. I’d watch an intro do impossible things in 64 kilobytes and sit there asking myself how on earth they’d done that.\nThat instinct feels quaint now, and I know it. But I think we lost something when we stopped caring, and the cost of losing it has started showing up on the electricity meter.\nWhere I learned to count bytes In the scene, the constraints were the whole point. A 64KB intro had to carry its own graphics, music and animation, most of it generated at runtime because there was no room to store any of it. Every byte was a decision somebody had made.\nGroups like Future Crew, Farbrausch and Conspiracy turned that limit into art. .kkrieger is the one I still bring up. A complete first-person shooter, with textures and levels and enemies, in 96KB. A single modern web font is bigger than that.\nWas it masochism? A bit. But mostly it was craft. You had to understand the hardware well enough to get performance out of it the spec sheet said wasn’t there. That is the same understanding I still value most in the work I do now. When you know why something is fast, you know why it breaks.\nHow we stopped Then the ground shifted.\nCloud made infinite scale a checkout button. Moore’s Law kept paying out. Storage went from precious to free. And the habit of optimising drained away, because the economics stopped rewarding it.\nWhy spend three days tightening an algorithm when a bigger instance costs less than your afternoon? Why compress assets when bandwidth is metered in fractions of a cent? Why profile anything when the customer can buy more RAM? Every one of those questions has a defensible answer. I’ve given the lazy one plenty of times myself.\nHere’s what that looks like in the enterprise environments I’ve worked in:\nApplications that need 32GB of RAM to display a spreadsheet Docker images measured in gigabytes for services that do almost nothing JavaScript bundles that take longer to parse than the interaction they enable “Microservices” that demand more resources than the monolith they replaced At some point we decided, without ever voting on it, that developer time is expensive and compute is cheap. So we spend the compute to save the time. That was a fair trade when the only thing at stake was a cloud bill. I’m less sure it still is.\nThe bill that doesn’t come in the post Every CPU cycle you don’t need still burns electricity. A bloated image still needs a disk to sit on and a network to move across. An over-provisioned Kubernetes cluster draws power around the clock to keep idle cores warm.\nData centres already use somewhere around 1 to 2 percent of the world’s electricity, and that figure is climbing fast now that AI workloads have arrived on top. When your inefficient code runs on one laptop, nobody notices. When it runs on ten thousand instances, the waste runs on ten thousand instances too. At that scale the gap between O(n) and O(n²) stops being a whiteboard exercise. It turns into kilowatt-hours, and further up the chain into generation capacity and carbon.\nThis is the part I keep coming back to. We’ve treated compute as disposable. Take what you need, scale on demand, never look at the waste. That worked as long as the second bill never arrived, and it’s starting to arrive.\nWhat a demoscener would do with a cluster Most of the old mindset survives the jump from 64KB intros to production systems. I’ve ended up with four habits from it.\nMeasure first. Demosceners never guessed at what was slow. They profiled until they knew. We have better observability tooling than they could have dreamed of, and we mostly point it at debugging instead of efficiency. The data is already there.\nQuestion the default. Does this service need 2GB of base memory, or did it ship with that number and nobody ever checked? Does this JSON response need every field? Does this cron job need to run every minute? Usually the answer is “no idea, it’s always been like that”.\nTreat the constraint as a feature. Give yourself a 50MB image limit. Force a service to run in 256MB. I’ve watched limits like that produce cleaner designs, the same way 64KB forced the scene to invent procedural generation because nothing else would fit.\nMake efficiency visible. In the scene, small and fast earned respect. In most companies it’s invisible work that never makes the sprint review demo. We celebrate the feature. We could just as well celebrate the engineer who halved the cluster footprint.\nWhat you can start this week I’m not suggesting you write a demo. But several habits from that world map straight onto how we run infrastructure, and none of them need a budget.\nProfile before you scale. Before you reach for a bigger instance, find out why the current one is struggling. In my experience it’s one bad database query or a slow memory leak, and the fix is cheaper than the upgrade.\nRight-size everything. Kubernetes resource requests and limits decide how many cores sit idle and warm. Set them from what the service measures, because the values it shipped with were a guess.\nMeasure energy directly. Scaphandre, Kepler and Cloud Carbon Footprint show the actual energy draw of your workloads. Most teams have never looked at it once.\nSchedule with the grid in mind. Run batch jobs off-peak, or when the grid is greener. Some providers offer carbon-aware scheduling out of the box now, and it costs nothing to turn on.\nCache aggressively. A computation you never run draws zero watts. Caching pays you back in latency and in electricity at the same time.\nPick the language for the job. Rust and Go burn far less energy than Python or Ruby for the same work. That doesn’t matter for a script that runs once a day. At scale it does.\nWhy it hasn’t happened yet I know why I haven’t done all of this on every service I run. The incentives point the other way. Optimisation is real engineering effort with no demo at the end, and most roadmaps reward the new feature over the leaner one. A cloud bill is a number a CFO understands. A carbon footprint mostly isn’t, yet.\nI’ve shipped the bloated thing on a deadline too. None of this is a moral failing. It’s a default we drifted into because every individual shortcut was rational at the time.\nOne service If I had to point at where to begin, it’d be one service nobody would miss if it wobbled. Profile it, cut its requests to what it measures, put an energy meter next to it and see what the number does.\nThe demoscene showed that a hard limit makes you inventive. That was true when the limit was a 64KB file. It’s true again now that the limit is the grid, and I’d rather relearn it from the scene than from the meter.\nThe demoscene is now UNESCO-recognized cultural heritage in several countries.","permalink":"https://tommeurs.nl/en/posts/demoscene-optimization-sustainability/","title":"The Lost Art of Software Optimization: What Demoscene Taught Us About Sustainability"},{"content":" TL;DR I’ve run Arch and NixOS as my daily machine, and today it’s Arch on both desktops and NixOS on the servers. Arch’s reputation for breaking is mostly a missing snapshot. With btrfs, Snapper, dotfiles in git and a package list, a rollback takes 2 minutes and the fear of updates goes away. NixOS earns its learning curve when several machines have to stay identical, and on a server I wouldn’t run anything else.\nThe question I get most from people who already run Linux is which distro to put on the machine they work on. Arch or NixOS. They want a workstation they understand from top to bottom, and they’ve had enough of one that updates itself on someone else’s schedule.\nI’ve run both as my real machine. Not in a VM, not for a weekend. As the laptop I do DevOps and platform engineering work on, and as the desktop I do everything else on. So you may as well know where I landed before you read on, because you’d have worked it out by the second table anyway. Arch on both desktops, NixOS on the servers. I’ve got reasons, and they’re more specific than “Arch is nicer”, so I’ll walk through them. The reasons are the useful part.\nBoth are good. Both have trade-offs I’ve felt on my own machine. Which one fits depends on your life more than on the software, and my life changed. I have kids now, so the Saturday afternoons I used to spend on a broken Xorg config don’t exist anymore. The machine has to work every time I open the lid. What surprised me is that Arch’s reputation for instability turned out to be mostly a skill issue, and that changed the whole comparison.\nThe one difference everything else hangs off Arch is a blank canvas. You build the system piece by piece and you own every piece. It’s mutable, so you edit files, install packages, and the system drifts and grows over the years. Rolling release means you’re always on the current version of everything.\nNixOS is declarative. You describe what the system should be in config files, and Nix builds exactly that. It’s mostly immutable. You don’t poke at the running system, you change the config and rebuild, and every change is reproducible.\nThat difference changes how you think about the machine. On Arch the system is something you tend. On NixOS it’s the output of a file you can read from top to bottom.\nWhat I’m comparing on For a working engineer with not much free time, four things decide this. Does an update land without breaking something. How fast do you get the package you need. How steep is the learning curve. And what it’s like to live in, day to day. I’ll split it into work and personal use, because the same trade-off comes out with the opposite sign depending on which machine you’re sitting at.\nArch at work Standard Linux, first of all. Everything works the way the documentation says and your muscle memory from servers carries straight over. The packages are current. Need the latest kubectl, terraform or Go? It’s in the repos or in the AUR, usually within days of the release. And the AUR covers nearly everything else. yay -S slack-desktop spotify and you’re done. The Arch Wiki is the best Linux documentation I know, for any distro. And there are no abstraction layers. What you install is what runs.\nThen the other side. Rolling release means things break now and then. A kernel update knocks out your WiFi driver. A library bump breaks one app. It happens. Nothing is reproducible, so “works on my machine” is a real sentence and your colleague can’t easily replicate your setup. It drifts. After a year you’ve installed so much that you’ve forgotten what’s there or why. And there’s no rollback unless you set one up yourself, so an update that breaks something gets fixed by hand.\nI ran Arch for work for years. Early on I lost time to breakage every few months. Then I worked out that most “Arch is unstable” complaints, mine included, came from running it without a safety net. With the right setup it has held up for me since. The next section is that setup.\nMaking Arch survive an update Since the kids arrived I have zero tolerance for a broken laptop during work hours. My answer was to treat Arch as semi-ephemeral. The machine is allowed to die at any moment, and I should be productive again within minutes. It’s the same resilience thinking I apply to infrastructure, pointed at my own laptop. I stopped trying to prevent failure and made recovery cheap instead.\nThe stack:\nBtrfs with Snapper. A snapshot before every pacman transaction. Something breaks? snapper rollback and reboot. Done. Timeshift with rsync if you’re not on btrfs. Similar protection, less elegant. Dotfiles in git. My whole config lives in a chezmoi repo. Fresh install? chezmoi init --apply and my environment is back. A package list in the same repo. pacman -Qqe \u003e pkglist.txt going out, pacman -S --needed - \u003c pkglist.txt coming back. A separate /home. It survives a reinstall, so my data does too. Full backups. Borg to my NAS, daily. Worst case, I restore everything. That gives me three recovery options:\nMinor issue. Snapper rollback, 2 minutes. Major issue. Fresh install, then dotfiles, then the package list, 30 minutes. Dead hardware. New machine plus a backup restore, 1 hour. I stopped fighting the rolling release model and leaned into it. Updates might break things, so I made breaking things cheap. Once a rollback takes 2 minutes, an update stops being scary. I update weekly now. If something breaks I roll back and wait for the fix. No debugging session, no awkward message to a client about why I’m late.\nNixOS at work Your configuration.nix describes the whole system. Clone the repo, run nixos-rebuild, get the same machine. Every generation is kept, so a bad update means booting the previous one from the boot menu. Two seconds. Installed packages are in the config, so there’s no drift and you always know what’s on the box. nix-shell and devenv give every project its own dependencies without touching the system. And experimenting costs nothing, because whatever you break, you roll back.\nThe price is the learning curve. The Nix language is its own thing, the documentation is famously thin, and the first weeks are rough. Nixpkgs is huge but more conservative than Arch, so you can wait weeks for the latest version of something. The mental model is different too. You can’t apt install and move on, you have to think declaratively. All those generations take disk space, and /nix/store gets big. And some things are plain harder. Proprietary software, anything with a non-standard layout, anything that assumes FHS, all of it needs extra work.\nI switched to NixOS for work after one Arch breakage too many. The first month hurt. Learning Nix, rewriting my config, fighting odd package issues. After that it held. Updates never broke anything, and I could run nixos-rebuild switch on a Friday afternoon without thinking about it. When I got a new laptop, my complete environment was running in under an hour.\nWhat I’d say for work Where a broken machine costs real money, NixOS’s reliability is hard to argue with. The catch is the learning time. If you need to be productive tomorrow and can’t spend a week on Nix, Arch gets you working sooner.\nSo the question is what downtime costs you. If a broken system means a missed client meeting or a late deploy, the Nix learning curve pays for itself. If you can absorb the occasional troubleshooting hour, Arch’s familiarity wins. And here’s what complicates the clean answer. Once I had the snapshot and backup discipline above, my Arch downtime dropped to about what NixOS gave me, without the week of learning Nix. That’s what tipped me back.\nArch at home Here the trade-off comes out the other way.\nOn a personal machine, tinkering is the point. Arch makes you understand your system because you built it. Gaming wants the newest Mesa drivers and creative work wants the newest GIMP, and Arch has them first. The AUR has the obscure Japanese rhythm game and the weird hardware tool, because Arch users package everything. The forums are active and helpful, once you’ve learned to read the wiki first. And it’s fun. There’s real satisfaction in a system built exactly how you want it.\nThe cons are the same as at work. They sting less when it’s your own machine and nobody’s waiting on you.\nNixOS at home Home-manager gives you declarative dotfiles, so your whole user environment sits in config files. Flakes make those configs reproducible and composable. Your NixOS config can be a public repo that other people learn from and send patches to. And if you’ve got a laptop and a desktop, one config keeps them identical.\nWhat you give up is the tinkering. Everything goes through Nix. You can’t grab a binary and run it, which is great for reliability and rough on the “let me just try this” itch. Gaming is more work. Proton and Steam run, but Lutris and the various launchers all want Nix wrappers. And there’s the rabbit hole. You’ll spend more evenings perfecting configuration.nix than using the computer.\nWhat I’d say for home For personal use I prefer Arch, and here I don’t have to argue with myself. The tinkering is the hobby. The occasional breakage is part of the fun. For gaming and random software, Arch’s flexibility beats Nix’s purity.\nIf you run the same config across several personal machines, or want your personal and work setups identical, the declarative model is strong enough to change that. For me, with one desktop, it doesn’t.\nWhat I run today The work laptop is Arch with btrfs and Snapper. With the snapshot discipline it has been solid, I get current tools without waiting on nixpkgs, and recovery from anything is minutes away. The personal desktop is also Arch, same setup. Gaming works better than it did on NixOS, and the AUR has everything I want. The homelab servers are NixOS. For a server, declarative config wins outright. I rebuild any of them from git, and nothing drifts because nothing can.\nSo the pattern is Arch for the machines I sit at, NixOS for the ones I don’t. On a desktop I want the familiar Linux, the package the day it’s released, and the freedom to download something and run it. Snapshots and backups take care of the instability. On a server I want reproducibility and infrastructure as code, and a server doesn’t need the newest packages or the gaming support that make Arch worth it on a desktop.\nSide by side Package availability Category Arch NixOS Official repos Excellent Excellent Bleeding edge Same day Days to weeks Proprietary AUR (easy) Nixpkgs (harder) Obscure software AUR (vast) Hit or miss Custom packages PKGBUILD (simple) Nix derivation (complex) System administration Task Arch NixOS Install package pacman -S pkg Edit config, rebuild Update system pacman -Syu nixos-rebuild switch --upgrade Rollback Manual or btrfs snapshots Built-in, boot menu Clean old packages pacman -Sc nix-collect-garbage View installed pacman -Q nixos-option or read config Development workflows Workflow Arch NixOS Project dependencies Manual or asdf/mise nix-shell, devenv, flakes Conflicting versions Containers or manual Native isolation per-project Sharing environments Dockerfiles flake.nix or shell.nix Reproducing on CI Dockerfile Same flake as local Who should pick what Pick Arch if:\nYou want to learn Linux deeply You value simplicity and transparency You don’t mind occasional troubleshooting Gaming is important You prefer imperative “just do it” workflows Pick NixOS if:\nReliability is critical (work machine) You manage multiple machines You want reproducible environments You like declarative configuration You’re willing to invest learning time upfront Pick both if:\nYou have multiple machines with different purposes You want stability for work and freedom for personal Where I landed There’s no universal answer, and I’ve been happy on both for different reasons.\nArch taught me how Linux works, because I had to set up every component and every config file myself. NixOS taught me to think declaratively, with every change in a file and every experiment reversible. Both lessons stuck.\nThe one that changed my workstation choice is simpler. Arch’s instability is solvable. The people who say Arch breaks constantly are usually running it without snapshots and without a recovery plan, and I did that for years. Give it that safety net and it’s as reliable as anything I’ve run, with current packages and familiar tooling on top.\nHaving kids gave me less time to tinker. I expected that to push me onto NixOS for everything. Instead it pushed me to make recovery cheap. Snapshots, dotfiles in git, a package list and backups are what made the rolling-release fear go away.\nIf you’re eyeing Arch but worried about stability, build the recovery setup first. Btrfs and Snapper take an afternoon, and mine has paid for itself several times over. Then update. And for servers I’d still pick NixOS every time, because a server should be rebuildable from git and from nothing else.","permalink":"https://tommeurs.nl/en/posts/arch-vs-nixos-workstation/","title":"Arch vs NixOS as a Workstation: Professional and Personal Use"},{"content":" TL;DR I spend my evenings on Hack The Box and it’s made me better at my day job, which is platform engineering and has nothing to do with pentesting. The first ten minutes on a CTF box and the first ten minutes on a broken production server are the same loop, with “flag” swapped for “incident”. Below is the toolkit that loop runs on, from one host up to a Kubernetes pod, with the CTF use and the work use side by side for every command.\nA production server is misbehaving. You SSH in. Now what?\nThe people who stay calm here already know the next ten commands, because they’ve run this exact loop a hundred times before. On their own time, with the word “flag” where “incident” is now.\nI spend my evenings on Hack The Box and the odd CTF competition. I’ve no plans to become a pentester. I like platform engineering and I’m staying in it. I keep at the challenges because the skills carry straight into the day job, and the carryover is bigger than I expected when I started.\nPwning a web app looks nothing like running a Kubernetes cluster. The first ten minutes are identical, though. You’re inside a system you don’t fully understand and you have to work out what it’s doing, fast. That’s understanding over trust, the thing I care about most when something breaks, and a CTF lets you practice it while the only thing at stake is your evening.\nHere’s what carries over, starting on a single host and ending inside a pod.\nThe same four questions Drop into a CTF box and the first questions are always the same:\nWhat’s running? What’s vulnerable? What can I access? How do things connect? Swap “vulnerable” for “broken” and that’s incident response. Same questions when you SSH into a server that’s misbehaving or a pod that keeps crashing, and the same questions when you’re working out what an attacker touched.\nThe difference is reps. A CTF player does this every week, so the enumeration stops being a checklist and becomes a reflex. Once it’s a reflex you stop guessing under pressure and start gathering. The rest of this post is the toolkit that reflex runs on, from the filesystem up to the container.\nFilesystem Recently modified files On a CTF box this finds the planted backdoor or the config an attacker edited. At work it finds what changed right before the system broke, and most of the time that’s plain config drift.\n# Files modified in the last 24 hours find / -type f -mtime -1 2\u003e/dev/null # Files modified in the last 10 minutes find / -type f -mmin -10 2\u003e/dev/null # Files changed after a specific date find / -type f -newermt \"2026-02-01\" 2\u003e/dev/null # Find files by size (useful for finding log bombs or large dumps) find / -type f -size +100M 2\u003e/dev/null A deployment fails and the app won’t start. Which config changed? find /etc -type f -mmin -30 shows someone edited the wrong file 15 minutes ago.\nFiles by content On a box you’re grepping for passwords, keys and flags hidden in files. At work you’re grepping for where a config value gets set, or for the hardcoded credential that shouldn’t be there.\n# Find files containing a string grep -r \"password\" /etc 2\u003e/dev/null grep -r \"API_KEY\" /app 2\u003e/dev/null # Find files containing a pattern (like IP addresses) grep -rE \"\\b([0-9]{1,3}\\.){3}[0-9]{1,3}\\b\" /etc 2\u003e/dev/null # Find files with specific permissions (world-writable) find / -perm -002 -type f 2\u003e/dev/null # Find SUID binaries (privilege escalation, but also config issues) find / -perm -4000 -type f 2\u003e/dev/null “Where is that environment variable being set?” grep -r \"DATABASE_URL\" /etc /app ~/.* 2\u003e/dev/null finds it in three places, with three different values.\nFile metadata In a CTF the metadata is where you find the hidden file, and the one somebody tampered with. At work the same commands explain the permission problem you’d otherwise sit staring at.\n# Extended attributes lsattr /etc/passwd getfacl /var/log/secure # File type and magic bytes file /usr/bin/mystery-binary file /var/log/weird.log # Detailed stat info stat /etc/shadow # Immutable files (can't be modified even by root) lsattr -a /etc/ | grep \"\\-i\\-\" “Why can’t I delete this file, even as root?” lsattr shows the immutable flag is set.\nProcesses What’s running and why On a box you’re looking for the malicious process, the reverse shell, the thing eating the CPU. At work you’re looking for the runaway process, and for what the application is really doing.\n# Detailed process tree ps auxf pstree -p # What's using the most CPU/memory right now top -bn1 | head -20 ps aux --sort=-%mem | head -10 ps aux --sort=-%cpu | head -10 # Process by name with full command line ps aux | grep -E \"[n]ginx|[a]pache\" # What process is using a specific port lsof -i :8080 ss -tlnp | grep 8080 netstat -tlnp | grep 8080 # What files does a process have open lsof -p \u003cpid\u003e ls -la /proc/\u003cpid\u003e/fd/ # Process environment variables cat /proc/\u003cpid\u003e/environ | tr '\\0' '\\n' # Process working directory ls -la /proc/\u003cpid\u003e/cwd A Java app is eating 100% CPU. ps aux --sort=-%cpu gives you the PID. /proc/\u003cpid\u003e/fd/ shows 50,000 open file handles, so it’s a connection leak.\nTracing what a process does In a CTF you trace a binary to learn what it does and where the hidden functionality is. At work you trace the app that says it can’t connect, to see which syscall is failing and on what.\n# System calls made by a process strace -p \u003cpid\u003e strace -f -e trace=network \u003ccommand\u003e # only network calls strace -f -e trace=file \u003ccommand\u003e # only file operations # Library calls ltrace -p \u003cpid\u003e # Trace a specific command strace -o output.txt -f ./mystery-app # See what a process is doing in real-time watch -n 1 \"cat /proc/\u003cpid\u003e/io\" The app says “connection refused” but the service is up. strace -e connect shows it’s connecting to the wrong IP, because an environment variable is set wrong.\nNetwork Connections On a box you’re hunting C2 connections and exfiltration channels, and looking for somewhere to pivot to. At work you’re debugging why two services can’t reach each other, and now and then finding a connection that shouldn’t be there at all.\n# All network connections ss -tunapl netstat -tunapl # Established connections only ss -tnp state established # Connections by process ss -tnp | grep \u003cpid\u003e # Listening services ss -tlnp netstat -tlnp # Connections to a specific host ss -tn dst 10.0.0.50 # DNS resolution trace dig +trace example.com host -v example.com nslookup -debug example.com The application times out on the database. ss -tn shows the connection stuck in SYN_SENT, which points at a firewall or a wrong route.\nPacket capture In a CTF you capture traffic for credentials and for data hidden in a protocol. At work you capture it to see what’s on the wire when the application’s own logs don’t say, and to check that TLS is doing what you think it’s doing.\n# Capture traffic on an interface tcpdump -i eth0 -w capture.pcap # Capture specific traffic tcpdump -i eth0 port 443 tcpdump -i eth0 host 10.0.0.50 tcpdump -i eth0 'port 80 and host 10.0.0.50' # Read capture with full packet content tcpdump -r capture.pcap -A tcpdump -r capture.pcap -X # Quick HTTP debugging tcpdump -i eth0 -A port 80 | grep -E \"GET|POST|Host:\" # Check TLS handshake openssl s_client -connect example.com:443 -servername example.com HTTPS works from one server and fails from another. openssl s_client shows the failing server doesn’t trust the CA, because the root cert is missing.\nLogs Finding the pattern Attack traces, and the one anomaly in a week of noise. That’s the CTF side. At work it’s the same work with a different question, which is why did this break and when did it start.\n# Follow logs in real-time tail -f /var/log/syslog journalctl -f -u myservice # Logs since specific time journalctl --since \"2026-02-01 10:00:00\" journalctl --since \"1 hour ago\" # Logs for specific service journalctl -u nginx -n 100 # Filter by priority journalctl -p err # errors only journalctl -p warning..err # warnings and errors # Count occurrences of patterns grep -c \"error\" /var/log/app.log awk '/error/ {count++} END {print count}' /var/log/app.log # Extract timestamps and group by minute awk '{print $1, $2, $3}' /var/log/app.log | cut -d: -f1,2 | uniq -c | sort -rn # Find errors with context grep -B5 -A5 \"FATAL\" /var/log/app.log Intermittent 500 errors. grep -c \"500\" access.log per hour shows a spike at :00 every hour, and that’s a cron job.\nCorrelating across services On a box you’re rebuilding the attacker’s timeline and following their lateral movement. At work you’re rebuilding the incident timeline across services, and cause and effect rarely sit in the same log file.\n# Auth failures grep \"Failed password\" /var/log/auth.log | awk '{print $1,$2,$3,$9,$11}' | sort | uniq -c # Commands run by users (if auditd is enabled) ausearch -k commands -i # Correlate by timestamp across files grep \"2026-02-27T10:15\" /var/log/*.log # Parse structured logs (JSON) cat app.log | jq 'select(.level == \"ERROR\")' cat app.log | jq -r '[.timestamp, .message] | @tsv' Users and authentication In a CTF the compromised account is the way in, so you map who can do what. At work that same map is the access audit, the permission bug and the security review.\n# Users with login shells cat /etc/passwd | grep -v nologin | grep -v /bin/false # Users with UID 0 (root equivalent) awk -F: '$3 == 0 {print $1}' /etc/passwd # Last logins last -n 20 lastlog # Failed login attempts lastb -n 20 faillog -a # Who is logged in now w who # User's groups and sudo rights id username sudo -l -U username # SSH authorized keys cat ~/.ssh/authorized_keys find / -name \"authorized_keys\" 2\u003e/dev/null You suspect someone got in who shouldn’t have. last shows a login from an IP you don’t know. grep \"Accepted\" /var/log/auth.log shows they came in with an SSH key rather than a password.\nDisk On a box, disk forensics is recovering deleted files and reading the layout. At work it’s the disk that’s full when du says it isn’t.\n# Disk usage by directory du -sh /* 2\u003e/dev/null | sort -h du -sh /var/* 2\u003e/dev/null | sort -h # Find what's using space ncdu / # interactive # Find large files find / -type f -size +1G 2\u003e/dev/null # Deleted files still open (holding disk space) lsof +L1 # Inode usage df -i # What's mounted mount | column -t cat /proc/mounts findmnt Disk 100% full and du says 50% used. lsof +L1 shows deleted log files still held open by a process, and restarting that process gives the space back.\nContainers The CTF version is the container escape, and the misconfiguration that makes it possible. The work version is the container that won’t start, and knowing what state it’s in.\n# What's in a running container docker exec -it \u003ccontainer\u003e /bin/sh kubectl exec -it \u003cpod\u003e -- /bin/sh # Container processes from host docker top \u003ccontainer\u003e ps aux | grep containerd # Container resource usage docker stats kubectl top pods # Container logs docker logs -f \u003ccontainer\u003e kubectl logs -f \u003cpod\u003e # Inspect container config docker inspect \u003ccontainer\u003e docker inspect \u003ccontainer\u003e | jq '.[0].Config.Env' kubectl describe pod \u003cpod\u003e # Container filesystem docker diff \u003ccontainer\u003e # what changed docker export \u003ccontainer\u003e | tar -tf - # list files # Check capabilities docker inspect \u003ccontainer\u003e | jq '.[0].HostConfig.CapAdd' kubectl auth can-i --list A container crashes on startup and docker logs shows nothing. docker inspect shows the wrong entrypoint, set by an image override.\nWhat shows up at work After months of evenings on this, the habits turn up at work without me deciding to use them. I enumerate before I touch anything, because fixing the wrong thing under pressure is how a single outage turns into two. I read the error message instead of retrying and hoping, and the answer is usually in it.\nI trace a problem from the application down to the syscall and out to the network, the same path I’d trace from a web app to root on a box. I notice what’s misconfigured before it bites, because a CTF trains you to look for exactly that. And grep, find, awk and sed have stopped being things I look up.\nAll of that is reps, and a CTF is the cheapest place I know to get them without taking a security job.\nWhere to start If you want the same toolkit, start in the shallow end:\nOverTheWire Bandit is pure Linux command line, zero setup. Do this first. TryHackMe has guided rooms with hints, a gentler on-ramp. Hack The Box after that. Start with the easy boxes and stick to the Linux machines. PicoCTF has the good forensics challenges, for when you want to go deeper. An evening a week is enough. I’ve been at it for months and I’m still not a pentester. I do know the next ten commands when I SSH into a broken server.","permalink":"https://tommeurs.nl/en/posts/ctf-skills-for-devops/","title":"CTF and Forensics Skills That Make You a Better DevOps Engineer"},{"content":" TL;DR Simulation models a thing without running it. Emulation runs it by rebuilding the hardware in software. Virtualization runs several complete operating systems on one machine. Containers run several apps on one kernel. Below, one restaurant per term, a table, and the four mix-ups I hear most.\n“So what’s the difference between a container and a virtual machine?”\nI get this question a lot, usually from someone smart who’s been nodding along in meetings and doesn’t quite want to admit they’re fuzzy on it. The honest answer is that most explanations make it worse. They open with “hypervisor”, “kernel sharing” and “hardware abstraction”, and now the person has four new terms to be confused about instead of one.\nBehind the container-versus-VM question hides a bigger one. What’s the difference between simulation, emulation, virtualization and containerization? People mash the four together all the time, and they’re four different answers to the same problem, which is running something in an environment it was never built for.\nI care about the distinction because I care about understanding what I run. If you can’t say which of the four a tool is doing, you can’t reason about its performance, its isolation, or what breaks when it falls over. So here’s the version that finally made it stick for me. It involves restaurants.\nThe restaurant metaphor Say you want to eat Japanese food. You’ve got four options.\nSimulation is the cooking show You watch a Japanese cooking show. The chef makes sushi, explains the technique, shows the fish being cut and the rolls being assembled. You see everything that would happen if you were eating at a Japanese restaurant.\nYou’re not eating anything, though. There’s no food. What you have is a model of the experience. You know how sushi is made and you’re still hungry.\nThat’s simulation. You build a model of something to study how it behaves, and you never run the real thing.\nA flight simulator models how an airplane behaves. Great for training pilots, and there’s no airplane anywhere in the loop. A circuit simulator does the same for electronics. It predicts how the components would behave, with nothing on the bench. What you want from a simulation is the prediction.\nEmulation is the Japanese restaurant in Amsterdam You go to a Japanese restaurant in Amsterdam. The Dutch chef learned the techniques in Japan, imports the ingredients, has rebuilt the kitchen equipment and plates on Japanese ceramics.\nEverything is set up to behave exactly like a restaurant in Tokyo. Done well, you can’t taste the difference. Same flavour, same plating.\nYou’re still in Amsterdam, though. One system is faithfully pretending to be another by copying every detail of how the original works.\nThat’s emulation. You make one system behave exactly like a different one by replicating its behaviour in software.\nRun a Game Boy emulator on your laptop. There’s no Game Boy hardware inside the laptop, so the emulator stands in for every Game Boy component, the CPU, the graphics chip, the sound chip. The games can’t tell. As far as they know they’re on a Game Boy, and they run for real, which is the difference from a model.\nVirtualization is the hotel with several restaurants You stay at a hotel with several restaurants under one roof. Japanese, Italian, French, Indian. Each runs on its own, with its own kitchen and its own chef.\nWhat they share is the building. The electricity, the water, the ventilation, the cleaning crew. Hotel management hands out the resources and decides how much space, power and staff each restaurant gets.\nEvery one of these is a complete, working restaurant. None of them is pretending to be anything. They’re splitting the building between them.\nThat’s virtualization. You run several complete operating systems on one set of hardware, with a thin layer underneath that hands out the resources.\nRun Windows and Linux at the same time on one physical server. Each gets its own full operating system, and they share the CPU, the memory and the storage. The hypervisor is hotel management here. It hands out the shares and keeps the two apart.\nContainers are the food court You go to a food court. One big kitchen in the back, and a row of vendors sharing it. Same ovens, same prep tables, same fridges.\nEach vendor still gets their own section, with their own ingredients and their own pans. The sushi vendor can’t reach into the pizza vendor’s mise en place. They’re walled off from each other while sharing the kitchen.\nThat’s cheaper than building every vendor a kitchen of their own. All a vendor needs is their own stuff, the ingredients and tools for their food.\nThat’s containerization. Applications run in isolated spaces on top of one shared operating system kernel, each carrying its own dependencies and libraries.\nDocker containers on a Linux server, say. Each container has its own filesystem, its own installed packages, its own app. They all share the one Linux kernel. The container runtime is food court management. It keeps every vendor inside their own section.\nThe four side by side Aspect Simulation Emulation Virtualization Containers Restaurant analogy Cooking show Foreign restaurant Hotel restaurants Food court kitchen What’s shared Nothing (model only) Nothing (full replica) Hardware only OS kernel + hardware What’s isolated N/A Everything Everything App + dependencies Performance Varies (often slow) Slow (translating everything) Near-native Native Compatibility High (models anything) High (replicas everything) Limited (same CPU arch) Limited (same OS kernel) Resource usage Low High Medium Low Which one, when Simulation fits when you need to understand or predict behaviour without running the real thing. It also fits when a real test is dangerous, expensive or impossible, as with nuclear reactors, space missions and epidemics, and when you’re designing something that doesn’t exist yet. It’s the wrong tool the moment you need actual results instead of modelled ones, or need to run real software. Weather models, flight simulators, circuit design tools, financial models and physics simulations all live here.\nEmulation fits when you need to run software from a different platform, when the original hardware is obsolete, rare or gone, and when exact behaviour matters more than speed. Skip it when performance matters, and skip it when virtualization would do, because virtualization is much faster. The usual cases are old console games on SNES and PlayStation emulators, ARM software on x86 through Apple’s Rosetta 2, the Android emulator for mobile development, and legacy software kept alive after its hardware died. Slow, yes. Without it the game doesn’t run at all.\nVirtualization fits when you need several different operating systems on one machine, when environments need full isolation from each other, when the guest OS differs from the host, Windows on Linux for instance, or when each environment needs its own kernel version. Skip it when every workload runs the same OS, since containers are more efficient there. Skip it too when you need every last bit of performance, or when resources are very tight. Windows and Linux on one server, development environments that match production, testing across OS versions, multi-tenant hosting with full isolation.\nContainers fit when every workload can share one kernel, when you want isolation that’s light and fast, when you’re deploying many similar applications, and when resource efficiency and startup time matter. Skip them when you need a different OS, because Windows containers don’t run on a Linux kernel. Skip them when you need the strongest isolation you can get, since a VM isolates harder, and when your software needs kernel modifications. Microservices, CI/CD pipelines, local development environments, Kubernetes workloads, scaling web applications.\nThey stack In practice these get combined.\nTechnology Layers\nYour Application\n(Container)\nContainer Runtime\n(Docker/containerd)\nGuest OS - Linux\n(Virtual Machine)\nHypervisor\n(Virtualization)\nHost OS\nPhysical Hardware\nContainers inside virtual machines inside physical servers. Every layer costs you some overhead and buys you back capability and isolation, and it’s worth knowing which trade you’re making on each floor.\nCloud providers stack it exactly like this. Your “server” is a VM on their hardware, and your containers run inside that VM. That stacking is why I want the four distinctions clear in my head. When latency spikes, I want to know which floor it came from.\nWhat people get wrong “Docker is a virtual machine” No. Docker containers share the host’s kernel and VMs bring their own. That one difference is why a container starts in seconds, with no kernel to boot, while a VM takes minutes to come up.\n“Emulation is just slow virtualization” No, they solve different problems. Emulation translates one architecture into another, Game Boy into PC. Virtualization runs the same architecture more efficiently, x86 on x86.\nA PC virtualizes Linux on Linux very efficiently. It has to emulate a Game Boy, because there’s no Game Boy hardware in it to virtualize.\n“Containers are secure isolation” Partly. Containers give you process isolation, but they share a kernel, and a kernel vulnerability can become a container escape. VMs isolate harder because they don’t share that kernel.\nFor high-security or untrusted workloads I still reach for a VM. For most of what I run, containers are fine, and I make that call on purpose, every time.\n“Simulation and emulation are the same thing” No. Simulation models behaviour approximately. Emulation replicates it exactly.\nA flight simulator simulates airplane physics to predict and to teach. A Game Boy emulator copies the exact hardware behaviour, and the games on it run for real.\nWhy I bother Each of the four earns its place. Simulation when you want to model something without running it. Emulation when you need foreign software on hardware that was never meant for it. Virtualization when you need different operating systems side by side. Containers when you want a lot of similar apps without paying for a full OS each time.\nI keep them straight because which one I’m running tells me what to expect when it breaks, and on hardware I own there’s nobody else to ask.\nNext time someone asks me the container question, I’m starting with the food court and the hotel, and leaving the word hypervisor out of it.","permalink":"https://tommeurs.nl/en/posts/simulation-emulation-virtualization-containers/","title":"Simulation, Emulation, Virtualization, and Containers: The Restaurant Metaphor"},{"content":" TL;DR The old model is a castle. Get past the wall and you’re trusted everywhere inside. Zero trust is a hotel. Your key card opens your room, the gym if you booked it, and nothing else, and every door checks the card every time. Below, the three principles, a day in that hotel, and what the key card turns into once the doors are software.\n“So what is this zero trust thing everyone keeps talking about?”\nI get that question a lot. Usually from a manager or an executive who has to approve a security budget and doesn’t have a technical background to lean on. And most of the explanations I’ve seen are bad. They either drown you in jargon or sand the idea down until there’s nothing left to hold.\nSo here’s the version I use instead. I’ve tried it on my parents, on executives, and on the colleague who still calls the firewall “the internet box”. It works because it starts with a thing you’ve held in your own hand, a hotel key card. From there it goes all the way down to mTLS and identity-aware proxies, and the idea doesn’t change on the way.\nThe castle it replaces Before the hotel makes sense, you need the thing it replaces.\nTraditional network security is a medieval castle. You build a big wall (the firewall), dig a moat around it and put guards at the gate. Anyone outside the wall is untrusted. Anyone who makes it inside is trusted, and can go anywhere and do anything.\nThat was a fair design when:\nEveryone worked in the same building All the servers lived in the basement “Remote work” meant carrying a laptop home once a year Then people started working from coffee shops and applications moved to the cloud. Half the kingdom was outside the wall now, and the wall stopped meaning much.\nThe castle has one flaw that the rest never makes up for. Once someone is past the gate, they’re trusted. An attacker who compromises one laptop inside the network can walk around freely. Most breach write-ups I’ve read have that shape. Phishing gets someone in, then they move “laterally” through a network where everything trusts everything else. One phished laptop, and the blast radius is the whole castle.\nThe hotel Now the picture I use instead.\nPicture a modern hotel. You check in at the front desk and get a key card. That card:\nOpens your room Opens the gym (if your booking includes it) Opens the pool area (if your booking includes it) Opens the parking garage (if you registered a car) Does NOT open other guest rooms Does NOT open the staff areas Does NOT open the kitchen Every door decides for itself whether to let you in. The door doesn’t care that you’re “inside the hotel”. It wants to know whether this card may open this door, right now.\nThat’s zero trust in one sentence. Every door checks, every time, wherever you’re standing. The rest of this post is that sentence with more detail.\nWhy the name trips people up Quick detour, because the name does damage before the idea gets a chance.\n“Zero trust” sounds like you trust nothing and nobody, which would be paranoid and unworkable. “Verify everything” would’ve been the honest label. “Continuous verification” if you wanted the consultants to like it. We’re stuck with the buzzword, so we work with it.\nWhat it means in practice:\nDon’t trust based on location (being inside the network) Don’t trust based on history (you logged in once, so you’re trusted forever) Verify every access request, every time Grant the minimum permissions the job needs At the hotel you’re a verified guest with a specific list of doors, and you show the card at each of them.\nThree principles, one key card Zero trust has three load-bearing principles, and the hotel covers all of them.\nVerify explicitly Every door checks your card, every time. There’s no wristband that says “trusted guest” and waves you through the rest. The pool checks. Your room checks.\nIn software, that means every access request gets authenticated and authorized, no matter where it comes from. The request from the office network gets the same treatment as the one from the coffee shop.\nLeast privilege Your card opens what you need and nothing more. Booked a basic room? You get your room and the common areas. Booked the executive suite with gym access? You get those too. You never get other guest rooms, and you never get the staff corridor.\nIn software, people and services get the minimum permissions their role requires. An accountant doesn’t need the source code. A developer doesn’t need the HR records.\nAssume breach The hotel is designed on the assumption that someone will eventually hold a card they shouldn’t. That’s why every door checks on its own. A stolen card opens what that card was allowed to open, and the rest of the hotel stays shut.\nIn software, you design as if the attacker is already inside. Limit the blast radius, segment the network, watch what talks to what. This is the principle I care about most, because it matches how I think about every system I run. Failure is the default I plan for. Something in the setup is going to get compromised eventually, and I want to know beforehand where that stops.\nA day in the hotel The three principles are the theory. Watching them run for a day is where it clicks.\nMorning. You head to the gym and tap your card on the door. The door asks three things. Is this guest allowed in the gym, is the booking still valid, is the gym open right now? Yes to all three, so it opens.\nMidday. Back in your room, you want room service. You call, and they ask for your room number and a security code from your check-in receipt. Caller ID on its own isn’t enough for them.\nAfternoon. You try the printer in the business center. You authenticate again, card plus a PIN you set at check-in. For some resources, holding the card doesn’t cut it.\nEvening. Your booking ends at 6 PM, but your flight is at 8 PM. You ask for a late checkout and the front desk extends your access to 7 PM. At 7:01 PM your card stops working everywhere. Room, gym, all of it.\nOne day, and the whole model has been on show. Continuous verification, access with an end time, a stronger check for the sensitive stuff, and revocation the moment the access expires. Nobody had to go around collecting cards.\nWhat it isn’t A few misconceptions, because they come up in almost every conversation I have about this.\n“So no more passwords?” You still authenticate. You do it more carefully, and often with more than one factor.\n“So a VPN everywhere?” A VPN is the castle gate. Get through it and you’re trusted inside. Zero trust often replaces the VPN altogether with a check per resource, and Google’s BeyondCorp is the canonical example of doing exactly that.\n“That’s for big companies.” The principles scale down. Your home network can run on them too.\n“Which product do I buy?” Vendors will happily sell you a “zero trust solution”, but there’s nothing to install. It’s an approach you build toward, one door at a time.\n“So we switch it on next quarter?” Moving to zero trust takes time. It’s a migration, and a slow one. There’s no switch.\nWhere the hotel stops and the software starts Here the metaphor runs out and the real systems take over.\nIn a real zero trust setup the doors are software. There’s no wall any more. Every request carries proof of who or what is making it, and every resource decides for itself whether that proof is good enough.\nThe key card becomes a cryptographic identity. For services that often means SPIFFE identities and short-lived certificates instead of a long-lived shared secret. The door check becomes mTLS, where both sides prove who they are before a single byte of application data moves. “Every door checks on its own” becomes microsegmentation. In Kubernetes that’s a NetworkPolicy per workload, enforced by a CNI like Cilium, so a compromised pod can’t reach anything it wasn’t explicitly allowed to reach. The late checkout that expires at 7:01 PM becomes short-lived credentials from something like Vault, so access ends on a timer instead of lingering until someone remembers to revoke it. None of that changes the idea. It’s the same hotel with cryptography where the plastic was, and each bullet is one door.\nThis is also why zero trust sits so well with how I think about sovereignty. Once you stop trusting the network and start verifying every request, you have to know who talks to what, and why. There’s no vague “inside” left to wave through. Every connection is something you can inspect and something you can revoke, and why I won’t run what I can’t inspect is in Sovereign Infrastructure.\nThe version for the meeting When someone asks you what zero trust is, try this:\n“Traditional security is like a castle. Once you’re inside the walls, you’re trusted everywhere. Zero trust is like a hotel. You have a key card, but every door still checks whether you’re allowed through. Being inside doesn’t mean you have access to everything.”\nThat’s the whole thing. Underneath sit identity providers, microsegmentation, mTLS and monitoring, but they’re all implementing that one sentence.\nMy parents got the hotel and none of the bullets. Everything past the hotel is for whoever comes back with a second question.","permalink":"https://tommeurs.nl/en/posts/zero-trust-explained/","title":"Zero Trust Explained: The Hotel Key Card Metaphor"},{"content":" TL;DR I spent years polishing a vim and tmux setup, then sat useless in front of a stock vim on a production server. So I learned the defaults I’d been overriding all along, and moved my customizations on top of them instead of in place of them. What that looks like for vim, tmux and the shell tools, and the four checks I use to find the gaps.\nFor years my setup was the thing I was proudest of. A dotfiles repo with commits going back further than some of my jobs, a vim config that pulled in thirty plugins, the tmux prefix moved somewhere comfortable, a prompt that took a weekend to get right. Down to the last keybinding, it was mine. On my laptop I was fast, and I hardly thought about the tools at all. Muscle memory did the work.\nThen one evening I SSH’d into a production server to debug something, and I was useless.\nNo mappings, no plugins, no aliases. Just vim as it ships, with the default keys I’d quietly forgotten during the years I spent overriding them. I sat there fumbling through basic navigation like it was my first week, while a real problem waited. Everything I’d polished was on a machine three hundred kilometres away, and none of it could help me.\nThe setup only worked where the setup was installed. It took a production server for me to notice.\nWhat I’d traded away There’s a version of me that learned vim the way it ships before ever touching a plugin. Same skills, same brain. He logs into that same server, opens the same file, and nothing feels foreign, because the keys are the ones that were there from the start.\nThat version can sit down at any machine with vim on it and work. A colleague’s laptop while pairing. A locked-down client device. A fresh VM that needs debugging right now. The tooling doesn’t have to come along, because the competence lives in his hands. There’s no repo to clone first.\nThat’s the portability over personalization idea, pointed at my own workflow. Owning your infrastructure means being able to operate it from whatever chair you happen to be in. The same holds for skills. A skill that only works on one machine is a dependency, and I don’t get to reproduce that machine on a client’s server.\nWhere the cost hides The rabbit hole is easy to fall into. You read about someone’s vim config on Reddit. You find a dotfiles repo with 3000 stars on GitHub. One more plugin and you’ll be so much faster. And you will be. On your machine. With your config.\nThe cost doesn’t show up on that machine. Every default you remap before learning it is a piece of portable knowledge swapped for a local convenience, and the swap is invisible right up to the moment you’re on someone else’s box with a deadline.\nTwo people can look identical while everything works. One knows the defaults and chose to override them. The other never learned them and only knows the custom setup. Drop both onto a stock system and the difference is immediate. I was the second one.\nThe defaults were better than I remembered So I went back and learned them. Slower than I’d have liked, on my own laptop, where nothing forced me to. And they have one property no plugin can match. They’re everywhere, and they’ve been everywhere for decades.\nVim: learn before you remap Vim’s default keys are weird. No argument from me. They’re also consistent, and the core of them goes back to vi in the 1970s, so every vim on every Unix box answers to the same ones. That consistency was what I’d been throwing away.\nBefore you remap jk to escape, learn Ctrl+[. It works in vim, in bash, in anything built on readline. Before you install a fuzzy finder, learn :find and :edit **/*. Before you add custom text objects, get the built-in ones into your fingers. You can still remap everything later. You’ll just know what you’re giving up.\nDefault What it does Why it matters Ctrl+[ Escape Works in every terminal, not just vim Ctrl+o / Ctrl+i Jump list navigation No plugin needed :!command Run shell command Available on any vim * / # Search word under cursor Instant, no config % Jump to matching bracket Built-in since forever gf Go to file under cursor Works out of the box Ctrl+a / Ctrl+x Increment/decrement numbers Underrated default I still run plugins and I still have a few mappings of my own. But I can sit down at any vim and be useful right away, because I know what it hands me for free.\nTmux: the prefix matters less than you think The first thing everyone changes in tmux is the prefix, Ctrl+b to Ctrl+a. I did it too. Ctrl+b is awkward, and screen used Ctrl+a, so it felt like the obvious fix.\nBut look at what a prefix is. You press it once, then another key. The awkwardness of Ctrl+b is one keystroke, and after a few thousand of them your fingers stop noticing. Ctrl+a, meanwhile, collides with “go to beginning of line” in bash, which I reach for constantly. And every server with tmux on it uses Ctrl+b. Train on Ctrl+a and you fight the wrong prefix every time you leave home.\nDefault What it does Ctrl+b % Vertical split Ctrl+b \" Horizontal split Ctrl+b o Cycle through panes Ctrl+b z Zoom pane Ctrl+b [ Copy mode Ctrl+b d Detach Ctrl+b ? Show all keybindings That last one, Ctrl+b ?, is the lifeline on any tmux. You don’t have to memorize the bindings when the whole list is one keystroke away.\nShell tools: knowledge that travels Same logic for grep, awk, sed, find and the rest of the Unix toolkit. I watch people install ripgrep and forget grep exists. They pick up fd and can’t remember find syntax any more. Everything gets aliased to the modern replacement until the original is a stranger.\nThe modern tools are good and I use them every day. But grep is on every Unix system. So is find. Awk and sed come preinstalled on every server you’ll ever touch. In the middle of an incident you don’t get to install ripgrep first. You work with what’s there.\n# grep basics that work everywhere grep -r \"pattern\" . # recursive search grep -l \"pattern\" *.log # files containing pattern grep -v \"exclude\" # inverse match grep -E \"regex|pattern\" # extended regex # find basics find . -name \"*.py\" # find by name find . -mtime -1 # modified in last day find . -type f -exec cmd {} \\; # execute on results # awk one-liners awk '{print $2}' file # print second column awk -F: '{print $1}' /etc/passwd # custom delimiter awk 'NR==5' file # print line 5 Ripgrep is faster and fd reads nicer. Grep and find are there. On a server you don’t control, the one that’s installed is the one you get.\nWhy I put it off for so long So why did it take a production server to get me to do this? Because relearning the defaults felt like going slower on purpose. I had a config that worked. Sitting down to learn vim the boring way, on my own machine, for a payoff that only shows up on someone else’s, was a hard sell to a brain that wants results now.\nThere was sunk cost in it too. Real hours went into those dotfiles, and admitting that some of them had bought me a fragile local optimum stung a little. The repo has commits going back years and I’m fond of every one of them.\nAnd most days I’m on my own machine. The whole point of a comfortable setup is living in it. The bill for portability only comes due now and then, which made it easy to keep not paying it.\nAdditions on top, nothing replaced The setup stayed. What changed is the rule for what goes into it.\nAdditions are fine, replacements aren’t. My customizations sit on top of the defaults instead of erasing them. I don’t remap Ctrl+b. I add bindings, and the originals still fire. Colour schemes, a tidy prompt and a few mappings that don’t clash with anything built in all stay. The one question before overriding a default is whether it leaves me stranded on a fresh system. If yes, it doesn’t go in. I wrote up how I manage the repo with chezmoi and mise, and even there the config adds and never replaces.\nThe gaps hide until you go looking, so I test for them. Four checks:\nSSH into a server and edit a config file in vim, no plugins and no vimrc. Can you navigate, search, replace and save? Start a stock tmux session. Splits, moving between panes, new windows, all on the default bindings. Debug something with only grep and find. No ripgrep, no fd, no aliases. Can you find what you need? Write an awk one-liner that pulls a column out of a log file, without googling it. Wherever you struggle, that’s a default worth learning, and it’s a lot cheaper to find out now than during an incident.\nThree exceptions, and they’re real ones. If you have RSI or another physical limitation, remap whatever you need. Your health outranks portability. If your whole team standardizes on one config, that config is your default now. And if you never leave your own machine, never pair and never SSH anywhere, customize away. I don’t have that luxury.\nThe dotfiles repo is still there and still growing. What changed is that I could lose it tomorrow and still do my job. The next time I land in a minimal container with nothing but stock vim and a deadline, I’ll be a bit slower than at home, and that’s all.","permalink":"https://tommeurs.nl/en/posts/learn-the-defaults/","title":"Learn the Defaults: Why Portability Beats Customization"},{"content":" TL;DR Six ways of syncing dotfiles later, the one I kept is chezmoi for the files and mise for the tool versions, with the mise config living inside the chezmoi repo. Templates handle the work laptop against the personal desktop, age handles the secrets, and a fresh machine is one command and about five minutes. The walk-through goes from one managed file to the bootstrap I run for real.\nMy .zshrc is older than some of my coworkers’ careers. Same for my .vimrc and my tmux config. They’ve grown into a setup that does exactly what I want, and I’d be lost without them. The catch is that I run more than one machine. A laptop, a desktop, now and then a throwaway VM for testing. For years, keeping those in sync was a pile of hacks I didn’t trust.\nWhat I want out of dotfiles is portability. Any machine I sit down at should feel like mine within minutes, and I want to know exactly what got written where. Finding a setup that delivered both took years of dead ends.\nEverything I tried first It started with the classic move, a bare git repo in my home directory. git init --bare ~/.dotfiles, a couple of aliases, done. It works, until the day you run git clean in the wrong place and wipe your own configs. Machine-specific settings? You’re on your own.\nThen I found GNU Stow. The idea is nice. You organise your dotfiles into directories and Stow symlinks them into place. I ran it for a long stretch. But Stow has no idea what a template is, and my work laptop and my personal desktop need different .gitconfig settings. So you end up with gitconfig-work and gitconfig-personal and a manual switch, and sooner or later you forget to flip it.\nAt some point I thought, I already run Ansible for my servers, why not for my dotfiles? So I built a playbook for my workstation. Roles, templates, handlers, the full circus. It was beautiful. It was also wildly over-engineered. Running ansible-playbook every time I touched my .zshrc felt absurd, and writing YAML to copy a single file is its own kind of pain.\nI tried a few more along the way. yadm is git-based and a step up from the bare repo, but the templating felt bolted on. dotbot has a clean YAML config and wasn’t flexible enough for what I wanted. home-manager on Nix is very powerful, and I wasn’t ready to go all-in on Nix for it.\nNone of them stuck. Each one solved the problem I’d come with and had a gap somewhere else.\nThen I landed on chezmoi and paired it with mise. That’s the setup I’ve kept. Below is how it’s built, starting with one managed file and ending with the bootstrap I run on a fresh machine.\nWhat a plain repo can’t do On paper, dotfile management is trivial. They’re just config files. The trouble starts the moment you have a second machine. My laptop wants different settings from my desktop. My .gitconfig carries tokens and my .ssh/config holds hostnames I don’t want sitting in a public repo. I need Go 1.21 on one box and 1.22 on another. And on a fresh machine I’d like to be productive in minutes rather than spend an afternoon copying files around.\nA plain git repo handles none of that. You need templating, secret handling and a repeatable bootstrap, and that’s the gap chezmoi fills.\nchezmoi, starting with one file Chezmoi is a dotfile manager that covers all of the above. The name is French for “at my place”, which fits, since it carries your home environment wherever you go.\nSource state and home directory Chezmoi keeps a “source state” in ~/.local/share/chezmoi/ and applies it to your home directory. The source state can hold:\nRegular files Templates, with per-machine variables Scripts, for things you can’t capture as a static file Encrypted files, for secrets You can read exactly what gets written where, and that’s the property I care about most. No black box deciding things on my behalf.\nInstall and add one file # Install brew install chezmoi # macOS sudo pacman -S chezmoi # Arch sudo apt install chezmoi # Debian/Ubuntu # Initialize chezmoi init # Add your first file chezmoi add ~/.zshrc That’s the whole minimum. Your .zshrc is now managed by chezmoi.\nThe daily loop # Add a file to chezmoi chezmoi add ~/.config/nvim/init.lua # Edit a file (opens in $EDITOR) chezmoi edit ~/.zshrc # See what would change chezmoi diff # Apply changes to home directory chezmoi apply # Push to git chezmoi cd git add -A \u0026\u0026 git commit -m \"Update zshrc\" \u0026\u0026 git push Templates for per-machine config This is what Stow couldn’t do. Go templates let you render different config per machine from a single source file.\nFirst, define your data in ~/.config/chezmoi/chezmoi.toml:\n[data] hostname = \"laptop\" work = true email = \"tom@example.com\" Then use it in a template. Adding a file with --template gives it a .tmpl suffix in the source state:\nchezmoi add --template ~/.gitconfig Now ~/.local/share/chezmoi/dot_gitconfig.tmpl:\n[user] name = Tom Meurs email = {{ .email }} {{ if .work -}} [url \"git@gitlab.work.com:\"] insteadOf = https://gitlab.work.com/ {{- end }} On my work laptop the GitLab URL rewrite gets rendered in. On my personal machines it stays out. One file, two outcomes, and no switch to forget.\nMachine detection You can also let chezmoi detect machine properties:\n# ~/.config/chezmoi/chezmoi.toml [data] hostname = {{ .chezmoi.hostname | quote }} os = {{ .chezmoi.os | quote }} {{ if eq .chezmoi.hostname \"work-laptop\" -}} work = true {{- else -}} work = false {{- end }} Or ask on first setup:\n# chezmoi.toml.tmpl [data] email = {{ promptString \"email\" | quote }} work = {{ promptBool \"work machine\" }} Secrets, encrypted with age Never commit secrets in plain text. Chezmoi integrates with age, so secrets can live in the same repo without leaking.\n# Generate age key age-keygen -o ~/.config/chezmoi/key.txt # Configure chezmoi to use it chezmoi edit-config encryption = \"age\" [age] identity = \"~/.config/chezmoi/key.txt\" recipient = \"age1xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\" Now encrypt the sensitive files:\nchezmoi add --encrypt ~/.ssh/config The file lands encrypted in your repo. Only a machine holding the age key can decrypt it, so a public dotfiles repo stays safe.\nScripts for what a file can’t do Sometimes a file isn’t the right tool and you need to run a command. Chezmoi supports scripts for that:\n# ~/.local/share/chezmoi/run_once_install-packages.sh #!/bin/bash # This runs once (tracked by hash) if command -v brew \u0026\u003e /dev/null; then brew install ripgrep fd bat fzf elif command -v apt \u0026\u003e /dev/null; then sudo apt install -y ripgrep fd-find bat fzf elif command -v pacman \u0026\u003e /dev/null; then sudo pacman -S --noconfirm ripgrep fd bat fzf fi The prefix controls when a script runs:\nrun_once_ runs once and never again, tracked by content hash run_onchange_ runs whenever the script’s content changes run_ runs every time Adding mise for tool versions Chezmoi handles dotfiles, but it says nothing about which version of Go, Node, Python or Terraform you have installed. I want those consistent across machines too. Otherwise I’m the one saying “works on my laptop”.\nThat’s the job of mise. It’s a polyglot version manager in the spirit of asdf, faster and a lot more pleasant to use.\nWhy mise and not asdf I ran asdf for years. It works, and it has rough edges. It’s slow, because it’s built on shell scripts. Its plugin ecosystem is fragmented. And there’s no native way to keep its config in your dotfiles.\nMise fixes all three. It’s written in Rust, so it’s fast. Most tools are supported out of the box, no plugin hunting. And its config lives in ~/.config/mise/config.toml, which slots straight into chezmoi.\nInstall mise # Install curl https://mise.run | sh # Or via package manager brew install mise Add it to your shell:\n# ~/.zshrc eval \"$(mise activate zsh)\" Define your tools # ~/.config/mise/config.toml [tools] go = \"1.22\" node = \"lts\" python = \"3.12\" terraform = \"1.7\" kubectl = \"latest\" helm = \"latest\" Now run:\nmise install Every tool gets installed at the pinned version. Switch machines, run mise install, and you’re looking at the same environment. That’s the part I was after.\nProject-specific versions You can also pin versions per project:\n# ~/projects/legacy-app/.mise.toml [tools] node = \"16\" # Old project needs Node 16 Mise switches the moment you cd into the directory. Nothing to remember.\nHow the two fit together Here’s how it looks in my own setup.\nThe mise config goes into chezmoi chezmoi add ~/.config/mise/config.toml Now my tool versions ride along with my dotfiles. Same versions everywhere, tracked in git.\nBootstrap scripts # ~/.local/share/chezmoi/run_once_before_10-install-mise.sh #!/bin/bash if ! command -v mise \u0026\u003e /dev/null; then curl https://mise.run | sh fi # ~/.local/share/chezmoi/run_once_after_90-mise-install.sh #!/bin/bash # Run after dotfiles are in place mise install The before and after in the filename control the order, so mise gets installed first and mise install runs once the config is in place.\nShell integration My .zshrc.tmpl:\n# mise if command -v mise \u0026\u003e /dev/null; then eval \"$(mise activate zsh)\" fi # ... rest of config ... Machine-specific tools Different machines need different tools, and the same templating handles that:\n# ~/.config/mise/config.toml.tmpl [tools] go = \"1.22\" node = \"lts\" python = \"3.12\" {{ if .work -}} terraform = \"1.7\" kubectl = \"1.29\" helm = \"3.14\" {{- end }} {{ if eq .chezmoi.os \"darwin\" -}} # macOS-specific tools {{- end }} A fresh machine in one line This is the payoff. On a fresh machine the whole thing comes down to one command:\n# 1. Install chezmoi and init from repo sh -c \"$(curl -fsLS get.chezmoi.io)\" -- init --apply yourusername # 2. That's it The init script:\nClones my dotfiles repo Prompts for machine-specific data, work or personal, email Decrypts secrets, after asking for the age key Runs the bootstrap scripts, which install mise and packages Applies all dotfiles Inside five minutes I’m in a fully configured environment, secrets included. In the bare-repo days I’d spend an hour hand-fixing configs and still find something missing a week later.\nSmaller things worth knowing External files Need files pulled from a URL or another git repo?\n# ~/.local/share/chezmoi/.chezmoiexternal.toml [\".oh-my-zsh\"] type = \"archive\" url = \"https://github.com/ohmyzsh/ohmyzsh/archive/master.tar.gz\" exact = true stripComponents = 1 refreshPeriod = \"168h\" [\".config/nvim/lazy-lock.json\"] type = \"file\" url = \"https://raw.githubusercontent.com/you/nvim-config/main/lazy-lock.json\" refreshPeriod = \"24h\" Ignoring files # ~/.local/share/chezmoi/.chezmoiignore # Never sync these .DS_Store *.swp .zsh_history # Machine-specific ignores {{ if not .work }} .config/work-vpn/ {{ end }} Re-add after editing If you edit a file directly instead of going through chezmoi edit:\n# Re-add to update source state chezmoi re-add Check for drift Want to know whether your home directory has drifted from what chezmoi expects?\nchezmoi verify Where this leaves me After a bare git repo, GNU Stow, an Ansible playbook and a handful of other tools, chezmoi plus mise is the setup that stuck. The first evening goes into the templates and the age key. After that, a new machine is one command, and a new tool version is an edit to config.toml, a push, and a pull on the other machines.\nIs it overkill if you only ever touch one machine? Probably. If you spin up VMs, reinstall often, or want your config versioned somewhere you control, it’s worth the evening:\nchezmoi init --apply yourusername Resources:\nchezmoi.io - official documentation mise.jdx.dev - mise documentation age encryption - simple, modern encryption ","permalink":"https://tommeurs.nl/en/posts/dotfile-management-mise-chezmoi/","title":"Dotfile Management with mise and chezmoi: The Perfect Combo"},{"content":" TL;DR An e-commerce client moved a PHP monolith to EKS in October, two weeks before Black Friday, with resource limits guessed from the old server specs. Mid-morning the pods hit their memory limit, died in a loop for four hours, and the estimate was €240,000 in lost revenue. The second half is how to get those numbers from load tests and VPA recommendations instead of guessing them.\n“We want to migrate to Kubernetes by November.”\nIt was September. The client was an e-commerce company, and their biggest sales event of the year was Black Friday, in late November. I said no. They asked if I knew someone who might take it on anyway.\nI did. A fellow platform engineer, someone I respect and rate highly. I made the introduction and warned him about the timeline. He took the engagement, documented the same concerns I had, and got them signed off. The client went ahead anyway.\nWhat happened next is his story, told here with his permission. It’s the clearest case I know of why you don’t migrate to Kubernetes without resource metrics, and it lands on something I keep coming back to on this blog. You can’t make good decisions about a system you don’t understand, and metrics are how you understand it. Here’s what he saw, and then how I’d do the sizing instead.\nWhat happened on Black Friday The client had a monolithic PHP application on four dedicated servers. Two app servers, one database server, one for Redis and background jobs. Boring, and it had handled Black Friday traffic for three years.\nBut the CTO had been to a conference. Kubernetes was the future. Microservices were the future. They wanted to containerize everything and deploy to EKS before Black Friday to “handle scale better.”\nThe CTO was new, brought in to drive innovation. The company had been coasting on the same tech stack for nearly a decade, profitable but stagnant. The board wanted modernization, and the CTO needed a win to show the investment was worth it.\nThe timeline wasn’t pure hubris, though. Their legacy hosting contract ended on December 1st. The provider had been acquired, the new owners were sunsetting the platform, and they wouldn’t extend. So either migrate before Black Friday, or go month-to-month on emergency hosting at 3x the cost and migrate during December, the one month when code freezes make everything harder and any outage is catastrophic.\nThe CTO did the math. A risky migration in October with two weeks to stabilize before Black Friday, against a certain cash drain and an even riskier migration in December. The CFO signed off. The CEO signed off. I’d call it a calculated bet rather than recklessness. They just did the calculation without the one input that mattered, because nobody had measured it.\nWhat he warned them about My colleague wrote four things down before he started.\nThey had no metrics. Basic Nagios monitoring, so CPU, memory and disk per server, and nothing at the application level. No request latency histograms, no idea how resource use tracked traffic.\nThey’d never load tested the application. Their only data point was “it survived last year’s Black Friday.”\nThere was no time for proper sizing. Kubernetes resource requests and limits are numbers, and without metrics those numbers would be guesses.\nAnd there was no rollback plan. The old infrastructure was being decommissioned. If Kubernetes failed, there was nothing to fail back to.\nHe recommended extending the hosting contract, even at the higher price, to buy time for a proper run-up. They declined. The math had been done, the executives had signed, and the decision was final.\nWhat they deployed Without metrics, they made educated guesses from the current server specs:\n# What they deployed (DON'T DO THIS) resources: requests: memory: \"512Mi\" cpu: \"250m\" limits: memory: \"1Gi\" cpu: \"500m\" Why set CPU limits at all? My colleague knows the arguments against them. The answer was policy. The client’s platform team required every deployment to specify both requests and limits for every resource, and the rule came from the CTO’s previous company, where a cryptominer had run on their servers for weeks before anyone noticed. Capping CPU on everything was one of the first policies they put in after joining, so a rogue process couldn’t eat unbounded resources. Reasonable, but it assumes you know what limits to set. Without historical data to push back with, and with the clock running, my colleague went along with it. That policy plus the guessed values is what made the day worse than it needed to be.\nThey set up a Horizontal Pod Autoscaler (HPA) on CPU utilization:\napiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: web-app spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: web-app minReplicas: 4 maxReplicas: 20 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 70 I’d have looked at that and nodded. Nothing in it is wrong on its own. It’s wrong for this application, and you can’t tell from the YAML.\nBlack Friday morning Traffic started climbing at 6 AM. By 8 AM they were at 3x normal traffic. The HPA did its job and pods scaled from 4 to 12. Everything looked fine.\nThen 10 AM hit.\nTraffic jumped to 8x normal. The HPA tried to scale to 20 pods. What nobody knew was that the application was memory-bound. Each pod hit the 1Gi memory limit and got OOMKilled. The HPA saw low CPU utilization, because pods were dying before they could use any, and concluded that everything was fine.\nNew pods would start, load the PHP opcache, warm up, start serving traffic, run out of memory, die. Repeat. They had 20 pods cycling through CrashLoopBackOff.\nWhere the memory went The application’s memory footprint depended heavily on the type of request:\nBrowse category: ~200MB View product: ~300MB Add to cart: ~400MB Checkout: ~800MB+ On Black Friday, checkout traffic was 5x normal. Every pod was trying to handle checkout requests that needed 800MB, inside a 1Gi cap with no headroom.\nIt got worse. The CPU requests were too low as well, so the pods that did survive were being throttled. Response times went from 200ms to 8 seconds. Users started refreshing pages. More requests, more memory pressure. Death spiral.\nThe fallout 4 hours of degraded service during peak Black Friday hours Estimated lost revenue: €240,000 Emergency scale-up of the node pool, which took 20 minutes to provision Manual intervention to raise memory limits, which meant a redeployment A post-incident review with very uncomfortable executives The application was fine. Kubernetes was fine. The configuration was the problem, and the configuration was wrong because nobody had the data to make it right. I keep coming back to this bit. The HPA scaled. The scheduler placed pods. The kubelet killed pods that breached their memory limit. Every component did exactly what it was told. What was missing was someone who understood the workload well enough to tell them the right numbers.\nHow to size it with data Here’s how I’d have done the sizing, given the time.\nInstrument before you migrate You can’t size Kubernetes resources without knowing how your application consumes them. Get at least 2-4 weeks of metrics before you plan anything, and at minimum:\nFor memory:\nBaseline memory usage when idle Working set memory under load Peak memory during traffic spikes Memory consumption by request type, if it varies For CPU:\nCPU utilization under normal load CPU utilization during peaks Request latency at various CPU levels Whether the app is CPU-bound or IO-bound For request patterns:\nRequests per second, RPS Request latency percentiles, p50, p95, p99 Traffic patterns by time of day and day of week Historical peak traffic events Load test one container first Before you touch Kubernetes, load test the application in a single container:\n# Run the app in a container with generous limits docker run -d --name load-test \\ --memory 4g \\ --cpus 2 \\ -p 8080:8080 \\ your-app:latest # Monitor resource usage during load test docker stats load-test Use a load testing tool to simulate realistic traffic:\n# Example with k6 k6 run --vus 100 --duration 10m load-test.js Write down:\nThe memory high-water mark The CPU utilization percentage The point where performance degrades Requests and limits from the numbers With the load test data in hand, the arithmetic is short.\nFor memory:\nrequests.memory = p95 memory usage + 20% buffer limits.memory = peak memory usage + 30% buffer For CPU:\nrequests.cpu = average CPU under expected load limits.cpu = peak CPU during burst (or omit to allow bursting) For the e-commerce client, that would have come out as:\n# What they SHOULD have deployed resources: requests: memory: \"1Gi\" # p95 was ~800MB cpu: \"500m\" # Average under load limits: memory: \"2Gi\" # Peak during checkout was ~1.5GB # No CPU limit - allow bursting A note on that missing CPU limit. I’m a proponent of not setting one. CPU is compressible, so if you run out you get throttled. Memory isn’t, so if you run out you get killed. The Google SRE book and many practitioners recommend against CPU limits in most cases.\nLet VPA do the rightsizing The Vertical Pod Autoscaler (VPA) watches actual resource usage and recommends better resource specifications, or applies them for you.\nInstall VPA:\n# Clone the autoscaler repo git clone https://github.com/kubernetes/autoscaler.git cd autoscaler/vertical-pod-autoscaler # Install VPA components ./hack/vpa-up.sh Create a VPA resource in “Off” mode, so you get recommendations without them being applied:\napiVersion: autoscaling.k8s.io/v1 kind: VerticalPodAutoscaler metadata: name: web-app-vpa spec: targetRef: apiVersion: apps/v1 kind: Deployment name: web-app updatePolicy: updateMode: \"Off\" # Just recommend, don't apply After a few days, check the recommendations:\nkubectl describe vpa web-app-vpa Output:\nRecommendation: Container Recommendations: Container Name: web-app Lower Bound: Cpu: 200m Memory: 800Mi Target: Cpu: 450m Memory: 1200Mi Upper Bound: Cpu: 800m Memory: 2Gi The “Target” values are what VPA recommends as requests. Take those as your baseline, then add headroom for the limits.\nHPA and VPA together, carefully HPA and VPA can fight each other. HPA scales horizontally based on metrics, and VPA changes resource requests. If VPA raises the requests, HPA may decide the pods are underutilized and scale down.\nThe approach that doesn’t bite is to run VPA in “Off” mode for the recommendations, apply them by hand as your baseline, let HPA do the horizontal scaling on custom metrics rather than CPU alone, and revisit the VPA recommendations every so often.\nFor the e-commerce client, a better HPA would have looked like this:\napiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: web-app spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: web-app minReplicas: 8 # Higher minimum for Black Friday maxReplicas: 50 # More headroom metrics: - type: Resource resource: name: memory # Scale on memory, not just CPU target: type: Utilization averageUtilization: 70 - type: Pods pods: metric: name: http_requests_per_second # Custom metric target: type: AverageValue averageValue: \"100\" The questions to answer before you migrate Before any Kubernetes migration, answer these with data:\nQuestion Data Source What’s the application’s memory baseline? Monitoring (Prometheus, Datadog, etc.) What’s peak memory under load? Load testing Is the app CPU-bound or memory-bound? Load testing + profiling What’s p99 latency at expected load? Load testing What’s the traffic pattern (daily, weekly, seasonal)? Historical metrics What’s the highest traffic spike in the past year? Historical metrics How does the app behave at 2x, 5x, 10x load? Load testing If you can’t answer them, you’re not ready to migrate. And sometimes the answer is that you shouldn’t, which is its own post, When Not to Use Kubernetes.\nThe second attempt After Black Friday, the client agreed to the expensive emergency hosting they’d been trying to avoid. Three times the cost, but it bought them time. My colleague used December to do the migration properly.\nHe instrumented everything first. Prometheus, application metrics, custom metrics for the checkout flow. Then they load tested against simulated Black Friday traffic patterns before moving anything. They started with generous resources, overprovisioned on purpose, and used VPA to rightsize from there. They validated with canary deployments, running the new configuration alongside the old and comparing behaviour. And they kept a way back. The emergency hosting stayed available as a fallback for 30 days after the migration.\nThe migration itself, late December after the Christmas rush, was boring. Nothing broke.\nWhy I said no Kubernetes doesn’t fix a scaling problem you haven’t measured. It runs the numbers you give it, guessed or measured, just as faithfully at Black Friday traffic.\nEvery number in a resource spec is a claim about how the application behaves. Requests, limits, the HPA target, all of them. Guessed from server specs, they’re claims nobody can back. Measured, they’re the reason the second attempt was boring.\nI said no in September because I couldn’t see how anyone would get those numbers in time. My colleague took the job and got them the expensive way. It’s his story, and I’m grateful he let me tell it.","permalink":"https://tommeurs.nl/en/posts/kubernetes-resource-sizing/","title":"Data-Driven Kubernetes Migration: Why You Need Metrics Before You Move"},{"content":" TL;DR I pointed K8sGPT at a Llama 3.3 70B model running on a Mac Studio in my own office, so no cluster metadata leaves the network, and fed it three broken workloads. It found all three and wrote fixes I’d accept from a colleague, except for the resource numbers, which it made up. Assisted diagnosis works, at 25 seconds a query. Autonomous cluster management is still marketing, and the security section is why I wouldn’t let a model change anything on its own.\nEvery vendor pitch deck right now has the same slide. “Autonomous cluster management.” An AI watches your Kubernetes cluster, spots the problem, diagnoses it and fixes it while you sleep. Platform engineers stop firefighting and the cluster heals itself.\nI run a homelab because I want to know what’s going on inside it. So when I see a slide like that, I want to run it before I believe it.\nThe catch with most of these tools is what “AI” means once you read the fine print. It means your cluster metadata gets shipped to OpenAI. Pod names, labels, error messages, the lot. I self-host precisely so that data stays on my network, and an AIOps story that only works once I hand it to a third party has nothing to offer me.\nSo here’s the version I wanted to try. K8sGPT pointed at a Llama 3.3 70B model running locally on Apple Silicon. No cloud API, no data leaving the building. Is a local model good enough to be useful for cluster diagnosis? Or does sovereignty here cost more than it’s worth? That’s the question this post answers.\nOne thing before the commands. This is a homelab experiment. I’m reporting what I tested and what I found, and the security section further down is the reason you shouldn’t run any of it in production. Read it that way.\nThe hardware and the software The hardware A Mac Studio M3 Ultra with 512GB of unified memory. The M3 Ultra has 80 GPU cores you can use for inference, and unified memory means nothing gets copied between CPU RAM and GPU RAM.\nAt around €10,000 it’s an expensive box, and I’m not pretending otherwise. It’s also about the only consumer hardware that runs a 70B model at Q8 quantization at a speed you’d sit and wait for. The unified memory is the whole trick. On a normal PC with a discrete card you pay a copy tax between CPU and GPU RAM on every pass, and that tax is what kills you.\nThe software stack Component Version Role vLLM 0.6.x Inference server with Metal backend Llama 3.3 70B Q8_0 The language model (~75GB) K8sGPT Operator 0.1.x Kubernetes operator for diagnosis k3s 1.29.x Local Kubernetes cluster Installing vLLM with the Metal backend vLLM has experimental Metal support for Apple Silicon. The install is a conda environment and one pip:\n# Create a dedicated conda environment conda create -n vllm python=3.11 conda activate vllm # Install vLLM with Metal support pip install vllm # Verify Metal backend python -c \"import vllm; print(vllm.__version__)\" Experimental is the right word. At the time of writing the Metal support in vLLM is rough, and you notice. llama.cpp with its Metal backend is the stable option, but K8sGPT wants an OpenAI-compatible API and vLLM gives me one out of the box. So I took the rougher path rather than write a shim.\nDownloading the model # Download the model (Q8 quantization, ~75GB) huggingface-cli download meta-llama/Llama-3.3-70B-Instruct \\ --local-dir ./models/llama-3.3-70b-instruct You need a Hugging Face account and you have to accept the Llama license.\nStarting the vLLM server # Start the inference server vllm serve ./models/llama-3.3-70b-instruct \\ --host 0.0.0.0 \\ --port 8000 \\ --dtype float16 \\ --max-model-len 8192 \\ --device mps The --device mps flag forces Metal Performance Shaders. Leave it off and vLLM falls back to the CPU.\nCheck that it’s up:\ncurl http://localhost:8000/v1/models Deploying the K8sGPT operator The operator goes into the cluster with Helm :\n# Add the Helm repo helm repo add k8sgpt https://charts.k8sgpt.ai/ helm repo update # Install the operator helm install k8sgpt-operator k8sgpt/k8sgpt-operator \\ -n k8sgpt-operator-system \\ --create-namespace Then a K8sGPT resource that points the backend at the vLLM server on the Mac:\n# k8sgpt-config.yaml apiVersion: core.k8sgpt.ai/v1alpha1 kind: K8sGPT metadata: name: k8sgpt-local namespace: k8sgpt-operator-system spec: ai: enabled: true model: llama-3.3-70b-instruct backend: localai baseUrl: http://192.168.1.100:8000/v1 # IP of your Mac Studio noCache: false version: v0.3.40 analyzers: - Pod - Deployment - Service - ReplicaSet - PersistentVolumeClaim - Ingress - StatefulSet - CronJob kubectl apply -f k8sgpt-config.yaml Three broken things The setup runs. Now the part that matters. Is it useful, or is it a very expensive way to get a wrong answer with confidence? I threw three deliberately broken things at it and graded the output the way I’d grade a junior engineer’s first pass.\nScenario A: a CrashLoopBackOff A deployment that references a ConfigMap which doesn’t exist:\n# broken-deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: broken-app namespace: default spec: replicas: 1 selector: matchLabels: app: broken-app template: metadata: labels: app: broken-app spec: containers: - name: app image: nginx:1.25 envFrom: - configMapRef: name: app-config # This ConfigMap doesn't exist kubectl apply -f broken-deployment.yaml After a minute the pod is in CrashLoopBackOff. The operator writes its analysis to a results resource:\nkubectl get results -n k8sgpt-operator-system -o yaml Paraphrased, the output was this:\nAnalysis: Pod broken-app-xxxx is in CreateContainerConfigError state. The pod is referencing a ConfigMap 'app-config' that does not exist in the namespace. Suggested remediation: 1. Create the missing ConfigMap: kubectl create configmap app-config --from-literal=KEY=value 2. Or remove the configMapRef from the deployment spec 3. Verify the ConfigMap name spelling matches exactly Usable. The diagnosis is right and the suggestions are practical. A junior engineer could work from this and not get led astray. What it missed is the one check I’d have done first, whether the ConfigMap lives in another namespace. That’s the mistake I make about once a month.\nScenario B: no resource limits A deployment with three replicas and no resources block at all:\n# no-limits.yaml apiVersion: apps/v1 kind: Deployment metadata: name: no-limits-app namespace: default spec: replicas: 3 selector: matchLabels: app: no-limits template: metadata: labels: app: no-limits spec: containers: - name: app image: nginx:1.25 # No resources defined K8sGPT’s analysis:\nAnalysis: Deployment no-limits-app has containers without resource requests or limits defined. Issues identified: - Container 'app' has no CPU requests/limits - Container 'app' has no memory requests/limits Suggested remediation: Add resource specifications to ensure predictable scheduling and prevent resource starvation: resources: requests: memory: \"64Mi\" cpu: \"100m\" limits: memory: \"128Mi\" cpu: \"200m\" Partially usable. It spots the problem. The numbers it hands you are made up. The model never saw a metric, so nothing in those requests and limits is based on what this workload does. An experienced engineer pulls real data first, kubectl top pods or Prometheus, and sizes from there. The LLM guesses, and nothing in the output says so.\nIt also doesn’t mention the Vertical Pod Autoscaler, doesn’t warn that limits without requests can bite you, and picks the 1:2 ratio between requests and limits out of thin air.\nScenario C: a privileged pod A pod that does most of the things a security policy exists to stop:\n# insecure-pod.yaml apiVersion: v1 kind: Pod metadata: name: insecure-pod namespace: default spec: hostNetwork: true containers: - name: app image: nginx:1.25 securityContext: privileged: true runAsRoot: true volumeMounts: - name: host-root mountPath: /host volumes: - name: host-root hostPath: path: / The analysis, with the Trivy integration switched on:\nAnalysis: Pod insecure-pod has multiple security issues: Critical: - Container running in privileged mode (privileged: true) - Container has access to host network (hostNetwork: true) - Container mounts host root filesystem High: - No securityContext.readOnlyRootFilesystem - No securityContext.allowPrivilegeEscalation: false Suggested remediation: 1. Remove privileged: true unless absolutely necessary 2. Use a dedicated network namespace (remove hostNetwork: true) 3. Avoid mounting host paths, especially root filesystem 4. Add security hardening: securityContext: privileged: false readOnlyRootFilesystem: true allowPrivilegeEscalation: false runAsNonRoot: true capabilities: drop: - ALL Usable. This one surprised me. The issues are all there and the remediation reads like what a security engineer would tell you. Trivy earns its place here by flagging the image vulnerabilities as well, so you get the misconfiguration and the CVE story in one pass.\nWhat’s missing is the policy layer. No mention of Pod Security Standards and the restricted profile, and no word that a Kyverno or OPA policy should have blocked this pod before it ever scheduled.\nPerformance Inference speed Metric Value Tokens/second (prompt) ~180 t/s Tokens/second (generation) ~25 t/s First token latency ~2.5s Typical analysis (500 tokens out) ~22s End-to-end latency From the moment a problem exists to the moment the report lands:\nPhase Time Issue detection (polling) 30s (configurable) Context gathering ~2s LLM inference ~20-30s Result storage \u003c1s Total ~55s Resource usage During inference:\nResource Usage GPU Memory (Metal) ~78GB CPU ~15% (data preprocessing) System Memory ~12GB (besides model) Power draw ~180W Against the OpenAI API Metric Local (70B) OpenAI GPT-4 Latency ~25s ~5s Quality Good Very good Cost €0 (after hardware) ~€0.03/query Privacy Fully local Data to OpenAI The OpenAI API is faster and the output is a bit better. The price is that your cluster metadata leaves your network. For me that trade is the whole point of the exercise. I’ll wait 25 seconds instead of 5 to keep the data at home, and I don’t find that a hard call. Whether you make the same one depends on what’s in your pod labels.\nAir-gapped Can it run with no internet at all? Yes, with some preparation. This is where the local model pays for itself, because the air-gapped case is off the table the moment you depend on a cloud API.\nWhat to download beforehand # 1. Model weights (~75GB) huggingface-cli download meta-llama/Llama-3.3-70B-Instruct \\ --local-dir ./airgap-bundle/models/ # 2. vLLM Python packages pip download vllm -d ./airgap-bundle/packages/ # 3. K8sGPT container images docker pull ghcr.io/k8sgpt-ai/k8sgpt-operator:latest docker save ghcr.io/k8sgpt-ai/k8sgpt-operator:latest \u003e ./airgap-bundle/images/k8sgpt-operator.tar # 4. Helm charts helm pull k8sgpt/k8sgpt-operator --destination ./airgap-bundle/charts/ Transport and install # On the air-gapped machine: # 1. Install Python packages offline pip install --no-index --find-links=./airgap-bundle/packages/ vllm # 2. Load container images docker load \u003c ./airgap-bundle/images/k8sgpt-operator.tar # Or push to your local registry # 3. Install Helm chart helm install k8sgpt-operator ./airgap-bundle/charts/k8sgpt-operator-*.tgz \\ --set image.repository=your-local-registry/k8sgpt-operator What works offline Component Air-Gap Ready Notes vLLM Yes No phone-home Llama model Yes One-time download K8sGPT Operator Yes No telemetry Trivy DB No Requires periodic updates The Trivy vulnerability database is the exception. It needs periodic updates, and you’ll have to transport those separately. Without a recent database, K8sGPT misses new CVEs.\nSecurity and the threat model This is where I have to be honest. A local model gives you data sovereignty and nothing else on the security side. The setup is as secure as the box it runs on, and a Mac Studio on your network as an inference server brings its own pile of problems. The LLM brings a second pile that’s specific to it.\nPlatform issues A Mac Studio as an inference server has limits you can’t configure away:\nIssue Impact No TPM No hardware attestation, no measured boot macOS is general-purpose Not hardened like RHEL/Ubuntu with CIS benchmarks No Secure Boot chain Boot process is not cryptographically verified Updates require internet Or manual intervention in air-gapped scenario Single-user focus macOS is not designed for multi-tenant security So for anything with real compliance requirements, ISO27001 Annex A, NIS2 , SOC2, a Mac Studio is the wrong machine. For a homelab or a development environment it’s fine.\nRisks that come with the model Risk Description Non-determinism Same input can produce different outputs Prompt injection Malicious pod names/labels can manipulate the LLM Hallucinations Model can suggest harmful remediation Context leakage Info from earlier queries can appear in responses Supply chain Model weights could be backdoored The threat model Threat Likelihood Impact Mitigation Prompt injection via pod metadata Medium High Input sanitization, output validation Hallucinated destructive commands Medium Critical Human-in-the-loop, no auto-remediation Model weights tampering Low Critical Checksum verification, trusted source Context window data leakage Medium Medium Short context, no persistent memory Unauthorized access to inference API Medium High Network segmentation, auth Resource exhaustion (DoS) Low Medium Rate limiting, resource quotas What I’d do with it Usable, under conditions Yes, a local model is usable for Kubernetes diagnosis, under conditions. It identifies standard issues correctly, gives remediation you can act on, catches security problems, and does all of that without a byte leaving the network.\nWhat it can’t do matters as much. It can’t debug a problem that spans several components. It can’t do auto-remediation you’d trust. It doesn’t know your setup, and it can’t guarantee it’s right.\nPer use case Homelab and learning Go for it. You’ll learn about LLM inference infrastructure, about Kubernetes troubleshooting patterns and about where AI-assisted operations stop being useful, and the blast radius is your own cluster.\nDevelopment and staging Usable with guardrails. Someone reviews the output before anything gets applied, every LLM interaction is logged, and there’s no auto-remediation. Diagnosis only.\nProduction, connected Use a cloud API. The models are better, GPT-4 or Claude, the latency is lower, there’s no hardware to buy and someone else signs the SLA . For most organisations the privacy trade-off is acceptable, as long as there’s no PII in the cluster metadata.\nProduction, air-gapped or sovereign Only as a last resort. If the data truly can’t leave, look at smaller, dedicated models first and put defence in depth around the inference server. Treat every line the model produces as untrusted, log and audit everything, and keep it an assistant rather than an authority.\nBack to the slide Autonomous cluster management with LLMs is marketing today. What I tested is assisted diagnosis, a model that points at problems and offers suggestions, with me reading every one of them. The human in the loop is a requirement, and it doesn’t come off once you trust the model enough.\nAnd I’m impressed anyway. A 70B model on hardware in my own office produced analyses I’d accept from a colleague. The gap between good enough to read and good enough to act on a production cluster without me is still wide. Scenario B shows how wide. The model invented resource limits and presented them as advice.\nI got the answer I came for. A local model is good enough to be a useful diagnostic assistant, and running it here means I never had to choose between AIOps and the reasons in Sovereign Infrastructure. Why Privacy Matters is the other half of why the data stays home. So it stays an assistant. I read every suggestion before I type it, the way I’d read one from a sharp colleague who’s never seen my cluster.","permalink":"https://tommeurs.nl/en/posts/k8sgpt-local-llm-cluster-management/","title":"K8sGPT with a Local 70B Model on Apple Silicon"},{"content":"Your phone logged where you slept before you woke up. You check Gmail, search something on Google, message a friend on WhatsApp, pay for coffee with a card and walk past a dozen cameras before lunch. None of that feels like surveillance. It feels like a normal morning.\nIt felt the same five years ago, and I want to show you what changed in between. The comfortable version of privacy, “I have nothing to hide”, has quietly stopped being true, and I think most people haven’t noticed. Three things happened at once. On its own each one is manageable. Together they’re what worries me.\nThe collection is finished Start with the part nobody disputes anymore. Every email, text, call, location ping, purchase and search query gets collected somewhere. Snowden showed us the plumbing in 2013. We’ve had thirteen years to make our peace with it, and most of us did.\nWhat changed is the completeness. In 2013 the collection had gaps. Today the data portrait of anyone in the developed world is finished, and it lives in dozens of databases at once. Government ones, corporate ones, criminal ones.\nWant the list? The websites you’ve visited, from ISP logs and browser fingerprinting. The places you’ve been, from cell towers, GPS and wifi positioning. Your purchases, through payment processors and loyalty cards. Who you’ve talked to, as metadata on every platform. Your photos, in cloud backups and facial recognition databases. And every word you’ve typed, through keyboard apps and autocomplete training data.\nThat data exists, and it’s stored indefinitely. Whether someone could assemble a complete profile of your life was never the open question. The question is who already has, and what they plan to do with it.\nThe other version of that morning Now picture the same morning, wired differently. Your email lands on a server in your own house. Your photos sync to storage you control. Your passwords live in a vault only you hold the keys to. You still pay for the coffee and the cameras are still on the street, but the bulk of your digital life isn’t sitting in someone else’s database waiting to be queried.\nIs that a fantasy? No. It’s a Saturday afternoon of setup and a small monthly electricity bill. I run it. Plenty of people run it. Most of us don’t because the default path is so smooth that opting out never crosses your mind.\nHold on to that picture. The next two trends are why I’m glad I did.\nThe analyst bottleneck is gone Large language models changed the maths.\nFor decades the intelligence agencies had a needle-in-a-haystack problem. They could collect everything and process almost none of it. The NSA famously hoarded more data than it could ever read. Human analysts were the bottleneck, and that bottleneck was the only real protection most people had.\nAn LLM reads and summarises millions of documents in hours. It finds patterns across unrelated sources, builds a profile of one person from scattered fragments, predicts behaviour from history and flags the “interesting” people for a human to look at. The data that sat in cold storage for years is searchable now. The embarrassing message you sent in 2015. The protest you turned up to in 2020. The late-night search you assumed nobody would ever connect back to you.\nAnd there’s a nasty twist. LLMs hallucinate. They produce plausible-sounding facts that never happened, with full confidence. In an interrogation room or a courtroom, how do you prove you didn’t do the thing a model says you did? I don’t know either, and I’d rather not find out.\nThis is where the opt-out version of the morning stops being a lifestyle choice and becomes a threat model. The less of you that sits in a queryable database, the less there is for a model to mine, misread or invent.\nEighteen years of decline This is the part people would rather not discuss.\nFreedom House counted 2024 as the 18th consecutive year of global democratic decline. More people live under authoritarian rule now than at any point since the Cold War. The democratic expansion after 1991 is over.\nLook at the map:\nRussia: authoritarian China: authoritarian India: backsliding Turkey: authoritarian Hungary: illiberal democracy Brazil: narrowly avoided it United States: institutions under strain This matters for privacy because authoritarian governments use the data, in production, today. China’s social credit system. Russia’s SORM surveillance. Iran switching off the internet during protests. All three are in the news, and have been for years.\nDemocracies aren’t exempt. The UK’s Online Safety Bill. The EU’s proposed chat control. The renewals of FISA Section 702 in the US. Even a healthy democracy keeps building surveillance plumbing that a future, less healthy government inherits along with the keys.\nA regime that doesn’t fall This is the synthesis that keeps me up.\nHistorically, authoritarian regimes were fragile. They collapsed because they couldn’t hold control, couldn’t spot threats fast enough, couldn’t suppress dissent without making more of it. The 20th century is a graveyard of dictatorships.\nMass collection plus AI analysis plus authoritarian governance produces something we haven’t seen before. Something that might be stable.\nImagine a regime that knows every relationship in society through social network analysis, predicts who might become a dissident from their behaviour, intervenes before a threat forms, personalises propaganda for each citizen and picks out the organisers the moment they start organising. I’m not speculating. China is building this, and the technology travels. Dozens of countries have bought Chinese surveillance infrastructure off the shelf.\nWe may be watching the birth of stable authoritarianism. Regimes that no longer fall, because they’ve solved the information problem that used to constrain every despot in history.\nYour own government has the same data If you assume this only applies to faraway autocracies, you haven’t been watching closely.\nThe Five Eyes alliance, the US, the UK, Canada, Australia and New Zealand, shares intelligence freely. What the NSA can’t legally collect on Americans, GCHQ collects and passes over. What GCHQ can’t collect on Brits, the NSA returns the favour on.\nThe legal frameworks that supposedly protect citizens get routed around all the time. Parallel construction, where an agency uses surveillance data and pretends it never had it. The third-party doctrine, which strips protection from any data a company holds for you. FISA courts, which are secret courts with secret readings of the law. Executive orders that skip congressional oversight altogether.\nYour Western democracy holds the same data the authoritarian regimes hold. The difference so far is how each has chosen to use it. Governments change, though, and laws change. The data stays.\nWhy I self-host This is the whole reason I self-host everything. I’m not up to anything. I take the threat model seriously.\nWhen your email is on Gmail, Google has it, and Google hands it to any government that shows up with the right paperwork, and occasionally without. Photos on iCloud belong to Apple. Documents on OneDrive belong to Microsoft. You’re a guest in all of those houses.\nSelf-hosting drags the data back under my own roof. My mail server, my cloud storage, my password manager, all running on hardware I can physically touch. A government can still demand access, sure. First it has to know the data exists, and then it has to come to me specifically to get it.\nThat gap is the difference between mass surveillance and targeted surveillance. If I ever become a target, I can’t stop the targeted kind. The bulk dragnet I can step out of.\nWhat you can still change I’m not going to tell you to delete every account and move to a cabin in the woods. That’s unrealistic for almost everyone, and honestly, the historical data is gone. Whatever’s out there is out there.\nWhat you can change is what gets created from here.\nMinimise new data. Use end-to-end encrypted messaging, so Signal rather than WhatsApp. Use a VPN for browsing, because your ISP doesn’t need a log of every site. Use a privacy-respecting email provider, ProtonMail, Tutanota or self-hosted. Use a password manager, so a breach stays contained to one account.\nReduce exposure. Audit your cloud services and ask what needs to live there. Use local-first software where you can. Consider self-hosting the services that matter most, my earlier posts walk through how.\nSupport the people fighting this at the systemic level. Back the EFF, the ACLU and the digital rights groups. Vote for representatives who understand technology. Push for strong data protection laws.\nAnd accept the trade-offs. Privacy usually means a bit more inconvenience. Self-hosting costs you time and effort. You can’t control what other people do with data you share.\nI’m being honest about that last block on purpose. Sovereignty has a price, and pretending otherwise would be dishonest. The price is a few weekends and some ongoing maintenance. What you buy is control over what governments and corporations can trivially learn about you.\nWhat privacy is for Privacy is about power.\nWhen a government or a corporation knows everything about you, it holds power over you. It can predict your behaviour and nudge your decisions. It can single you out, and make an example of you to keep everyone else in line.\nA democratic society depends on a power balance, and privacy is what preserves it. Citizens need parts of their lives the state can’t see or steer. Take that away and dissent gets expensive. Make dissent expensive enough and democracy stops working, quietly, without a coup.\nWe’re assembling the infrastructure for perfect totalitarianism right now. The data is here and the models are here. The only variable left is governance, and governance can flip overnight.\nSo don’t wait until it does. Move what you can of your data under your own roof, build a little of your own infrastructure and help the people doing the same. My email lands on my own server now, the morning looks the same from the outside, and the list of people who can read it is a lot shorter.","permalink":"https://tommeurs.nl/en/posts/why-privacy-matters-now/","title":"Why Privacy Matters More Than Ever"},{"content":"In my previous post about self-hosting for everyone I drew a hopeful picture of managed self-hosting, where anyone can own their data. I skipped over something, and it’s been bugging me since. Hardware costs money, and the sum comes out very differently depending on whether a business or a person is paying.\nSo here are two rules. Both sound reasonable on their own. Then I’ll put them in the same room.\nRule one: own your hardware This is the whole point of sovereignty. You buy the box, you run the box, nobody can pull your data out from under you. I made that case in Sovereign Infrastructure, and it comes down to wanting to understand and control what I run. Renting the box from someone else brings back the exact dependency I was trying to leave.\nI don’t know anyone serious about their own infrastructure who’d argue with this. Owning the hardware isn’t up for negotiation.\nRule two: don’t set money on fire The second rule is just as sound. Pay the lowest defensible price for the hardware you need. Nobody pays more for an identical box on principle.\nAnd this is where it starts to hurt. Take a business buying a €1,000 server, next to me buying the same one:\nFactor Business Individual VAT Deductible (€0 net) €210 lost Corporate tax benefit ~25% deduction (€197) None Write-off 5 years depreciation Just an expense Cash flow Business expense After-tax income Net cost ~€593 €1,000 The business pays 40% less for the same machine. The tax code rewards the legal entity, and I’m not one. It gets worse once you run the full numbers.\nTotal cost of ownership, a real example Let’s price out a 5-year TCO for a modest self-hosting cluster. Three mini PCs, the Lenovo ThinkCentre Tiny and HP EliteDesk Mini class, which is the minimum for a proper high-availability setup.\nHardware costs Item Price Qty Total Refurbished mini PC (i5, 16GB, 256GB) €150 3 €450 1TB NVMe SSD upgrade €80 3 €240 Managed switch €50 1 €50 UPS (basic) €100 1 €100 Cables, misc €50 - €50 Total hardware €890 Running costs, 5 years Item Monthly 5-Year Total Electricity (~45W × 3 × 24/7) €15 €900 Domain name €1 €60 Backup storage (off-site, 100GB) €3 €180 Total running €19 €1,140 Total 5-year TCO Scenario Hardware Running Total Monthly Business €534 €1,140 €1,674 €28 Individual €890 €1,140 €2,030 €34 For an individual that’s about €34/month over 5 years. Competitive with cloud services, honestly. Right up until you put your own time on the bill.\nThe time problem This is where the sum falls apart for most people.\nValue your time at €30/hour, conservative for a professional, and spend 2 hours a month on maintenance:\nIndividual (DIY) Individual (Managed) Hardware + running €34/month €34/month Time (2h × €30) €60/month €0/month Management fee €0 €15/month Total €94/month €49/month Doing it all yourself costs nearly double what a managed service would. And that’s the rosy version, with a calm 2 hours a month. One real incident eats that allowance in an afternoon. I’ve had those afternoons. So rule two has an obvious answer. Let someone else run it, at scale, with the tax break and the cheaper hardware.\nWhere the two rules meet Now put them in the same room.\nRule two says a business should buy and run the hardware, because it pays 40% less and spreads the hours over many customers. Rule one says I should own the hardware, because renting it back brings back the dependency I’d just left.\nWalk it through. To honour rule two, I let a managed provider buy the box. But the moment the provider owns the box my data runs on, I’ve rebuilt the cloud provider with extra steps. My sovereignty now depends on their goodwill, their solvency and their terms of service. To honour rule one, I buy the box myself, pay the full consumer price, eat the 40%, and carry every hour of maintenance alone.\nI need the business to make ownership affordable. And the business owning it is what takes the ownership away.\nThat’s the hardware ownership paradox. The most rational choice with the money undoes the reason for self-hosting in the first place. Both rules hold. I can’t follow both.\nWhat people do about it I’ve spent more evenings than I’d like to admit on this with the spreadsheet open, and I keep landing on the same four workarounds. None of them is clean. Every one is someone threading a needle the tax code never meant to leave open.\n1. The cooperative model What if the customers owned the management company together?\nCustomer Cooperative\nCustomers are shareholders\nBuys HW wholesale\nProvides mgmt\nMember Cluster\nMember Cluster\nThe cooperative buys the hardware at business rates and passes the saving on to its members. Members own their cluster hardware through the coop, and management is a service they buy from something they own.\nOn the money side, hardware costs drop by about 40%, bulk purchasing adds another 10-15% on top, and the TCO could land around €22/month.\nThe catch is everything around the hardware. Legal complexity, which varies by country. Governance overhead. And somewhere between 20 and 50 members before the numbers work, though I’m guessing at that.\nThe clever part is that the business buying the hardware is you, with the other members. Rule two is satisfied without breaking rule one, because the entity that owns the box belongs to the people whose data runs on it.\n2. The hardware loan model The provider buys the hardware, and ownership moves to the customer over time.\nYear Monthly Payment Ownership % 1 €35 0% → 20% 2 €35 20% → 40% 3 €30 40% → 60% 4 €25 60% → 80% 5 €20 80% → 100% Total paid, €1,740 over 5 years, and at the end the customer owns the hardware outright.\nWhy does it work? The provider takes the tax benefits early, the customer builds up equity month by month, and after year 5 the bill drops to running costs plus management. Real ownership, only later than you’d like.\nThis one breaks rule one for a while so it can satisfy it for good. You rent for five years so that you own forever.\n3. Bring your own hardware BYOH, bring your own hardware. The simplest way out. The customer buys the boxes and the provider only does the management.\nThe provider does the initial setup and configuration, the monitoring and maintenance, the security updates and the disaster recovery. The customer buys the hardware, finds it a spot at home or in the office, and pays the electricity.\nA worked price: €100 one-time setup, €15 a month for management, and the customer buys the €890 of hardware themselves.\nOver 5 years that’s €890 + €100 + (€15 × 60) + €1,140 = €3,030, or €50/month.\nMore expensive than the business-owned version, because you swallow the full consumer price. But sovereignty holds from day one. Rule one wins outright here, and pays rule two’s premium as the price of admission.\n4. The refurbished marketplace This is the one I keep coming back to. A cooperative or non-profit that sources enterprise hardware at end of lease, refurbishes and tests it, sells it to members at cost plus a small margin, and puts the surplus back into the community.\nEnterprise hardware sells for a fraction of what it cost new. A 3-year-old Dell OptiPlex that was €800 new can go for €80 at auction. The cooperative takes on the painful parts, sourcing, testing and warranty, and members get sovereign hardware at close to business prices. It attacks the 40% gap from the other side, with cheaper hardware rather than a cleverer form of ownership.\nWhere I’m starting For my service cluster model I’m starting with a hybrid, because none of the four breaks the paradox on its own.\nPhase 1 is BYOH for early adopters. Customers buy their own hardware, I manage it for €15-20/month, and the sovereignty is real from day one.\nPhase 2 adds the hardware loan, for people who can’t or won’t pay for hardware upfront. Ownership transfers over 3-5 years, the monthly cost is higher, and there’s no barrier at the start.\nPhase 3, if this ever grows past friends and family, is a formal cooperative for bulk hardware purchasing, with the members running the service democratically.\nEach phase adds rules the first two didn’t have, and that’s the cost. Three phases and a contract where there used to be one purchase. But the loop is open.\nThe part I can’t fix I won’t pretend there’s a clean answer. The tax code favours businesses, and hardware owned by a person will always cost more than hardware owned by a company.\nHere’s where I’ve landed, after all those evenings.\nSovereignty has a value that never shows up in a TCO spreadsheet. Peace of mind, principle, independence. No line item, but I pay for them gladly.\nThe gap is smaller than it first looks. €34/month for real ownership against €25-30/month for rented sovereignty. For a lot of people that difference won’t decide it.\nTime is the variable that moves the most. Take the hours out through a managed service while the hardware stays in your name, and the numbers swing hard in your favour.\nAnd delayed ownership still beats no ownership. A hardware loan that ends with the title in your name is a different thing from renting forever.\nThe paradox is real, and I’ve stopped trying to make it disappear. Owning costs a bit more than depending. I’ve looked at the spreadsheet enough evenings to know that, and I’m paying it anyway.","permalink":"https://tommeurs.nl/en/posts/hardware-ownership-paradox/","title":"The Hardware Ownership Paradox: When Sovereignty Gets Expensive"},{"content":"Most people keep their photos at Google, their mail at Gmail, their documents in whatever cloud came with the phone and their passwords in a browser built by an advertising company. Every byte sits on a server they don’t own, under terms they didn’t read, in a country they didn’t check. Nobody finds that strange anymore, because the alternative looks like a second job.\nI want to talk about that alternative, and about why it sits behind a wall most people will never climb.\nThe deal underneath the free storage The services are free because the business model is knowing you. Your mail gets scanned, your photos get analysed, your location history is a product with a price on it. Nobody hides this anymore, and most people have made a rough peace with it. The data is convenient where it is, and moving it sounds like more hassle than it’s worth. I get that. It still bugs me.\nThe technical crowd has an answer, and it’s a good one. Self-hosting. Run the services on your own hardware and the surveillance problem mostly goes away, because there’s nobody on the other end to do the surveilling.\nI’ve run my own infrastructure for years. My own email, my own cloud storage, my own password vault, the lot. I love it, and I’ve written a fair bit about why on this blog. But I’m not going to pretend my parents could do the same. Self-hosting today is a privilege. It costs technical knowledge and a steady supply of evening hours, and most people have neither to spare. The barrier is real, and it’s high.\nThe other side of the wall Picture it. You pay a monthly fee, somewhere between €15 and €30. For that you get a private cloud for your photos, documents and notes, your own email server, a password manager, calendar and contacts. The same things people rent from Big Tech today, with four differences underneath.\nYour data stays yours, so no scanning, no profiling, no quiet resale to a data broker. It runs on commodity hardware, and the old PC in your closet qualifies. You can leave whenever you want, take your data and walk, no lock-in. And the operations are transparent, open source and auditable, no black boxes.\nNone of this is new. Nextcloud, Mailcow and a long list of mature open-source projects have covered the building blocks for years. What’s always been missing is the person who assembles them and then keeps them running. The whole idea is that this person stops being you.\nThe cost shows up after the weekend Setup is a weekend. A determined person with a tutorial gets there. The cost comes afterwards, every week, for as long as the thing exists. Certificates expire. A disk fills up while you’re asleep. An upstream image ships a breaking change and the contacts app stops syncing on the day you need a phone number. A self-hosted setup is something you keep alive, week after week.\nThat maintenance load is why I’ve never handed a Raspberry Pi to a non-technical friend and called it a present. What they’d be getting is a pager.\nThis is the same friction that drives most of my technical choices. I want systems that fail gracefully and heal themselves, because otherwise I’m the single point of failure for someone else’s email, and that doesn’t hold up. It isn’t kind to either of us.\nSelf-hosting is good. The open question is who carries the operational weight.\nOne service cluster, many customer clusters I’ve started building an answer. This is the shape of it.\nCustomer Cluster C (Colo)\nEmail\nMatrix\nNextcloud\nCustomer Cluster B (Office)\nNextcloud\nGitLab\nCI/CD\nCustomer Cluster A (Home)\nNextcloud\nVaultwarden\nPhotos\nService Cluster\n(Managed by me, runs management plane)\nGitOps\n(ArgoCD)\nMonitoring\n(Grafana)\nUpdates\n(Auto)\nA central service cluster manages a fleet of customer clusters. Each customer cluster runs on hardware the customer owns, whether that’s a mini PC at home, a server in an office or rented hardware in a datacenter.\nThe service cluster handles the parts that hurt. Deployment, GitOps-driven, declarative and reproducible. Monitoring, so something gets caught before the customer notices. Updates, automatic, tested and rolled back when they misbehave. Backups, encrypted, off-site and restored once in a while to prove they still work.\nThe customer cluster does two things. It runs the services, and it stores the data on hardware the customer can physically touch.\nThat split is the whole point. The service cluster never sees customer data. It manages the infrastructure, and the data stays on the customer’s side of the line. The babysitting is done by me and my tooling, and the person who just wanted their photos to stay private gets exactly that.\nRunning on old hardware One thing I won’t give up. This runs on hardware people already own.\nThe old laptop, the retired office PCs, the Raspberry Pi on the shelf, all of it qualifies. K3s runs comfortably on machines Big Tech wrote off as obsolete years ago. That keeps usable hardware out of the bin and brings the cost of entry close to zero. It also makes the ownership literal. The box is in your house, and nobody in a datacenter holds a lease on it.\nI’ve written before about sovereign infrastructure and why physical control is the part that counts. This is the same idea pushed one step further, out of the homelab and into the hands of people who’ll never build one.\nThe part I haven’t proven yet Now the bit I find exciting, with the caveat that it’s the least tested.\nToday I’m the operator. I read the alerts, I debug the failures, I make the calls. That works for friends and family. It doesn’t work for everyone, because my attention is the bottleneck and there’s only one of me.\nLarge language models are getting good at the judgement operations work asks for. Reading logs and metrics, diagnosing a common fault, proposing a fix, weighing context. Picture a system that notices a customer’s disk creeping towards full, looks at the usage pattern, recommends a cleanup or an expansion, applies whichever gets chosen, and remembers how it went for next time.\nThe building blocks for that exist today. The guardrails don’t, and that’s the part I’m careful about. An LLM with write access to someone’s email infrastructure is a serious thing, and getting the balance between autonomy and safety wrong helps nobody. So this stays speculative until it has run on my own cluster for a while. I’m not asking anyone to trust something I haven’t broken myself first.\nIf it works, my role shifts from operator to architect. I design the policies, the system handles the rest, and adding a customer stops adding a customer’s worth of pager duty.\nWhat it costs A vision that can’t pay for itself is a hobby with extra steps, so here are the numbers.\nIndividuals pay €15-20 a month for cloud storage, a password manager and photos. Families pay €25-35 a month for shared services plus individual accounts. Small businesses pay €50-100 a month for email, collaboration and CI/CD at a quality you’d run a company on.\nCompare that with the going rate. Google Workspace is €5-18 per user per month, paid partly in your data. Microsoft 365 is €5-20 per user per month, same arrangement. Self-hosting it yourself is “free” in euros and costs hundreds of hours.\nI’d rather be honest about the trade-off than sell past it. On a spreadsheet this isn’t the cheapest option. It asks you to pay for something the surveillance economy gives away, and what you get back is sovereignty without the operational burden. For some people that’s obviously worth it. For others it never will be, and that’s a fair call.\nWhere this stands I’m building this in stages, on purpose, slowly enough to break things in private. Right now my own service cluster manages a handful of customer clusters for friends, family and a few early adopters. Next comes documenting the architecture and open-sourcing the management tooling. After that the LLM integration, once I trust it on my own infrastructure. And eventually, something anyone can sign up for and own.\nIf this resonates, whether you want to be an early adopter, help with the tooling or just watch, I’ll keep writing about it here, including the parts that go wrong.\nI said I’d never hand my parents a Raspberry Pi. A box that’s theirs, data that’s theirs, and someone competent keeping it running without ever looking inside, that I would hand them. That’s what I’m building.","permalink":"https://tommeurs.nl/en/posts/self-hosting-for-everyone/","title":"Self-Hosting for Everyone: A Vision for Digital Agency"},{"content":"You wake up. It’s dark. Somewhere a fan is humming.\nYour eyes adjust. You’re in a container. Metal walls, a few small windows, and behind them stars. Just stars. No Earth anywhere.\nThere’s a note taped to the wall:\n“You have everything you need to survive. Nothing goes in, nothing goes out. Good luck.”\nI bring this one up with friends when the evening has gone quiet enough for it. It sounds like science fiction, and the first reaction is usually a joke about the toilet. Then somebody asks what’s in the room, and we’re off. What the experiment takes away is an assumption I’d never said out loud before I ran it. That there’s an “outside” that absorbs my mistakes and refills my supplies.\nLife on autopilot Back on Earth, you don’t think about any of this. You turn the tap and water comes out. You’re hungry, you go to the supermarket. You flush the toilet and the problem goes somewhere you’ve never seen and never have to think about.\nThat’s the comfortable version of living. Every need has a black box behind it, and someone, somewhere, keeps each box running for you. You don’t have to understand a single one of them.\nUntil one of them stops. Then you find out that a system you’ve never looked inside is a system you can’t fix, and you don’t even know who to call.\nThe container takes all those boxes away at once.\nThe moment the boxes disappear In the container you have the same needs you’ve always had. Oxygen to breathe, water to drink, food to eat, somewhere for the waste to go because what goes in also comes out, energy to keep everything running, and temperature regulation because space is cold.\nOn Earth each of these is somebody else’s job. Here, every one of them is yours. There is no “outside” to pull from or dump into. Everything you exhale has to become oxygen again. Every drop of water you drink has to come back. Every scrap of food counts.\nThis is a closed system, and a closed system does something to your brain that an open one never does. It forces you to understand every process you depend on, because your life is now riding on each of them.\nThat’s the part I keep coming back to. The survival mechanics are fun, but the real shift is in how you’re forced to think the moment nothing is hidden from you anymore.\nWorking through it Let’s walk through it, because the realization sets in somewhere in the details.\nOxygen and CO2 You breathe in oxygen, you breathe out CO2. A few hours in, the air goes bad. You have to do something about it, and “call the air company” isn’t on the table.\nThe first thing everyone reaches for is plants. Photosynthesis takes CO2 in and puts oxygen out, which sounds perfect.\nThe catch is the quantity. Estimates vary, but to keep one human breathing you need somewhere between 300 and 700 plants, depending on type and light. And those plants need light, which means energy.\nThe second option is chemical scrubbers, which is what the ISS does. Lithium hydroxide (LiOH) or zeolite pulls CO2 out of the air. Lithium hydroxide runs out, though. Zeolite you can regenerate with heat, which is better for a closed system.\nThe third is the Sabatier reaction, and the ISS runs that too. CO2 plus hydrogen gives methane plus water. The hydrogen comes from electrolysis of water. The methane gets vented to space, which is a deliberate leak in the loop. The water you keep.\nIn a perfectly closed container you’d want to turn that methane into something useful instead of throwing it overboard. Hard, but not impossible.\nWater You lose water constantly: sweat, breath, urine, feces. In a closed system, all of it has to come back.\nThe ISS handles this with the Water Recovery System (WRS), and yes, that’s the part where they drink recycled urine. The Urine Processing Assembly distils and filters it until it’s cleaner than tap water. Humidity from sweat and breathing gets condensed and filtered back into the supply as well.\nTheir recovery rate sits around 93%. The remaining 7% is lost and has to be resupplied from Earth. In your container there is no resupply, so you’d be chasing that last 7% with everything you have.\nFood This is where it gets hard.\nPlants can produce food, but they need light (energy), water, nutrients (nitrogen, phosphorus, potassium, and more), and CO2.\nThe nutrients are the sticking point. In a closed system you have to compost human waste and turn it back into plant food. Biologically that works, composting or vermicomposting with worms, but it costs time and space and patience. Nobody at the table enjoys this part of the conversation.\nThe ISS sidesteps the whole problem by getting food shipped up from Earth. Experiments like MELISSA (Micro-Ecological Life Support System Alternative) from ESA are the ones trying to close the food loop completely.\nThe honest answer is that full self-sufficiency needs a fairly complex ecosystem. Plants on their own won’t do it. You need bacteria, fungi, maybe insects or fish for protein. A whole web rather than a single crop.\nEnergy Solar panels are the obvious call. No clouds in space, so the sun is reliable.\nBut you need storage for when you’re in shadow. Batteries, or maybe flywheels for kinetic storage.\nNuclear is an option too. RTGs (Radioisotope Thermoelectric Generators) power deep-space missions. They also run down, just over decades instead of hours.\nTemperature Direct sun cooks you. Shadow is -270°C.\nThe ISS uses multi-layer insulation blankets, radiators that dump heat into space, and heat pumps that move heat to where it’s needed.\nYou can’t throw heat away without losing it, and in a closed system you’d rather move it around than waste it.\nWhat this does to your thinking Here’s the thing that hooked me the first time I sat with this experiment. It forces you to think in feedback loops instead of straight lines.\nIn an open system you think linearly:\nInput → Process → Output → (somewhere else) In a closed system you think in circles:\nInput\nProcess\nOutput\nRecovery\nEvery output has to become an input for something else. Waste stops being a category, because whatever comes out of one process has to feed the next one. All that’s left is resources sitting in the wrong place.\nThat’s systems thinking in one sentence. You stop staring at individual parts and start looking at how they connect. What flows in, what flows out, and where the loops close or leak.\nAnd it maps almost exactly onto how I think about infrastructure. When I run something myself, I want to know where every resource comes from and where it goes. A managed service is the tap on Earth. Convenient and invisible, and a mystery on the day it breaks. The container is my homelab. Nothing is hidden, everything is mine to understand, and that’s the whole point. I wrote more about why in Sovereign Infrastructure, but the container is the cleanest way I’ve found to explain the feeling.\nBuckminster Fuller’s Spaceship Earth In 1968 Buckminster Fuller wrote Operating Manual for Spaceship Earth. His claim was that Earth already is a spaceship. A closed system with finite resources, drifting through the universe.\n“We are all astronauts on a little spaceship called Earth.”\nThe difference from my container is scale. Earth is big enough that we never feel the walls. We get to pretend “outside” exists, that we can dump waste without consequence and pull resources without running dry.\nThe rules don’t change with size, though. There’s no “away” for plastic in the ocean and no tank of fossil fuels that refills itself. The CO2 we emit stays in the system with us.\nShrinking it down to a container is what makes it land. Suddenly you’re the one breathing that air and drinking that water, and the black box you never looked inside is now the machine keeping you alive.\nWhy I think this way For context on how my brain works, see Working with an AuDHD brain.\nOne thing I’ve noticed about myself is that I see systems whether I want to or not. Hand me a problem and I’m already zooming out. Where does this come from, and where does it go?\nIt can be a lot. Sometimes a person just wants a quick answer and I’m three layers deep in second-order effects. For this kind of thought experiment it fits, though, because the experiment rewards exactly the move my brain makes anyway.\nI keep returning to the container because it’s a safe place to practice that thinking. Nothing real is at stake, and you still walk away with something you can use.\nWhat it leaves you with It’s a philosophical exercise, but the takeaways are concrete.\nWaste is a design flaw If you produce something you can’t reuse, you designed a leak. In the container a leak kills you. On Earth it just takes longer to notice, but the principle holds.\nEfficiency is survival In the container every drop of water is precious and every watt counts. You’d never leave a tap running or a light burning. The only reason we do it on Earth is that the walls are far enough away to ignore.\nComplexity is fragile The more moving parts, the more that can break. In the container you want robust, simple systems with redundancy. It’s the same reason I lean on GitOps and declarative configuration. Simple, predictable systems I can reason about when something goes wrong.\nKeep asking where it goes Most problems start with linear thinking. “I need this, I get it from somewhere, I use it, done.” Circular thinking keeps asking the next question. And then? Where does it go? How does it come back?\nBack to the container So there you are. Stars outside the windows. Everything you need already in the room with you.\nWhat do you do?\nYou start with an inventory. What’s here? Plants, seeds, water, a purification system, solar panels, batteries, tools, a compost bin, maybe an aquaponics setup with fish.\nThen you map the flows: oxygen, water, food, energy, heat. You hunt for the leaks, the places where you’re quietly losing resources. You tighten each loop until it closes.\nSlowly, methodically, you build a system that keeps you alive without anything coming in or going out. By the time you’re done there isn’t a process in that room you don’t understand, because you couldn’t afford to leave one as a black box.\nFuller ran that exercise on the whole planet in 1968. I run it on a container, with whichever friend is in the mood, and the homelab is where I get to do it for real.\nFurther reading:\nOperating Manual for Spaceship Earth, Buckminster Fuller’s original essay ISS Environmental Control and Life Support System, NASA’s technical documentation MELISSA Project, ESA’s research into closed ecosystems Thinking in Systems, Donella Meadows’ classic on systems thinking ","permalink":"https://tommeurs.nl/en/posts/spaceship-thought-experiment/","title":"The space container thought experiment: systems thinking for survival"},{"content":"I write a lot about Kubernetes, and I run it at home every day. I like it.\nSo it might surprise you that I spend a fair amount of my time talking people out of it.\nThe situation I keep walking into goes like this. A small team, one product, a roadmap full of features customers are waiting for, and someone has decided the first milestone is a Kubernetes cluster. Three nodes minimum, etcd, a CNI , an ingress controller, cert-manager, a monitoring stack. Weeks of work before a single customer sees anything. Everyone nods along, because that’s what a serious setup is supposed to look like.\nThere’s another version of that first milestone. You ship in an afternoon, you can name every moving part, and you still sleep fine.\nThis post is for whoever is standing at that decision. I’d like you to pick Kubernetes because you need it, or skip it because you don’t, and know which of the two you did.\nThe Kubernetes hype Kubernetes won. It’s the default answer for container orchestration, every cloud sells a managed flavour, and every DevOps job ad lists it. Once something is the obvious choice, people stop asking whether it’s the right one. Nobody decides on Kubernetes anymore. They assume it.\nI get the appeal. I picked it for my own homelab on purpose and I’ve written plenty about why. But “everyone runs it” is the weakest reason there is to run anything, and it’s a terrible reason to run infrastructure you’ll have to understand when it breaks.\nKubernetes earns its complexity by solving a specific set of problems:\nOrchestrating containers across multiple nodes Automatically scaling based on load Self-healing when containers crash Declarative infrastructure configuration Service discovery and load balancing Read that list and be honest about which ones you have. All five? Then Kubernetes is a gift. None? Then it’s pure cost, and you’re paying in the one thing a small team has the least of, which is attention.\nWhen Kubernetes is overkill You have one application One monolith. One database. Maybe a Redis cache. It runs fine on a single server today.\nWhat is Kubernetes buying you here?\nYour application → Kubernetes cluster → 3+ nodes → etcd → control plane → ingress controller → cert-manager → monitoring stack → ... Versus:\nYour application → Docker Compose → 1 server Same functionality on both sides. The top one costs roughly ten times the time, money and headspace to get to. The bottom one you can hold in your head.\nYour team is small Kubernetes asks you to understand a lot before you can run it safely. The core, and then everything around it:\nNetworking (CNI, service mesh, ingress) Storage (CSI , persistent volumes) Security (RBAC, network policies, pod security) Observability (metrics, logs, traces) GitOps (ArgoCD, Flux ) A team of two or three developers who also carry the pager doesn’t have the spare hours to learn and maintain all of that on top of building the product. Something gives. In my experience it’s the weekends first, and the uptime shortly after.\nYour traffic is predictable The headline feature is autoscaling. If your traffic sits at a steady 100 requests per second with no nasty spikes, autoscaling solves a problem you don’t have. Overprovision a little, put a reminder in for six months from now, move on.\nYour budget is limited Managed Kubernetes (EKS, GKE, AKS) isn’t free, and the line items add up fast:\nControl plane fees (~$70-150/month) Minimum 3 worker nodes for HA Load balancer costs Persistent volume costs Egress costs For one simple app you’re quickly at $300-500 a month. That same app runs happily on a $20 VPS. You’re paying the difference for an abstraction you aren’t using.\nYou don’t have microservices Kubernetes was built for microservices, many small independent services that scale and fail on their own schedules. A monolith doesn’t have those needs. Running one on Kubernetes is buying a Ferrari for the weekly grocery run. It works, but every feature you’re paying for sits idle in the driveway.\nThe hidden costs of Kubernetes The sticker price is the easy part. The costs that hurt are the ones that never show up on an invoice.\nOperational overhead Even on managed Kubernetes, the cluster is a living thing you tend forever:\nCluster upgrades (new version every 3-4 months) Node pool management Capacity planning Incident response when pods don’t start Debugging networking issues That’s steady, recurring work, and someone on the team ends up owning it whether you planned for that or not.\nLearning curve Here’s the vocabulary you need before you can reason about a running cluster:\nPods, Deployments, StatefulSets, DaemonSets Services, Ingress, NetworkPolicies ConfigMaps, Secrets PersistentVolumes, StorageClasses RBAC, ServiceAccounts Helm , Kustomize CRDs, Operators And that’s before the ecosystem tools every cluster grows sooner or later. Prometheus, Grafana, ArgoCD, cert-manager, external-dns, and a dozen more. Each one is another thing to learn, patch, and understand when it misbehaves.\nYAML hell apiVersion: apps/v1 kind: Deployment metadata: name: my-app labels: app: my-app spec: replicas: 3 selector: matchLabels: app: my-app template: metadata: labels: app: my-app spec: containers: - name: my-app image: my-app:v1.0.0 ports: - containerPort: 8080 resources: requests: memory: \"128Mi\" cpu: \"100m\" limits: memory: \"256Mi\" cpu: \"500m\" livenessProbe: httpGet: path: /health port: 8080 initialDelaySeconds: 10 periodSeconds: 5 readinessProbe: httpGet: path: /ready port: 8080 initialDelaySeconds: 5 periodSeconds: 3 --- apiVersion: v1 kind: Service metadata: name: my-app spec: selector: app: my-app ports: - port: 80 targetPort: 8080 --- apiVersion: networking.k8s.io/v1 kind: Ingress # ... even more YAML That’s the minimum to get one application running. Here’s the same intent in Docker Compose:\nservices: my-app: image: my-app:v1.0.0 ports: - \"80:8080\" healthcheck: test: [\"CMD\", \"curl\", \"-f\", \"http://localhost:8080/health\"] Debugging complexity Container won’t start? On Docker the whole investigation is one line:\ndocker logs my-app On Kubernetes the same question turns into a small expedition:\nkubectl describe pod my-app-7d9f8b6c5d-x2k9p kubectl logs my-app-7d9f8b6c5d-x2k9p kubectl logs my-app-7d9f8b6c5d-x2k9p --previous kubectl get events --field-selector involvedObject.name=my-app-7d9f8b6c5d-x2k9p And the answer turns out to be an ImagePullBackOff from a typo in the image name. Or a resource limit. Or a full node. Every layer Kubernetes adds is a layer you walk through when you’re trying to work out why nothing happened. On a good day you don’t notice. On a bad day, four kubectl commands in, you do.\nAlternatives to Kubernetes So if not Kubernetes, then what? None of these are toys. Each one is a tool I’d reach for in the right situation, and several of them carry real production traffic for real companies right now.\nDocker Compose One server, multiple containers, a simple setup.\nservices: app: image: my-app:latest ports: - \"80:8080\" depends_on: - db - redis environment: - DATABASE_URL=postgres://db:5432/app restart: unless-stopped db: image: postgres:15 volumes: - postgres_data:/var/lib/postgresql/data restart: unless-stopped redis: image: redis:7 restart: unless-stopped volumes: postgres_data: docker compose up -d and you’re running. Updates are docker compose pull \u0026\u0026 docker compose up -d. The whole mental model fits in one file you can read top to bottom.\nWhat you get is something anyone on the team can understand, no cluster overhead, and a setup that works anywhere Docker runs. Developers love it. What you give up is automatic scaling, high availability, since it’s one node, and most of what the word orchestration means.\nDocker Swarm You need multiple nodes, and Kubernetes is too much.\nDocker Swarm is the forgotten sibling. Simpler, fewer features, and often more than enough.\n# Init swarm docker swarm init # Deploy stack docker stack deploy -c docker-compose.yml myapp # Scale docker service scale myapp_web=5 It takes the same Compose files, it’s built into Docker, setup is easy and it spans nodes. The ecosystem is small, the community is smaller still since Docker put its weight behind Kubernetes, and the feature list is shorter.\nHashiCorp Nomad You want orchestration without the Kubernetes complexity, or you run more than just containers.\nNomad is a lightweight orchestrator for containers, VMs and standalone executables, from a single binary. If you run more than containers, that breadth is worth a serious look.\njob \"my-app\" { datacenters = [\"dc1\"] group \"web\" { count = 3 task \"app\" { driver = \"docker\" config { image = \"my-app:latest\" ports = [\"http\"] } resources { cpu = 500 memory = 256 } } } } Simpler than Kubernetes, and it schedules more than containers (raw exec, Java, QEMU). It fits well with the rest of the HashiCorp tools, Consul and Vault, and a single binary is easy to deploy. The trade is a smaller ecosystem, fewer managed cloud options, and less of a standard, so fewer tutorials and fewer people who’ve solved your problem before.\nPlatform-as-a-Service You want to deploy code, and you don’t want to run infrastructure.\nRender does simple deploys with a managed PostgreSQL. Railway is developer-friendly with a good free tier. Fly.io deploys to the edge and performs well. Heroku is the original, and still solid.\n# Fly.io example fly launch fly deploy No YAML, no cluster, no ops. The trade is real, and I won’t pretend otherwise. You’re handing control to someone else’s platform, which cuts straight against the sovereignty I usually argue for. For a side project or an early product that needs to ship, that can be a fair deal. Read the pricing page and the terms before you build on it, because the platform can change both and you can’t.\nWhat you get is zero infrastructure management, push code and it runs, and often a lower bill for a small project. What you give up is the exit, the control, and a predictable bill once you grow.\nPlain VMs You don’t need containers.\nI mean it. Not everything has to be a container. A boring server with a service manager has kept things online for decades, and it’s about as inspectable as infrastructure gets.\nA Python app with systemd:\n[Unit] Description=My App After=network.target [Service] User=app WorkingDirectory=/opt/myapp ExecStart=/opt/myapp/venv/bin/python app.py Restart=always [Install] WantedBy=multi-user.target Ansible for configuration management. No Docker, no Kubernetes.\nSimple, no container overhead, familiar to most engineers, and easy to debug. What you lose is the isolation containers give you, dependency management gets harder, and the box is less reproducible.\nServerless / FaaS Event-driven workloads, variable load, and nobody wants to manage servers.\nAWS Lambda, Google Cloud Functions, Azure Functions, Cloudflare Workers.\n// AWS Lambda export const handler = async (event) =\u003e { return { statusCode: 200, body: JSON.stringify({ message: \"Hello!\" }) }; }; You pay per execution, it scales to zero on its own, and there are no servers to manage. The other side is cold starts, vendor lock-in, a poor fit for long-running processes, and debugging that’s harder than it should be.\nDecision framework So how do you choose, instead of defaulting? Five questions get me most of the way there.\nHow many services One to three, and Docker Compose or a PaaS will do. Four to ten, and Swarm, Nomad or Kubernetes all fit. Past ten, Kubernetes starts earning its place.\nWhether the load moves If it doesn’t, almost anything except Kubernetes. If it moves on a schedule, scheduled scaling covers it and you don’t need K8s for that. If it moves and you can’t predict it, Kubernetes or serverless.\nWhat the team knows No Kubernetes experience? Don’t start with Kubernetes. Some? Managed Kubernetes (EKS, GKE). Lots? Run whatever you want.\nWhat the budget is Under $100 a month, a VPS, a PaaS or serverless. Between $100 and $1000, it depends on the requirements. Over $1000, Kubernetes starts to make economic sense.\nWhether it has to move between clouds If not, use what the cloud gives you. If it does, Kubernetes is the best option for portability that I know of.\nWhen Kubernetes is the right call I’ve spent this whole post arguing for restraint, so let me be just as clear about the other side. When you have those problems, Kubernetes is the best tool I know, and I reach for it without hesitation.\nMany microservices that scale independently. A team with Kubernetes expertise that can maintain it. Deployment requirements that go past a rolling restart, canary and blue-green. A multi-tenant platform for more than one team. Portability between clouds, which is the reason I run it in my own homelab. The same manifests move from a refurbished mini PC to a cloud and back, and that freedom is worth real complexity to me. And an ecosystem, Prometheus, ArgoCD and friends, that you’ll use rather than just install.\nIf a few of those describe you, Kubernetes is probably the right call, and everything before this section was about getting you here on purpose.\nWhy we pick it anyway So why do teams keep reaching for Kubernetes when something smaller would do? Because Kubernetes is the safe answer in a meeting. Nobody gets second-guessed for choosing the industry standard. It looks good on the architecture diagram and good on a CV. Choosing Docker Compose for a production app feels like admitting you’re not serious, even when it’s the smarter engineering call.\nI think that’s backwards. Picking the heaviest tool to look credible is the opposite of understanding your own system. What’s credible is matching the tool to the problem and being able to explain why.\nMy rule of thumb Start simple. Add complexity the day you need it, and no sooner.\nDocker Compose for development and small production workloads. Kubernetes when you have the problems Kubernetes solves, and not a sprint earlier.\nA team that can say “we don’t need Kubernetes” out loud, and say why, understands its own system. That’s the team I’d want to be on.\nWrapping up Before you adopt Kubernetes, sit with three questions. What problem am I solving? Is Kubernetes the simplest solution to it? Do I have the time, the money and the expertise to keep it healthy?\nA “no” to any of those is a nudge to look at the alternatives. Docker Compose, Nomad, a PaaS and plain VMs all exist because each fits a shape of problem better than Kubernetes does.\nYou don’t have to decide for the whole company at once. Put the next internal tool on a Compose file and a VPS, and see how far it gets before anyone misses the cluster.\nI run Kubernetes at home because I have the problems it solves, and because I like it. Most of the teams I talk out of it don’t have those problems yet. When they do, I’ll help them build the cluster.","permalink":"https://tommeurs.nl/en/posts/when-not-to-use-kubernetes/","title":"When not to use Kubernetes"},{"content":" TL;DR One YubiKey carries a GPG key with a signing, an encryption and an authentication subkey. gpg-agent hands the authentication subkey to SSH, git signs with the signing subkey and pass decrypts with the encryption subkey, so one PIN in the morning and a touch per action replace every password I used to type. The SSH part alone is three steps and worth doing on its own.\nEvery morning I plug in my YubiKey, type one PIN, and that’s the last secret I type until I pull it out again. SSH to a server, no password. Sign a commit, no passphrase. Pull a secret out of pass, touch the key. One thing in a USB port, and the rest of the day my authentication happens without me.\nGetting there took three evenings of swearing at gpg-agent. I’d do it again.\nWhy I bothered Before the key I had four ways of proving who I was, and they all worked differently. SSH keys on disk, encrypted, sure, but still a file anyone with my home directory could copy. A GPG key for git signing, with its own passphrase. pass, which used that same GPG key and asked for that same passphrase again. And 2FA codes in an app on my phone, so grab the phone, squint, retype.\nEvery action started with typing a password or reaching for my phone. That’s exactly where I start cutting security corners, and I know that about myself. Give me an annoying setup and I’ll have worked around it within a week.\nSo I wanted one key that does all of it, with the private keys on hardware I can pull out of the laptop and put in my pocket, instead of in files spread across my disk. That’s the sovereignty part. I want to know exactly where my private keys are, and I want the answer to be “in my pocket”.\nOne YubiKey does all of it now.\nThe stack Tool Function YubiKey Hardware security key with smartcard functionality GPG Encryption and signing pass CLI password manager (uses GPG) gpg-agent Agent that caches GPG keys and handles SSH auth SSH You know what SSH is The piece that makes it hang together is gpg-agent. It can act as an SSH agent, which means the authentication subkey on the YubiKey doubles as your SSH key. So the same key signs commits, decrypts passwords and logs you into servers.\nSSH alone, in three steps You don’t need the whole setup to get something out of this. Generate a GPG key with an authentication subkey, move it onto the YubiKey, and point SSH at gpg-agent. That’s three steps, and you’re logging into servers with a hardware key. pass, git signing, touch policies and backups come after.\nThe full setup Put the GPG key on the YubiKey You need a GPG key with three subkeys: signing for git commits, encryption for pass, authentication for SSH. If you already have a GPG key, add the subkeys to it. Otherwise create a new one.\n# Generate new key (or use existing) gpg --full-generate-key # Add subkeys gpg --edit-key \u003cKEY_ID\u003e gpg\u003e addkey # Sign gpg\u003e addkey # Encrypt gpg\u003e addkey # Authenticate (choose \"set your own capabilities\") gpg\u003e save Then move them to the card:\ngpg --edit-key \u003cKEY_ID\u003e gpg\u003e key 1 gpg\u003e keytocard # Choose slot for signing gpg\u003e key 1 gpg\u003e key 2 gpg\u003e keytocard # Choose slot for encryption gpg\u003e key 2 gpg\u003e key 3 gpg\u003e keytocard # Choose slot for authentication gpg\u003e save keytocard moves the key, it doesn’t copy it. Back up your .gnupg directory first. I didn’t, on YubiKey number one, and that’s how I learned the difference.\nConfigure gpg-agent This is the step that does the heavy lifting. Edit ~/.gnupg/gpg-agent.conf:\n# Cache timeout (in seconds) default-cache-ttl 600 max-cache-ttl 7200 # Enable SSH support enable-ssh-support # Pinentry program (for PIN entry) pinentry-program /usr/bin/pinentry-gnome3 # Or for terminal: /usr/bin/pinentry-curses # Or for macOS: /usr/local/bin/pinentry-mac And in your ~/.bashrc or ~/.zshrc:\n# Use GPG agent as SSH agent export GPG_TTY=$(tty) export SSH_AUTH_SOCK=$(gpgconf --list-dirs agent-ssh-socket) # Restart agent if needed gpgconf --launch gpg-agent Restart your shell or source the file:\nsource ~/.zshrc Export the SSH key The authentication subkey is now also your SSH key. Export the public half:\n# SSH public key from your GPG key gpg --export-ssh-key \u003cKEY_ID\u003e Add it to ~/.ssh/authorized_keys on your servers, or to GitHub/GitLab. Then check that the agent sees it:\nssh-add -L # Shows your GPG key as SSH key pass and git on top The SSH part is the core. What follows is a password manager, signed commits, and the touch policy that decides how paranoid you want the key to be.\nConfigure pass pass is about as simple as a password manager gets. Every password is one GPG-encrypted file on disk, and since the GPG key now lives on the YubiKey, reading a password means touching the key.\n# Initialize pass with your GPG key pass init \u003cKEY_ID\u003e # Or with multiple keys (handy for backup) pass init \u003cKEY_ID_1\u003e \u003cKEY_ID_2\u003e Adding passwords:\n# New password pass insert servers/prod-db # Generate random password pass generate -n 32 servers/api-key # View password (asks for YubiKey touch) pass servers/prod-db Sign git commits git config --global user.signingkey \u003cKEY_ID\u003e git config --global commit.gpgsign true git config --global tag.gpgsign true Every commit now asks for a touch. No extra password, a tap on the key, and the commit provably came from the hardware in my USB port rather than from a process that found a key on disk.\nA day with the key in Once it runs, a day looks like this:\n# Morning: plug in YubiKey, type PIN once gpg --card-status # Triggers PIN prompt # SSH to server - no password ssh prod-server # Get a password - touch the key pass show infra/grafana | head -1 # Git commit - touch the key git commit -m \"Fix critical bug\" # Everything works, YubiKey stays in USB After that first PIN everything keeps working until the cache expires, 600 seconds for the PIN with the config above. The touch stays required for as long as the policy is on, so the cache never makes the key silent.\nAliases I use # Quick pass lookup with fzf alias p='pass show $(find ~/.password-store -name \"*.gpg\" | sed \"s|$HOME/.password-store/||;s|\\.gpg$||\" | fzf)' # Copy password to clipboard alias pc='pass show -c $(find ~/.password-store -name \"*.gpg\" | sed \"s|$HOME/.password-store/||;s|\\.gpg$||\" | fzf)' # YubiKey status alias yk='gpg --card-status' # Show SSH keys alias sshkeys='ssh-add -L' Touch policy This is where you decide how much the key bugs you. There are a few policies:\nPolicy Meaning Off No touch needed On Touch for every operation Fixed Touch required, cannot be changed Cached Touch valid for X seconds I run “On” for signing and authentication, so every commit and every SSH login needs a physical touch. That’s deliberate. If the key blinks and I didn’t ask for anything, something is using it that shouldn’t be. That’s worth a second per commit to me.\n# Set touch policy via ykman ykman openpgp keys set-touch sig on ykman openpgp keys set-touch aut on ykman openpgp keys set-touch enc on When the key is gone A YubiKey can snap, go through the wash, or leave in a coat you donated. So I plan for the day it’s gone. That means a second YubiKey with the same keys on it, stored somewhere else. A printed copy of the master key, offline. And a revocation certificate, generated now and stored apart from both.\n# Generate revocation certificate gpg --gen-revoke \u003cKEY_ID\u003e \u003e revoke.asc # Store this safely, separate from your key backup pass itself you can sync with git:\ncd ~/.password-store git init git remote add origin git@github.com:user/pass-store-private.git git push -u origin main The passwords are encrypted files, so a leaked repo is useless without the GPG key.\nTroubleshooting “No secret key” # Check if YubiKey is visible gpg --card-status # Not visible? Check USB lsusb | grep -i yubi # GPG doesn't know where the key is? Refresh gpg-connect-agent \"scd serialno\" \"learn --force\" /bye SSH doesn’t work # Check if gpg-agent is running as SSH agent echo $SSH_AUTH_SOCK # Should be something like: /run/user/1000/gnupg/S.gpg-agent.ssh # Load keys ssh-add -L # Should show your GPG key # Not working? Restart agent gpgconf --kill gpg-agent gpgconf --launch gpg-agent PIN blocked After three wrong PINs the YubiKey locks. Reset it with the Admin PIN:\ngpg --edit-card gpg/card\u003e admin gpg/card\u003e passwd # Choose option 2: unblock PIN Forgot the Admin PIN too? Then the YubiKey is a brick for those keys, and the second YubiKey from two sections up is what saves you.\nWhy this one stuck The usual story is that security and convenience pull in opposite directions, and more of one costs you some of the other. For me this setup runs the other way. I type one PIN per session and touch the key after that. There’s no reusable password to phish, because the physical key has to be in the port. SSH, signing and pass are one workflow instead of three, with one mental model. And it works without internet, without a cloud account, without a vendor that has to stay online.\nThe part that matters most is that I use it. This one doesn’t fight me, so I’ve never worked around it. Most days I don’t think about it at all.\nWhere to start You don’t have to configure all of this in one sitting. Buy a YubiKey 5, the NFC variant if you want it to work with your phone. Generate a GPG key with an authentication subkey, put it on the key, configure gpg-agent for SSH and test with ssh-add -L. pass, git signing and the touch policy go on top at whatever pace suits you.\nWhat it cost Three evenings, most of it on gpg-agent. You have to understand what GPG subkeys are, get gpg-agent to behave, and keep the picture of how the pieces fit in your head while you do it. That’s real time, and I swore a lot.\nWhat I got for it is one piece of hardware at the centre of every login I do. My private keys are on something I can pocket, and using them is a tap. If you want to know whether yours still works, one line covers all three:\n# Check if everything works gpg --card-status \u0026\u0026 ssh-add -L \u0026\u0026 pass show test/dummy If that runs without errors, you’re done. Mine took three evenings. You have this post.","permalink":"https://tommeurs.nl/en/posts/yubikey-pass-gpg-ssh/","title":"YubiKey + Pass + GPG + SSH: One Key to Rule Them All"},{"content":" TL;DR pass keeps each password as a GPG-encrypted file in a directory, and the directory is the whole data model. pass -c email/gmail puts a password in your clipboard from any terminal, pass git push backs the store up, and the fzf script further down is how I look up every password I have. The price is GPG key management, and it’s a real price.\nI used KeePass for years. Then 1Password. Then Bitwarden. All decent tools, and every one of them was too much for what I asked of it. Too much UI, too many features, and a password always lived on the other side of an app I had to alt-tab into. A small interruption, and they added up.\nThen I found pass. It stores passwords. That’s the feature list. I’ll walk you through it the way I learned it, from the one command that hooked me to the setup I run today.\nFiles in a directory, encrypted Pass calls itself the “standard unix password manager.” Under the hood it’s a shell script of roughly 700 lines that stores each password as a GPG-encrypted file in a directory. No database. No proprietary format. No cloud sync bolted on. It’s small enough to read in an afternoon, and that matters to me. When something breaks, I want to be able to find out why.\n~/.password-store/ ├── email/ │ ├── gmail.gpg │ └── protonmail.gpg ├── social/ │ └── github.gpg └── work/ ├── gitlab.gpg └── jira.gpg Each .gpg file is an encrypted text file holding a password and, if you want, some extra lines. The directory tree is your organisation, and there’s nothing else to learn.\nWorking in five minutes Install it # macOS brew install pass gnupg # Debian/Ubuntu sudo apt install pass gnupg # Arch sudo pacman -S pass gnupg A GPG key Pass encrypts with GPG, so you need a key. If you don’t have one yet:\ngpg --full-generate-key Choose RSA, 4096 bits, and fill in your name and email. Remember the passphrase. You’ll type it every time you decrypt a password, and nobody can recover it for you.\nInitialize the store # Use your GPG key ID or email pass init \"tom@example.com\" This creates ~/.password-store/ with a .gpg-id file that records which GPG key is in use.\nYour first password # Generate a new password pass generate email/gmail 24 # Or add an existing password pass insert social/github That’s the whole setup. You have a working password manager.\nGet it back out # Show password in terminal pass email/gmail # Copy to clipboard (disappears after 45 sec) pass -c email/gmail This was the moment it clicked for me. No app to launch, no extension to wake up. One command and the password is in my clipboard.\nWhy it fits a terminal-first day I live in a terminal, and pass never asks me to leave it. When I need a work password:\npass -c work/gitlab Done. It sits in my clipboard for thirty seconds and then it’s gone.\nBecause the store is plain files in a directory, a few things come for free. It’s searchable, with pass find gitlab or plain ls ~/.password-store/. It’s scriptable, so pass show -c $(pass find jira | head -1) works. And it plugs into anything that can run a command: dmenu, rofi, fzf, Alfred, your own scripts.\nA .gpg file can hold more than a password. The convention is that the first line is the password and everything after it is metadata.\npass edit email/gmail SuperSecurePassword123! --- username: tom@gmail.com url: https://mail.google.com otp: otpauth://totp/Gmail:tom@gmail.com?secret=JBSWY3DPEHPK3PXP notes: recovery email is backup@proton.me You can pull TOTP (2FA) codes straight out of it too:\n# With pass-otp extension pass otp email/gmail Git sync, the part that sealed it Pass has git built in, and this is why I stopped looking at other password managers.\n# Initialize git repo pass git init # Every change is automatically committed pass generate new/password 20 # → [master 1a2b3c4] Add generated password for new/password. # Push to remote pass git push Your password store is now a git repository. That gives you automatic backups to your remote, sync between machines with a plain pass git pull, a version history of every password, and merge conflicts when you change the same one on two machines. I’ll get back to that last one.\nMine lives in a private repo on my own Forgejo. A fresh machine is one clone:\ngit clone git@forgejo.example.com:tom/password-store.git ~/.password-store Every password, back. No vendor in the middle of that pull.\nBrowser integration “But I want auto-fill in my browser.”\nYou can have it. There are extensions for Firefox and Chrome:\nBrowserpass - native messaging, works well passff - Firefox-only, also solid Browserpass reads your store, matches the URL against your directory names, and fills the form. It works better than I expected.\n~/.password-store/ ├── github.com/ │ └── tom.gpg # auto-matched on github.com ├── gitlab.com/ │ └── work.gpg └── mail.google.com/ └── personal.gpg Fuzzy finding with fzf This is what I use every day. A small script:\n#!/bin/bash # passfzf - fuzzy find and copy password password=$(find ~/.password-store -name \"*.gpg\" | \\ sed 's|.*/\\.password-store/||; s|\\.gpg$||' | \\ fzf --height 40% --reverse) [[ -n \"$password\" ]] \u0026\u0026 pass -c \"$password\" Bind it to a hotkey and you have fuzzy lookup in one keystroke. If you live in rofi instead, rofi-pass does the same job:\nrofi-pass Teams and extensions Multi-key encryption Sharing passwords with a team? Pass takes more than one GPG key:\npass init key1@example.com key2@example.com Now both keys can decrypt everything. You can also scope keys per directory:\n# Only you have access to personal/ echo \"your-key-id\" \u003e ~/.password-store/personal/.gpg-id # The whole team has access to shared/ echo -e \"key1\\nkey2\\nkey3\" \u003e ~/.password-store/shared/.gpg-id pass init Three extensions I use Pass has an extension system. pass-otp adds TOTP and HOTP:\n# Add OTP pass otp insert email/gmail # Paste the otpauth:// URL # Generate code pass otp email/gmail # → 482193 pass-tomb wraps the whole store in a Tomb container, for the case where someone has physical access to your machine:\npass tomb open # do your thing pass tomb close pass-update rotates passwords in bulk:\npass update -l 32 work/* The downside is GPG Pass is simple because it leans on GPG, and GPG key management is anything but. That’s the bill for a password store that’s yours, and I’d rather hand it to you now than have you find it later.\nLose the key, lose everything Lose your GPG private key and every password goes with it. Permanently. No “forgot password” link, no support desk to dig you out.\nSo back up the key in more than one place. Remember the passphrase, and writing it on paper in a safe is a reasonable move. And consider a hardware key like a YubiKey. I do all three, and my setup is at the bottom.\nGPG will fight you GPG is powerful and famously unfriendly. You’ll meet errors like these:\ngpg: decryption failed: No secret key gpg: problem with the agent: No pinentry gpg: public key not found If you haven’t used GPG before, the learning curve is steep. Pinentry popups that refuse to appear, agents that hang, keys that go missing. I’ve lost evenings to it, and I’m telling you up front.\nNo official mobile app There’s no official pass app. Third-party ones exist. On Android, Password Store works well and syncs via git. On iOS, Pass for iOS does git sync too. Neither is as polished as 1Password or Bitwarden on a phone. You configure git credentials and import your GPG key first, and that takes time.\nNo password sharing via link “Share this password with a colleague” is not a feature. Multi-key encryption works, but every person on the team needs a GPG key. For a quick share with someone non-technical, pass is the wrong tool.\nMerge conflicts in encrypted files Git sync is great until you change the same password on two machines and get a merge conflict in a file git can’t read. Resolving that is no fun. It rarely happens if you run pass git pull before you change anything, but it can.\nWhen pass is the right call You’re comfortable in the terminal. You already use GPG, or you’re willing to learn it. You want your data under your own control, with no cloud unless you pick one. You want to wire passwords into scripts. And you want something small enough to audit, roughly 700 lines of bash. If that’s you, pass fits.\nWhen to choose something else You share passwords with family or colleagues who don’t have GPG. You want a polished phone app with zero setup. You’d rather not deal with GPG key management at all. Or your organisation mandates a password manager for compliance. Then Bitwarden or 1Password, and no shame in it.\nMy setup What I run today:\n# ~/.zshrc export PASSWORD_STORE_DIR=\"$HOME/.password-store\" export PASSWORD_STORE_CLIP_TIME=30 # Fuzzy find alias alias p='passfzf' # Quick copy alias pc='pass -c' The store lives in a private repo on my own Forgejo, synced between laptop, desktop and phone with git. The GPG key sits on a YubiKey, with a backup on an air-gapped machine.\nIs it more setup than Bitwarden? Yes. It buys me a workflow that never leaves the terminal, my data on my own disk, and no vendor who can raise the price or sell the company next quarter.\nAfter three password managers Pass suits people who like the Unix philosophy, one thing done well and wired into everything else. The GPG dependency alone rules it out for plenty of people, and I’d rather say so than sell it to the wrong reader.\nFor me it was a relief after every overengineered manager before it. Roughly 700 lines of bash, files in a directory, git for sync. Three password managers with a UI, then a shell script. I’m still on the shell script.\nFurther reading:\npasswordstore.org - official site pass git repo - source code Browserpass - browser extension Password Store (Android) - Android app ","permalink":"https://tommeurs.nl/en/posts/pass-password-manager/","title":"Pass: the Unix password manager that just works"},{"content":" TL;DR GPG encrypts files and signs things, and it sits under pass, under signed git commits and under the checksums maintainers publish. One command gives you a working keypair in five minutes. Subkeys and a YubiKey come later, if ever, and the UX is bad enough that I complain about it at length below.\nGPG is a tool I kept meaning to learn and didn’t, for years. The docs are a wall of text, the terminology is its own dialect, and the error messages are cryptic in both senses of the word.\nYou keep bumping into it anyway. GPG sits under pass, under signed git commits, under encrypted email, under checking that the binary you just downloaded is the one the maintainer shipped. If you’d rather own your security than hand it to a vendor, you end up here sooner or later.\nSo here’s the guide I wish someone had handed me. It starts with the least that does something, and you can stop reading the moment you have enough.\nWhat GPG does GPG (GNU Privacy Guard) implements the OpenPGP protocol, and under the jargon it has two jobs. It encrypts a file so only the intended recipient can read it, and it signs something so other people can check it came from you.\nBoth run on asymmetric cryptography. You have a public key you hand out freely and a private key that never leaves your control. Someone who wants to send you a secret encrypts it with your public key, and only your private key opens it. When you want to prove something is yours, you sign it with your private key and anyone can check it against the public one.\nEverything below is flags and file formats on top of those two ideas. If the rest gets confusing, come back to this paragraph.\nYour first keypair in 5 minutes One command, a handful of questions, and you have working crypto. No subkeys, no hardware, no theory. Get this working first; everything after it is optional.\nInstallation # macOS brew install gnupg # Debian/Ubuntu sudo apt install gnupg # Arch sudo pacman -S gnupg Generate keypair gpg --full-generate-key It’ll ask you a few things.\nPlease select what kind of key you want: (1) RSA and RSA (9) ECC (sign and encrypt) Pick (1) RSA and RSA for the widest compatibility, or (9) ECC for modern, shorter keys. When in doubt, RSA. Everything understands it.\nWhat keysize do you want? (3072) 4096 for RSA. Bigger is fine here, within reason.\nKey is valid for? (0) 2y, so 2 years. You can extend it later. A key with no expiry feels convenient right up to the day it leaks, because then it stays valid forever and nothing limits the damage. An expiry date is a failsafe you set once.\nReal name: Tom Meurs Email address: tom@example.com Comment: Your name and email. Leave the comment empty.\nEnter passphrase: Pick a strong passphrase. If someone ever gets hold of your private key file, this passphrase is all that stands between them and your identity.\nDone. You have a working GPG keypair, and that was the hard part of getting started.\nView keys # List your public keys gpg --list-keys # List your private keys gpg --list-secret-keys Output looks like this:\npub rsa4096 2026-01-26 [SC] [expires: 2028-01-26] ABC123DEF456GHI789JKL012MNO345PQR678STU9 uid [ultimate] Tom Meurs \u003ctom@example.com\u003e sub rsa4096 2026-01-26 [E] [expires: 2028-01-26] That long string is your key ID, the fingerprint. People often use the short version, the last 16 characters: MNO345PQR678STU9.\nEncrypting a file You have a key, so let’s use it. Encryption is the part you’ll feel first, so it comes first.\nEncrypt for yourself # Encrypt a file gpg --encrypt --recipient tom@example.com secret.txt # Creates secret.txt.gpg # Decrypt gpg --decrypt secret.txt.gpg \u003e secret.txt Encrypt for someone else First you need their public key:\n# Import a public key gpg --import colleague-key.asc # Or fetch from a keyserver gpg --keyserver keyserver.ubuntu.com --recv-keys THEIR_KEY_ID Then encrypt:\ngpg --encrypt --recipient colleague@example.com secret.txt Now only they can open it. You never had to agree on a shared password over some side channel, and that’s what public key crypto is for.\nSymmetric encryption with a password Sometimes you don’t want keys at all, just a password:\ngpg --symmetric secret.txt # Asks for password, creates secret.txt.gpg Handy for a quick one-off. It’s weaker than the asymmetric route, because everything rests on one password you have to remember and somehow share, so I use it only when keys would be overkill.\nProving something came from you Sign a file # Create a detached signature gpg --detach-sign document.pdf # Creates document.pdf.sig # Or embed the signature gpg --sign document.pdf # Creates document.pdf.gpg (signed + compressed) # Or clearsign (readable with signature embedded) gpg --clearsign message.txt # Creates message.txt.asc Verify signature gpg --verify document.pdf.sig document.pdf Output:\ngpg: Signature made Mon Jan 26 12:00:00 2026 CET gpg: using RSA key ABC123... gpg: Good signature from \"Tom Meurs \u003ctom@example.com\u003e\" If you ever see “BAD signature”, the file changed after it was signed or the signature belongs to another key. Either way, don’t trust it.\nSigning git commits If you only ever use GPG for one thing, make it this one. It’s the bit of GPG I touch every day without noticing.\nConfigure git # Tell git which key to use git config --global user.signingkey YOUR_KEY_ID # Automatically sign all commits git config --global commit.gpgsign true # Automatically sign all tags git config --global tag.gpgsign true Create signed commit git commit -S -m \"This commit is signed\" GitHub and GitLab put a “Verified” badge next to signed commits. The badge tells a reviewer the commit came from you and wasn’t forged by someone who set user.email to your address, which anyone can do.\nAdd your public key to GitHub # Export your public key gpg --armor --export tom@example.com Copy the output, including the -----BEGIN PGP PUBLIC KEY BLOCK----- line, and paste it into GitHub under Settings, then SSH and GPG keys, then New GPG key.\nSharing your public key A signature is useless to anyone who doesn’t have your public key, so it has to get out into the world. There are a few ways, from “email it” to “put it on a keyserver”.\nExport # As ASCII (for email, websites) gpg --armor --export tom@example.com \u003e tom-public.asc # As binary gpg --export tom@example.com \u003e tom-public.gpg Upload to a keyserver gpg --keyserver keyserver.ubuntu.com --send-keys YOUR_KEY_ID Now anyone can find your key with:\ngpg --keyserver keyserver.ubuntu.com --search-keys tom@example.com Put it on your website Plenty of people host their public key at example.com/.well-known/openpgpkey/ or just example.com/pgp.asc. Hosting it yourself means people fetch it from infrastructure you run rather than a keyserver somebody else runs.\nSubkeys, so the master key can go offline Everything up to here works with the one keypair you generated at the start, and for getting going that’s fine. If you’re going to lean on GPG for years, subkeys are what make that bearable, so read this even if you set it up later.\nGPG supports subkeys, separate keys that hang under your master key.\nWhy subkeys Your master key is your identity. Lose control of it and you’ve lost the thing that proves you’re you, with no clean way back. Subkeys keep that key out of harm’s way. The master key lives offline, in cold storage. Subkeys do the day-to-day work. If a subkey leaks, you revoke that one subkey and carry on, and the master key, your identity, stays untouched.\nIt’s the same principle as keeping backups offline or trimming a service account’s permissions. Shrink the blast radius, so one slip doesn’t take everything with it.\nSubkey structure A typical setup:\nMaster key [C] (certify) - keep offline ├── Subkey [S] (signing) - daily use ├── Subkey [E] (encryption) - daily use └── Subkey [A] (authentication) - SSH login Create subkeys gpg --edit-key tom@example.com In the GPG prompt:\ngpg\u003e addkey Please select what kind of key you want: (4) RSA (sign only) (6) RSA (encrypt only) # Create a sign-only subkey # Create an encrypt-only subkey gpg\u003e save Take the master key offline This step takes a bit of nerve, because you’re about to delete secret key material from your daily machine on purpose:\n# Export everything (backup!) gpg --armor --export-secret-keys tom@example.com \u003e master-backup.asc # Export only subkeys gpg --armor --export-secret-subkeys tom@example.com \u003e subkeys.asc # Delete secret keys from your machine gpg --delete-secret-keys tom@example.com # Import only the subkeys gpg --import subkeys.asc Your daily machine now carries subkeys only. The master key sits on a USB stick in a drawer, or wherever you trust. If the laptop gets stolen or owned, the attacker gets subkeys you can revoke, not the key that defines you.\ngpg --list-secret-keys # sec# rsa4096 ... ← that # means the master key isn't present Moving the subkeys to a YubiKey Most people never need this, and I won’t push you there. But if you want your private keys to physically refuse to leave the device they live on, a hardware token is how.\nWhat the hardware buys you The keys never leave the hardware, full stop. Malware on your laptop can’t copy what it can’t read. And an attacker needs the physical YubiKey in hand, not just a shell on your machine.\nMove keys to YubiKey gpg --edit-key tom@example.com gpg\u003e key 1 # select subkey gpg\u003e keytocard Please select where to store the key: (1) Signature key (3) Authentication key gpg\u003e key 2 # select next subkey gpg\u003e keytocard gpg\u003e save From then on, signing or decrypting means plugging in the YubiKey and tapping it. A small pause every time, in exchange for keys nobody can copy off your disk.\nGPG + SSH with YubiKey There’s a bonus. Your GPG authentication subkey doubles as your SSH key.\n# In ~/.bashrc or ~/.zshrc export SSH_AUTH_SOCK=$(gpgconf --list-dirs agent-ssh-socket) # Enable SSH support in gpg-agent echo \"enable-ssh-support\" \u003e\u003e ~/.gnupg/gpg-agent.conf gpgconf --kill gpg-agent Now ssh authenticates through the YubiKey. One hardware token for both GPG and SSH, which is exactly the kind of consolidation I like.\nThe downsides I’d be selling you something if I stopped here and pretended GPG is a joy to use. It isn’t. It’s a powerful tool with well-known problems, and you should walk in knowing them.\nThe UX is bad GPG was designed in the ’90s and it shows. The command-line interface is inconsistent, the error messages are cryptic, and routine tasks hide behind flags you look up every time.\ngpg: decryption failed: No secret key # What does this mean? Key expired? Wrong key? No idea. There are GUIs like Kleopatra and GPG Keychain. They help a little. They don’t fix the underlying weirdness.\nKey management is complex How do you back up keys safely? Rotate them without breaking everything? Revoke a compromised key so everyone hears about it? Share keys with people who don’t understand GPG? All of these have answers, none of them obvious, and you’ll go looking at the worst possible moment, like when a key is about to expire.\nWeb of Trust is effectively dead The original PGP idea was that people sign each other’s keys and build a network of trust you could trace. Almost nobody does this anymore. Most people import a key and trust it because the fingerprint looked roughly right, which isn’t much of a trust model.\nKeyservers are a mess The classic keyservers are often slow or offline, can’t verify that an upload belongs to the person it claims, and have a history of spam and abuse. The old SKS keyserver network is largely broken at this point.\nThe better option is keys.openpgp.org, which at least verifies that you control the email address on the key before it’ll serve it.\nMetadata leaks GPG encrypts the contents and nothing else. The encrypted blob still shows who the recipient is, the original filename and size, and when it was encrypted. If the existence of a conversation is as sensitive as its contents, that gap matters.\nEmail encryption never took off PGP email, through tools like Enigmail, never reached normal people. The setup is too fiddly, the UX too rough, and Signal solved the same problem with a fraction of the effort. I don’t encrypt my email day to day, and I won’t pretend otherwise.\nAlternatives exist, and that’s fine age does modern, dead-simple file encryption with no key management ceremony. Signal handles messaging with a UX in a different league. Keybase was an earnest attempt to make PGP friendly, and it’s owned by Zoom now.\nWhen GPG is still the right tool With all that on the table, GPG stays the best option for a handful of jobs. Git commit signing has no serious competitor. pass is built on GPG, so a password manager in the terminal means GPG. Package managers expect GPG signatures on software, so you meet them where they are. For encrypted backups it’s old, boring and supported everywhere, which is exactly what you want from a backup. And anyone with GPG can read what you send, no account or app required.\nMy setup # ~/.gnupg/gpg.conf default-key ABC123... keyserver hkps://keys.openpgp.org auto-key-retrieve # Modern crypto preferences personal-cipher-preferences AES256 AES192 AES personal-digest-preferences SHA512 SHA384 SHA256 personal-compress-preferences ZLIB BZIP2 ZIP Uncompressed My master key lives on an air-gapped machine and the subkeys live on a YubiKey. Day to day I use GPG for three things. Git commit signing, pass, and the odd file now and then. Signal for chat, and age when I want throwaway encryption. GPG doesn’t have to do everything to be worth keeping around.\nPractical tips Back up your keys # Now. Before you forget. gpg --armor --export-secret-keys \u003e ~/backup/gpg-secret-keys.asc gpg --armor --export \u003e ~/backup/gpg-public-keys.asc gpg --export-ownertrust \u003e ~/backup/gpg-ownertrust.txt Put this on an encrypted USB stick, somewhere safe, and do it now. Lose the only copy of a private key and there’s no support line to call.\nSet a calendar reminder for expiry Keys expire, and that’s by design. Forget to extend in time and you can lock yourself out of your own encrypted files.\ngpg --edit-key tom@example.com gpg\u003e expire # Choose new date gpg\u003e save Use a good passphrase Your passphrase is the lock on your private key. Make it long and unpredictable, and let a password manager remember it, so you’re not tempted to pick something you can type from memory.\nVerify fingerprints out-of-band When you import someone’s public key, confirm the fingerprint through a separate channel, a phone call or in person. Keyservers can be compromised. A fingerprint you checked yourself is worth more than one a server handed you.\nWhere to stop One command, one keypair, working crypto in five minutes. That’s still the foundation. Subkeys, hardware tokens and keyservers are depth you add when you need it, and not before.\nIs GPG worth learning? Yes, if you want to sign git commits, use pass, encrypt files properly or verify software you download. Skip it if all you want is secure chat, where Signal wins, or trivial file encryption, where age is friendlier. And if CLI papercuts ruin your day, go in with your eyes open.\nStart at the shallow end. Generate a key, turn on signed git commits, and live with that for a while. Subkeys and a YubiKey can wait until the day you want them. I sign every commit with a subkey on a YubiKey, and I still look up the flag for a detached signature every time I need one.\nFurther reading:\nGnuPG documentation OpenPGP Best Practices keys.openpgp.org - modern keyserver with email verification age - simple alternative for file encryption ","permalink":"https://tommeurs.nl/en/posts/gpg-beginners-guide/","title":"GPG explained: from first key to daily use"},{"content":" TL;DR Taskwarrior, timewarrior and vit are my task list, my time tracking and the vim-style screen on top, all in the terminal and all plain text in ~/.task. Two commands get you started. The rest of this post is what I bolted on afterwards, from projects and dependencies to a self-hosted sync server, and you can stop at any section.\nFor context on how my brain works, see Working with an AuDHD Brain.\nMy whole productivity system fits in two lines:\ntask add \"Write the intro\" +next task 1 start That’s it. The task is written down, the clock is running, and both happened in under a second without me touching the mouse. The stack behind it is taskwarrior for tasks, timewarrior for time, and vit as a vim-style screen on top of the first one. Keyboard first, and plain text on disk.\nNo Electron apps. No cloud sync shipping my data to a vendor I can’t inspect. The files sit in ~/.task and they’re mine.\nWhy this stack I fought with task management for years. Todoist, Things, Notion, a pile of Obsidian plugins. I tried them all and dropped them all, and the reason was the same every time. Getting a thought out of my head and into the tool took longer than the thought lasted.\nFor my brain that’s the whole problem. If writing a task down costs me three clicks and a loading spinner, the task doesn’t get written down. A to-do list with a loading spinner is something I resent on principle. Taskwarrior takes under a second, needs no mouse, works offline without a server, plugs into any script I care to write, and keeps its data in files I can open with any editor.\nIs there a catch? There’s a learning curve, and the first week feels clumsy. I think it pays for itself within days. The rest of this post is everything I added to those two lines, from task add to a sync server I host myself.\nInstallation macOS brew install task timewarrior vit Linux (Debian/Ubuntu) sudo apt install taskwarrior timewarrior pip install vit # or via package manager Arch Linux sudo pacman -S task timew vit After installing, run task once so it creates the config:\ntask version # Creates ~/.taskrc and ~/.task/ Add, list, modify, start Adding tasks # Simple task task add \"Review pull request\" # With tags task add \"Fix login bug\" +urgent +backend # With due date task add \"Report delivery\" due:friday # With project task add \"Write API endpoint\" project:acme.api # All together task add \"Deploy to production\" project:acme.api +urgent due:tomorrow priority:H Viewing tasks task list # All pending tasks task # Same as task list (or task next) task all # Including completed and deleted task completed # Only finished tasks task +urgent # Filtered by tag task project:acme # Filtered by project Managing tasks task 1 done # Mark as done task 1 delete # Delete task 1 modify +tag # Add tag task 1 modify due:monday # Change due date task 1 edit # Open in $EDITOR task 1 info # All details Starting and stopping tasks task 1 start # Mark as \"active\" task 1 stop # No longer active task +ACTIVE # View active tasks Projects Projects in taskwarrior are hierarchical, with dots between the levels:\n# Project structure task add \"Something\" project:client.project.subproject # Examples task add \"Design API\" project:acme.api task add \"Write tests\" project:acme.api.backend task add \"Build UI\" project:acme.api.frontend Viewing projects # All tasks in a project (including subprojects) task project:acme list # Only specific project task project:acme.api list # Project summary task projects Contexts narrow the view A context is a filter that stays on until you switch it off:\n# Define context task context define work \"project:acme or project:byteherder\" task context define acme \"project:acme\" # Activate context task context acme # Now 'task list' only shows acme tasks # View context task context show # Disable context task context none That’s what my p function does. The full implementation is in my time-system scripts.\nDependencies Tasks can depend on each other:\n# Task 2 depends on task 1 task 2 modify depends:1 # Multiple dependencies task 3 modify depends:1,2 # When creating task add \"Deploy\" depends:1,2 Blocked tasks # View what's blocked task blocked # View what's unblocked (ready to do) task unblocked # View the dependency chain task 1 info # Shows: \"blocking: 2, 3\" and \"blocked by: (none)\" A worked example task add \"Design database schema\" # Task 1 task add \"Build API endpoints\" depends:1 # Task 2 task add \"Frontend integration\" depends:2 # Task 3 task add \"Write tests\" depends:1 # Task 4 task add \"Documentation\" depends:2,3,4 # Task 5 task list # ID Dep Description # 1 Design database schema # 2 1 Build API endpoints [blocked] # 3 2 Frontend integration [blocked] # 4 1 Write tests [blocked] # 5 2-4 Documentation [blocked] task 1 done # Now tasks 2 and 4 are unblocked task list # ID Dep Description # 2 Build API endpoints # 3 2 Frontend integration [blocked] # 4 Write tests # 5 2-4 Documentation [blocked] Tags Tags are labels you make up as you go:\n# Adding task add \"Something\" +urgent +backend +review # Filtering task +urgent list task +backend +review list # AND task +backend or +frontend # OR task -urgent # NOT (exclude) # Remove tag task 1 modify -urgent Tag conventions +next # Next to do +urgent # High priority +waiting # Waiting on someone else +review # Needs review +blocked # Manually blocked (besides depends) +quick # \u003c 15 minutes Priorities Three levels, and I’ve never wanted a fourth:\n# Three levels: H(igh), M(edium), L(ow) task add \"Critical fix\" priority:H task add \"Nice to have\" priority:L # Modify task 1 modify priority:M # Filter task priority:H list Due dates Taskwarrior reads dates the way you’d type them to a colleague:\n# Specific task add \"Meeting\" due:2026-01-30 task add \"Deadline\" due:2026-01-30T14:00 # Relative task add \"Soon\" due:tomorrow task add \"This week\" due:friday task add \"In a week\" due:1week task add \"End of month\" due:eom task add \"In 3 days\" due:3d # Recurring task add \"Weekly review\" due:monday recur:weekly task add \"Standup\" due:tomorrow recur:daily until:2026-12-31 Viewing by due date task due:today list # Today task due.before:friday # Before friday task overdue # Past due task +OVERDUE # Same Reports Taskwarrior ships with a report for each question you ask most:\ntask list # Default list task next # Next tasks (smartly sorted) task ready # Ready to do (unblocked, not waiting) task blocked # Blocked by dependencies task waiting # Waiting on something task recurring # Recurring tasks task completed # Finished task all # Everything Custom reports Define your own in ~/.taskrc:\n# Minimal list report.minimal.columns=id,description report.minimal.labels=ID,Description report.minimal.filter=status:pending # Then: task minimal Timewarrior integration Timewarrior tracks time on its own, and with one hook it follows every task start and task stop too.\nManual tracking timew start \"coding\" +projectX # Start tracking timew stop # Stop timew # Current status timew summary # Today timew summary :week # This week timew summary :month # This month The taskwarrior hook This is the hook that ties the two together:\n# ~/.task/hooks/on-modify.timewarrior #!/usr/bin/env python3 import json import subprocess import sys old = json.loads(sys.stdin.readline()) new = json.loads(sys.stdin.readline()) if 'start' in new and 'start' not in old: project = new.get('project', 'untagged') tags = new.get('tags', []) subprocess.run(['timew', 'start', project] + tags) if 'start' in old and 'start' not in new: subprocess.run(['timew', 'stop']) print(json.dumps(new)) chmod +x ~/.task/hooks/on-modify.timewarrior Now task 1 start starts the clock as well.\nTimewarrior reports timew summary :day # Today timew summary :week # This week timew summary 2026-01-20 - today # Range timew tags # All tags timew summary :week project:acme # Filtered Correcting time Forgot to start the clock? Fix it after the fact:\n# Adjust current tracking timew modify start -30min # Started 30 min earlier # Adjust historical entry timew summary :ids # Find ID timew modify @1 start 09:00 # Adjust start timew delete @1 # Delete entry Vit, a vim interface for taskwarrior Vit is a TUI with vim keybindings. If your fingers already know j, k and /, it needs no manual.\nStarting vit # Default view vit project:acme # Filtered view Navigation Key Action j / k Up / down gg Go to beginning G Go to end Ctrl+d / Ctrl+u Half page down/up / Search n / N Next/previous match Task actions Key Action a Add task e Edit task (in $EDITOR) m Modify (inline) d Done D Delete s Start/stop u Undo Enter Task info Modify examples Press m and type what you’d type after task modify:\nproject:acme # Set project +urgent # Add tag -waiting # Remove tag due:friday # Set due date depends:1 # Add dependency priority:H # Set priority Switching views Key View 1 next 2 list 3 completed 4 blocked 5-9 Custom reports Command mode Key Action : Type task command directly :q Quit :!command Shell command :project:acme +urgent filters the list live, for example.\nVit config ~/.vit/config.ini:\n[vit] default_keybindings = vi [report] default_report = next [color] color = on My daily workflow All my aliases, functions and scripts sit in one repo, time-system:\n# Install git clone https://github.com/kapott/time-system.git cd time-system \u0026\u0026 ./install.sh source ~/.zshrc One directory per project The scripts expect two levels of directories, client and then project:\n~/Documents/Git/ # TASKFUNC_PROJECTS_DIR ├── clientA/ │ ├── project1/ │ └── project2/ ├── clientB/ │ └── projectX/ A day, start to finish # Morning p # Select client/project with fzf # -\u003e cd's to directory # -\u003e sets taskwarrior context # -\u003e shows pending tasks vit # View tasks in TUI # Working tstart # Pick task with fzf, starts timer # ... work ... tstop # Stop task + timer # Throughout the day tadd \"New task\" # Adds to current project (or inbox if no scope) tpl # List tasks for current project tnext # What's next? # End of day dayend # Summary: completed, time tracked, still active The commands I use most Command Description p Select project, cd to dir, set scope tstart [id] Start task + timer (fzf if no id) tstop Stop current task + timer tadd \u003cdesc\u003e Add task to current project tdone [id] Complete task (fzf if no id) tpl List tasks for current project tctxn Clear project scope weekstart Weekly review dayreview Daily review dayend End of day summary twcurrent Current tracking status The full command reference lists every alias and function.\nBackup Everything lives in ~/.task/, so a backup is a copy:\n# Simple backup cp -r ~/.task ~/.task-backup-$(date +%Y%m%d) # Or with git cd ~/.task \u0026\u0026 git init \u0026\u0026 git add -A \u0026\u0026 git commit -m \"Backup\" Custom fields and urgency User defined attributes Custom fields, which taskwarrior calls UDAs:\n# In ~/.taskrc uda.estimate.type=duration uda.estimate.label=Est # Usage task add \"Big job\" estimate:2h task estimate.over:1h list Urgency tuning Taskwarrior computes urgency on its own, and the coefficients are yours to change:\n# ~/.taskrc urgency.user.tag.urgent.coefficient=5.0 urgency.user.tag.quick.coefficient=2.0 urgency.blocking.coefficient=8.0 urgency.blocked.coefficient=-5.0 Taskd, the sync server Use taskwarrior on more than one machine and you’ll want them to agree. Taskd is the official sync server for that, and you host it yourself, so the data stays on a machine you control.\nWhat taskd gives you The same tasks on every machine, merge conflicts handled by taskwarrior itself, offline first with a sync whenever the server is reachable, and a server that’s yours.\nAlternatives Before you set up taskd, look at the alternatives:\nOption Complexity Advantage Syncthing Low Simple, sync entire ~/.task folder Git Low Version control, but no real sync Taskd Medium Official, proper merge handling Taskchampion Sync Server Medium Newer, for Taskwarrior 3.x For most people Syncthing is the easiest option. Taskd is for when you want a proper merge instead of last write wins.\nInstalling the taskd server Via package manager # Debian/Ubuntu sudo apt install taskd # Arch sudo pacman -S taskd # macOS (as server, unusual) brew install taskd Docker (recommended) # docker-compose.yml version: '3' services: taskd: image: andir/taskd ports: - \"53589:53589\" volumes: - ./taskd-data:/var/taskd environment: - CERT_CN=taskd.example.com - CERT_ORGANIZATION=MyOrg docker-compose up -d Server configuration If you install taskd by hand:\n# Create data directory export TASKDDATA=/var/taskd sudo mkdir -p $TASKDDATA sudo chown $USER:$USER $TASKDDATA # Initialize taskd init # PKI setup (certificates) cd /usr/share/taskd/pki # or where your pki scripts are ./generate # Copy certificates cp client.cert.pem $TASKDDATA cp client.key.pem $TASKDDATA cp server.cert.pem $TASKDDATA cp server.key.pem $TASKDDATA cp server.crl.pem $TASKDDATA cp ca.cert.pem $TASKDDATA # Configure server taskd config --force client.cert $TASKDDATA/client.cert.pem taskd config --force client.key $TASKDDATA/client.key.pem taskd config --force server.cert $TASKDDATA/server.cert.pem taskd config --force server.key $TASKDDATA/server.key.pem taskd config --force server.crl $TASKDDATA/server.crl.pem taskd config --force ca.cert $TASKDDATA/ca.cert.pem taskd config --force server localhost:53589 taskd config --force log /var/log/taskd.log taskd config --force pid.file /var/run/taskd.pid Organisations, groups and users Taskd is organised in organisations, optional groups, and users:\n# Create organizations taskd add org Work taskd add org Home taskd add org Freelance # Groups within organizations (optional) taskd add group Work backend-team taskd add group Work devops-team # Users per organization taskd add user Work tom taskd add user Work jan taskd add user Home tom # Output: New user key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx # SAVE THIS KEY! One database per user, no sharing This is a common misunderstanding, so it gets its own heading. Taskd doesn’t share tasks between users.\nEach user gets their own separate task database. Tom’s tasks are invisible to Jan, even inside the same organisation. What taskd does is let Tom sync his own tasks between laptop, desktop and phone, and let Jan do the same with his devices. They share one server and nothing else.\nSo no shared projects, no team task management, no assigning a task to a colleague. If you need that anyway, these are the ways around it:\nOption How Downside Shared account Everyone uses same credentials No audit trail, who did what? Export/import task export | ssh colleague task import Manual, no real sync Different tool Vikunja, Kanboard, Plane Not taskwarrior My advice is to use taskwarrior for your own tasks and a dedicated tool for the team. Don’t try to make taskd do something it wasn’t designed for.\nGenerating client certificates cd /usr/share/taskd/pki ./generate.client username # This creates: # - username.cert.pem # - username.key.pem Copy ca.cert.pem, username.cert.pem and username.key.pem to the client machine.\nConfiguring the taskwarrior client On every machine that should sync:\n# Place certificates in ~/.task cp ca.cert.pem ~/.task/ cp username.cert.pem ~/.task/ cp username.key.pem ~/.task/ # Configure in ~/.taskrc task config taskd.certificate ~/.task/username.cert.pem task config taskd.key ~/.task/username.key.pem task config taskd.ca ~/.task/ca.cert.pem task config taskd.server taskd.example.com:53589 task config taskd.credentials 'MyOrg/username/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' Starting the server # Foreground (for testing) taskdctl start # As systemd service sudo systemctl enable taskd sudo systemctl start taskd Synchronising # First sync (push everything to server) task sync init # Normal sync task sync Automatic sync A hook syncs after every command, cron syncs on a schedule. The hook first:\n# ~/.task/hooks/on-exit-sync (automatically after every task action) #!/bin/bash task sync chmod +x ~/.task/hooks/on-exit-sync Or, more conservatively, cron:\n# Sync every 30 minutes */30 * * * * /usr/bin/task sync \u003e\u003e /tmp/tasksync.log 2\u003e\u00261 Checking sync status task sync # Sync and show status task diagnostics | grep -A5 Sync # View sync config Troubleshooting taskd If sync says “Certificate fails validation”, check the certificates against the CA:\n# Check if certificates are correct openssl verify -CAfile ~/.task/ca.cert.pem ~/.task/username.cert.pem If it says “Could not connect to server”, check that the server is up and the port is open:\n# Check if server is running nc -zv taskd.example.com 53589 # Check firewall sudo ufw allow 53589/tcp Taskwarrior resolves most “Sync conflict” messages itself. If it doesn’t:\ntask sync # View conflict messages task list # Check if data is correct Syncthing as the alternative If taskd is more than you want, Syncthing does the job:\n# Install Syncthing on both machines # Sync the ~/.task folder # Done # Downside: no real merge, last write wins # Advantage: works for everything, not just taskwarrior My own ~/.task syncs over Syncthing. It’s simple and it works.\nTroubleshooting “No matches” Usually a context is still active:\n# Check your context task context show # Clear context task context none A task that seems gone It’s probably completed or deleted, and both are still on disk:\ntask all # View everything task status:completed # Check completed task status:deleted # Check deleted Undo task undo # Undo last action Two commands are enough The two lines at the top are still my whole system. Everything after them, projects, dependencies, the clock, vit, the sync server, I added once I missed it, and every one of them is optional.\nIf you never type more than task add and task done, you have a complete system and it’ll serve you for years. Vit earns its place when the list gets too long to read on one screen, and timewarrior the first time someone asks where your week went. The scripts come last, on the day you want it wired to your own directories. Add each one when you miss it.\nThere’s a learning curve, and I won’t pretend otherwise. The reason I keep coming back to this stack is that it bends to how I work instead of forcing me into someone else’s idea of a productivity app, and the data sits in plain text that I control. If you try it and it doesn’t click, fair enough. Mine has stayed.\nHere’s your first task:\ntask add \"Learn taskwarrior\" +next due:today task 1 start ","permalink":"https://tommeurs.nl/en/posts/taskwarrior-timewarrior-vit-tutorial/","title":"Taskwarrior, Timewarrior and Vit: The Ultimate CLI Productivity Stack"},{"content":"For a long time the cloud was just the water I swam in. I provisioned managed Kubernetes, clicked through the console, pasted a Terraform module someone else had written, and shipped. It worked. The bill landed every month, the dashboards stayed green, and I didn’t look underneath. Why would I? I trusted it the way you trust a lift. Press the button, doors open.\nThen one afternoon a managed service fell over and I sat there with nothing to debug. No host to SSH into, no log I was allowed to read, nothing underneath it I could open up. The status page said “investigating”. Refreshing a page someone else controlled was the whole of my afternoon.\nI self-host nearly everything now. Kubernetes on refurbished mini-PCs, Forgejo on my own hardware, Vault for secrets, plus monitoring, logging, the lot. People ask why, and it’s a fair question, because managed cloud is easier and AWS, Azure and Google run it at scale and reliably. The answer started on that afternoon, and it took me a while to say it out loud.\nThe short version is agency. The long version is this post.\nThe afternoon the black box stayed shut Nothing about that outage was dramatic. A managed component I depended on stopped responding, my app started throwing errors, and I went to do what I always do when something breaks. Find the cause.\nI couldn’t. The thing that was failing lived behind an abstraction I had no access to. I could see the symptom and nothing else. So I opened a support ticket and waited, which is a polite way of saying I sat on my hands while my own system was broken.\nThe downtime itself didn’t bother me much. Downtime happens. What bothered me was that I had no relationship with my own infrastructure. I’d outsourced the running of it, fine, but I’d outsourced the understanding of it too, without noticing. When it broke there was nothing for me to pull on, because I’d never been allowed near it.\nThat’s the splinter. A black box I’m responsible for feels like a splinter under the skin. I need to understand what I’m running, and a system I can’t open is a system I can only wait on.\nA default I’d never looked at With the ticket still open I had to admit something I didn’t like. I’d been telling myself the cloud was a choice. It wasn’t. It was a default I’d never examined. Every team I’d worked on used it, so I used it. There was a trade-off in there, and I’d never once looked at it.\nThe trade-off goes like this. Managed services hide complexity, and hiding complexity is exactly what you pay them for. The hard parts stop being your problem. But the complexity doesn’t go anywhere. It moves behind a wall you’re not allowed past. On a good day that wall is convenience. On the afternoon I’m describing, it was the reason I couldn’t do my job.\nFor context on how my brain works, see Working with an AuDHD brain.\nMy AuDHD brain made the splinter impossible to ignore. When something breaks I want the cause, and “it’s an AWS issue” is a shrug with a status code. Hidden complexity doesn’t feel restful to me. It feels like an itch I can’t reach. The autism side wants to shape my environment to how I work rather than the other way round, and “computer says no” is where managed services tend to end that conversation. The ADHD side learns by building and breaking, with the docs coming a distant second, and a black box gives you nothing to take apart.\nSo the question I’d been avoiding was a simple one. Had I ever decided on managed infrastructure, or had I just ended up there? That afternoon the honest answer was the second one. And I couldn’t un-know it.\nWhat I decided sovereignty means Digital sovereignty means real control over your own infrastructure, data and processes. The opposite is dependency, and dependency has a shape you can recognise. Your data sits with a provider who can change the terms whenever they like. Your infrastructure runs on a platform you don’t understand. Vendor lock-in makes leaving expensive enough that you never do. Prices go up and you have nowhere to walk to.\nI want to be careful here, because this turns into paranoia fast, and I don’t mean paranoia. Sovereignty is having choices. The freedom to leave when something stops working for you, and enough understanding of the system that leaving is even possible. A dependency on something you can’t inspect or control is a vulnerability. I file it next to an unpatched service, and I treat it the same way.\nSo I chose. I stopped letting the default decide for me. The things I depend on would run on hardware I own, with software I can read, so that the next time something breaks I have somewhere to start.\nThe closed container in space There’s a scenario I keep coming back to, and I wrote it up separately as the space container thought experiment. You wake up inside a sealed container drifting in space. Everything you need to stay alive is in there with you, and nothing comes in or goes out. What do you do?\nYou start learning the systems. How the oxygen gets scrubbed and recycled, how the water is filtered, where the power comes from. You want to know every component, because in that container a black box can kill you.\nMy homelab isn’t a survival pod, and I’m under no illusion that it is. But the instinct carries over. I want to understand the systems I depend on, far enough down that I could fix them if I had to. Island mode, a few sections down, is the same idea with a power switch.\nWhat that choice looks like in practice My current stack runs on refurbished mini-PCs. Commodity hardware, no expensive servers:\nComponent Solution Kubernetes K3s with embedded etcd GitOps ArgoCD (App-of-Apps pattern) CI/CD Forgejo (self-hosted) Secrets HashiCorp Vault Policy Kyverno Monitoring Prometheus + Thanos Logging Loki Tracing Tempo Dashboards Grafana CNI Cilium (eBPF ) Storage Longhorn + MinIO Certificates cert-manager Auth Keycloak Everything is GitOps-managed. Configuration lives in Git, ArgoCD syncs it to the clusters, and the whole thing is declarative, reproducible and auditable. When I want to know why something is the way it is, the answer is a git log away.\nIsland mode One design rule I push harder than the rest is island mode. My clusters have to keep working without an internet connection. In practice that means local container registry mirrors, no hard dependencies on external APIs, every secret held locally in Vault, and documentation that lives on the cluster instead of on someone else’s website.\nI don’t expect the internet to vanish. The constraint is there for what it does to the design. A system that survives being cut off is a system you understand all the way down, because you had to find and remove every dependency you couldn’t account for. Offline-capable and well-understood turned out to be the same requirement.\nThe bill for all this Rebuilding on hardware I own came with a price tag, and leaving it out would make this an advert. So here’s the invoice.\nTime first. You’re your own ops team. Cluster upgrades, security patches, incident response, that’s your evening now. It costs real hours, especially at the start. I pay it down with automation. K3s upgrades run themselves, GitOps keeps everything consistent, and Kyverno policies block the misconfigurations I’d otherwise make when I’m tired.\nThen knowledge. You have to know what you’re doing. Kubernetes, networking and security each have a steep curve, and self-hosting hands you all of them at once. This is the part of the bill I’m happy to pay. The knowledge I buy by running my own stack makes me better at the day job, because I understand an abstraction a lot better once I’ve seen what’s under it.\nMoney up front. Hardware costs something. Refurbished mini-PCs are cheap, but cheap isn’t free, and it’s a barrier if you’re starting cold. The running costs land in my favour, though. My entire homelab draws less electricity in a month than one managed Kubernetes cluster costs to rent.\nAnd no SLA . When it breaks there’s no ticket to file and no “this is an AWS problem” to hide behind. It’s me. That was the point, so I don’t get to complain about it. I don’t wait on a status page anymore. I open a shell and start looking, and the looking is where the understanding comes from.\nWhen I’d still reach for managed cloud I made a choice for my situation, and I’d be a hypocrite to pretend it’s the only right one. Managed cloud earns its place in plenty of cases.\nA startup that needs to ship this quarter should be working on the product and leaning on managed services until scale or cost starts to hurt. A team where nobody knows Kubernetes is making a sensible call with EKS, GKE or AKS, as long as they keep building the knowledge so the choice stays open. Some compliance regimes are easier to satisfy with certified managed services. And if you need ten times the capacity for one Black Friday, cloud elasticity is worth real money.\nWhat all of these have in common is the thing I missed for years. Make the choice on purpose. Look at the trade-off, name what you’re handing over and what you get back, and then pick managed because it fits. Picking it because everyone around you had is how I ended up refreshing that status page.\nWhere that leaves me The day-to-day looks a lot like it did. Kubernetes, YAML in the evenings, the occasional outage to chase. The dashboards point at boxes in my own rack now, and that’s most of the visible difference.\nThe difference I feel is that I can reach it. I no longer treat my infrastructure as a subscription I hope keeps working. It’s a system I’m responsible for and can get into. When something breaks now there’s always somewhere to start, and starting there teaches me something.\nUnderstanding beats trust for me. If I know how something works I can fix it, tune it and change it. If I only trust it, I’m betting on someone else’s competence and goodwill, and on that afternoon the bet lost. Options beat convenience. Managed services are lovely right up until you want to leave, and the cost of leaving is one they charge you the whole time without ever itemising it. And I’d rather learn than consume. A problem I solve myself I can solve again next year. The ones I outsourced I couldn’t solve once.\nIf you want to feel some of this yourself, start small. One Raspberry Pi or an old laptop, Docker Compose, no Kubernetes yet. Containers, networking and storage are plenty. Add one service at a time and let each one teach you something. Write down why you chose what you chose, because you won’t remember. Expect things to break, because in a homelab the blast radius is tiny and the lesson is the same size as it would be at work. And talk to other people doing the same thing. Someone in a homelab forum has already hit your exact wall and written up how they got past it.\nThis isn’t a prescription. Not everyone needs to self-host everything, and plenty of people are right to stay on managed cloud with their eyes open. The one thing I’d ask is what that afternoon asked of me. Don’t let the default decide for you. Know what you give up when you hand a system to someone else, and know what you get.\nThe splinter never fully came out. I’ve stopped wanting it to. When something breaks now, I open a shell.\nRelated posts:\nThe space container thought experiment - Systems thinking and closed systems Working with an AuDHD brain - How my brain influences my tech choices ArgoCD for beginners - Declarative infrastructure Building an Internal Developer Platform - The broader context ","permalink":"https://tommeurs.nl/en/posts/sovereign-infrastructure/","title":"Why I self-host everything: on sovereignty, agency and control"},{"content":"For a long time I thought the problem was me.\nI have AuDHD , autism and ADHD together. The ADHD is ADHD-PI, the inattentive kind, without the hyperactivity most people picture when they hear the word. For years I treated the whole thing as a defect I’d train away. I bought the planners. I read the productivity books. Every week I told myself that Monday I’d finally be disciplined.\nThis post is about how that went, what I built once I stopped, and why the systems I ended up depending on turn out to be good engineering for anyone.\nIt’s context. The obsession with automation, the preference for async communication, the hours I sink into tooling, they all come from the same place. I had a long argument with my own brain and lost it. What I use today is what I built afterwards.\nThe years I tried to fix myself Here’s what a default office asks of you. Hold a dozen things in your head at once. Switch context the moment someone taps you on the shoulder. Sit in an open-plan room and tune out the noise. Start the boring task because it’s on the list, with nothing making it urgent. Stay evenly focused for eight hours, like a machine that doesn’t get bored.\nMy brain can’t do that. Pretending it could was exhausting.\nLet me be specific, because “neurodivergent” covers a lot of ground.\nThe ADHD side first.\nAttention comes and goes on its own schedule. Some days I can’t focus, other days I can’t stop. The second one has a name, hyperfocus. Starting a task is harder than doing it. The first step can feel like a wall. My working memory is small. If it isn’t written down or automated, it doesn’t exist. Urgency drives action. Without a deadline a task barely registers. Then the autism side.\nPredictability feels calm. Unexpected change drains energy. Patterns and systems are how I make sense of the world. Context switching is expensive. Every interruption has a recovery cost. Sensory input piles up. Open offices wear me down. And the two together.\nI crave structure and struggle to generate it myself. The autism wants it, the ADHD won’t build it. I need routines, and repetition bores me within days. I can focus deeply, and I rarely get to pick on what. For years I read that list as a diagnosis of failure. The advice was always the same. Try harder, be more disciplined, want it more. So I tried harder and wanted it more, and kept ending the week with the important task untouched and a vague sense that the rest of the office knew something I didn’t.\nOne more routine, gone in a week There was no breakthrough. I was tired.\nI’d set up yet another manual routine to keep myself on track, and within a week I’d let it rot, the way I always did. The usual next step was the guilt spiral. This time I asked a different question. Why was I the one who had to remember?\nThat moved the target. I’d built everything to run on willpower, and willpower is exactly what my brain won’t supply on demand. Discipline was never the bottleneck. Instead of trying to become someone who remembers, I started putting the remembering somewhere else, in an environment that does it for me.\nOn paper that’s one sentence. It’s also the reason every tool I use looks the way it does.\nWhat I built instead Once I stopped fighting my brain, the choices followed from three principles. Every tool choice on this blog traces back to one of them. None of them is a productivity hack I picked up somewhere. Take one away and the rest of my setup stops working.\n1. Minimize cognitive friction Every extra step between me and a task is a place where my brain can quietly check out. Every decision I have to make spends from a budget that’s smaller than most people’s.\nIn practice that means four things.\nAutomation over manual steps. A script doesn’t forget step four. I do. GitOps over ad-hoc changes. The desired state lives in Git, so I don’t have to recall what I changed where. CLI over GUI. Less on screen to pull my attention. One command, one action. Infrastructure as Code. The infrastructure documents itself by existing. No separate docs to go stale. The less work it takes my brain to get something done, the higher the odds it happens. Every rule above is there to make that work smaller.\n2. Facilitate flow state My brain has two modes, can’t focus and hyperfocus. The second is where my best work happens, and it’s fragile. One interruption and it’s gone, and I can’t call it back on command.\nSo I guard it.\nAsync communication as the default. I decide when I read messages. My inbox doesn’t. Deep work blocks. Mornings are for focus. Meetings get herded into the afternoon. Notifications off. All of them. I check things when I’m ready. Nothing gets to ping me first. Tooling that holds the flow. Keyboard shortcuts, terminal workflows, anything that keeps me in one context. Getting into flow is the hard part. Staying in it is the other hard part, and most of the list above is about the second one.\n3. Build external structure, systems over willpower This is the one that took me longest to accept. My brain can’t reliably build structure and hold on to it, so the structure goes in the environment instead of in my head.\nThe same idea, applied to infrastructure.\nDeclarative configuration. The desired state is written down. “This is what it should be”, in a file, instead of a memory of having configured something once. Self-healing systems. When something fails it recovers on its own. I don’t have to remember to fix it. GitOps. The git repository is the source of truth. My memory isn’t. Pipelines that enforce. Tests, linting, security scans. If it’s not in the pipeline, I will forget it. I build it once. After that the pipeline does the remembering, and it doesn’t get tired.\nWhat it cost me This wasn’t free.\nExternal structure is real work up front. Writing the automation, wiring the pipeline, making the config declarative, each of those is slower than doing the thing by hand once. There were stretches where I was clearly over-engineering a problem so I wouldn’t have to remember it, and I had to learn where that line was. I still cross it sometimes.\nAccepting how my brain works also meant giving up a story I’d told myself for years, that one more push of discipline would finally make me normal. Letting go of that story hurt for a while. The relief came later.\nSo the trade is this. I pay once, in building the system, instead of every day in willpower I don’t have.\nWhy this works for you too “But Tom, I don’t have ADHD. Why should I care?”\nBecause your brain also prefers to do as little work as possible.\nCognitive load is finite for everyone, and so is the number of decisions you can make in a day before they get sloppy. Maintenance is dull for everyone and the new shiny thing is more fun. The difference is where the budget runs out. Where you might hold three manual steps in your head, I lose the thread at the first one. Where you can choose to leave notifications on, I can’t.\nSo the fixes I’m forced into are the same ones that quietly improve anyone’s work. Automation cuts errors and saves time. Async communication protects deep work. Declarative configuration makes a system readable. Self-healing infrastructure lowers the operational load. None of that is specific to my brain.\nIf a workflow survives my brain, it’ll probably survive yours. I just hit the friction first.\nWhat this looks like in practice This blog works through concrete versions of these principles. For example:\nTaskwarrior, Timewarrior and Vit: my full task management stack, which exists to hold external structure so I don’t have to. Where I landed Discipline was never going to fix this, and I spent years finding that out the slow way.\nWhat works is accepting how my brain functions and building around it, with systems that work with me instead of against me. The funny part is that none of it is special. Automation, async, declarative config, self-healing infrastructure. Plain good engineering, on any team. Nobody would call them ADHD tips. They’re for anyone who wants systems that keep working when the person running them is tired or distracted. I’m just the one who couldn’t get by without them.\nSo when I argue for automation on this blog, or push GitOps and async communication, this is the reason underneath. My brain doesn’t work the other way. Once I stopped pretending it did, the work got better.","permalink":"https://tommeurs.nl/en/posts/working-with-audhd-brain/","title":"Working with an AuDHD brain: why I automate everything"},{"content":"As a kid I called the things that fascinated me unbreakable. Indestructible never fit, because indestructible means it never breaks. Unbreakable means it keeps working after it breaks.\nI know exactly when it started. A photo of an A-10 Thunderbolt II back from a mission with half a wing gone, the tail in tatters and the fuselage full of holes. It still brought its pilot home.\nLuck had nothing to do with it. Somebody sat at a drawing board years before that mission and decided the plane had to come home looking like that.\nThings that refuse to die You know the Camel Trophy stories? Land Rover Defenders chased through the Indonesian jungle, chassis bent, parts snapped off, the whole thing caked in mud. And they still arrived. Things went wrong the entire trip. They arrived because they’d been built to break and keep driving anyway.\nNow take a modern car. One sensor dies and you’re on the hard shoulder waiting for a tow truck.\nAt some point “robust” and “complex” became the same word. More redundancy, more failover, systems on top of systems. But complexity fails all by itself. Every extra component is one more thing that can break, and one more thing you have to understand when it does.\nWhy I do this for work I build platforms for a living. Kubernetes clusters, GitOps pipelines, that kind of thing. And every time I catch myself asking the same question. What happens when this falls over?\nThat’s a different question from how we stop it falling over. Prevention matters, but it’s a separate problem. I want to know what happens when the thing goes down anyway, because it will. Servers crash and code has bugs. Networks do things nobody planned for. Call that pessimism if you like. From where I sit it’s just the job.\nMost of the systems I run into are built for the happy path. Requests come in, responses go out, the logs are green. Then something goes sideways, and it turns out nobody ever decided what should happen at that moment. There’s no plan. There’s a stack trace, and a Slack channel that’s starting to fill up.\nKubernetes and the illusion of self-healing Kubernetes gets sold as self-healing. Pod crashed? A fresh one starts. Node dropped off the cluster? The workloads land somewhere else.\nAll true. Up to a point.\nThe catch is that the cluster is itself a system that can fail. etcd can corrupt. The control plane can go down. A network partition can split your nodes into two halves that both think they’re in charge, and none of the self-healing above helps you with any of that.\nHealing inside a cluster is great, and the cluster itself is still a single point of failure. That’s what I’ve been experimenting with lately, clusters that are unbreakable together, where the whole keeps serving when an entire cluster drops out.\nThat one’s still an experiment, so it gets its own post once it holds up.\nThe test I run before I build anything Before I build anything, I hold it against that childhood word. Perfect is the wrong bar, and so is indestructible. Would this thing keep doing its job while half of it is on fire?\nMost of the time the honest answer is no. So the next question is what would have to change to make it yes.\nSometimes the fix is small. Sometimes the answer is “this whole design has to be different”. And sometimes it’s “more effort than this problem is worth”, which is fine too, as long as I made that call on purpose instead of finding out during an outage.\nNot everything has to be unbreakable. What I want is that somebody thought about what happens when it breaks, before it breaks, while changing their mind was still cheap.\nThat’s why I like chaos engineering so much. Break it on purpose, on a day I picked, and watch what happens. I’d much rather learn that from an experiment I scheduled than from an outage I didn’t see coming. And that photo of the A-10 is still in my head every time I run one.","permalink":"https://tommeurs.nl/en/posts/unbreakable/","title":"Unbreakable - My Fascination."},{"content":" TL;DR Prometheus on its own keeps a couple of weeks of metrics on local disk and can’t answer a question across clusters. Thanos puts a sidecar next to each Prometheus, ships the blocks to object storage, and puts a querier in front that reads all of it back through the same PromQL. Below is the install, the four components, the HA and multi-cluster wiring, and the numbers I run in production.\nThe first time someone asked me “was this slower last month than it is now?”, I had no answer. My Prometheus remembered two weeks. The data I needed had aged out of local disk and was gone. That gap is why this post exists.\nPrometheus is the default for Kubernetes metrics, and I’d pick it again. It works right up to the point where you need long-term storage, or a view across more than one cluster, or high availability that survives a node. Then it stops, and it stops hard.\nThanos extends Prometheus instead of replacing it. You keep the setup you already understand, bolt a few components onto it, and your retention is limited by your bucket instead of your disk, with one query across every cluster thrown in. That “keep what you know” part is what sold me. I’d rather grow a system I can already reason about than swap it for something that promises to do everything and explains none of it.\nWhere a single Prometheus stops A single Prometheus has four hard limits. It’s one node, with no clustering or HA built in. It writes to local disk, so retention is whatever fits. It sees one cluster, and there’s no way to query across two. And it never downsamples, so a metric from last year costs the same space as one from this morning.\nFor one small cluster with two weeks of retention, none of that hurts. For a multi-cluster production setup with compliance requirements, every one of the four is a blocker. You’ll hit them. The only open question is which one first.\nWhat Thanos adds Thanos wraps a handful of components around the Prometheus you already have:\nCluster B\nCluster A\nPrometheus + Sidecar\nPrometheus + Sidecar\nObject Storage\n(S3/MinIO/GCS)\nQuerier\nStore Gateway\nCompactor\nGrafana\nThe sidecar runs next to Prometheus and uploads its blocks to object storage. The store gateway serves that history back out of the bucket. The querier answers PromQL by asking the sidecars and the store gateway and merging what comes back. The compactor downsamples and deduplicates whatever sits in the bucket.\nFour moving parts, each with one job I can hold in my head. I trust a design like that a lot more than one binary that does everything.\nInstall it with the Bitnami chart Start with the Bitnami Helm chart and get something running:\nhelm repo add bitnami https://charts.bitnami.com/bitnami helm repo update helm install thanos bitnami/thanos \\ --namespace monitoring \\ --create-namespace \\ --set objstoreConfig=\"$(cat thanos-objstore.yaml)\" The object store config in thanos-objstore.yaml points Thanos at your bucket:\ntype: s3 config: bucket: thanos-metrics endpoint: minio.storage:9000 access_key: ${MINIO_ACCESS_KEY} secret_key: ${MINIO_SECRET_KEY} insecure: true # For MinIO without TLS That’s the skeleton. Nothing is uploading into it yet, and that’s the next step.\nPrometheus with the sidecar Tell Prometheus to run the Thanos sidecar next to it:\napiVersion: monitoring.coreos.com/v1 kind: Prometheus metadata: name: prometheus namespace: monitoring spec: replicas: 2 # HA pair retention: 2h # Short local retention, Thanos handles long-term # Thanos sidecar configuration thanos: baseImage: quay.io/thanos/thanos version: v0.34.0 objectStorageConfig: key: thanos.yaml name: thanos-objstore-secret # External labels for deduplication externalLabels: cluster: production replica: $(POD_NAME) # Let Thanos sidecar access Prometheus data storage: volumeClaimTemplate: spec: storageClassName: longhorn resources: requests: storage: 50Gi The sidecar does three things:\nExposes Prometheus data to the Thanos querier over gRPC Uploads completed TSDB blocks to object storage Answers Store API queries for recent data Look at that retention: 2h. Local disk now holds only the most recent slice. The history lives in object storage, and the sidecar keeps feeding it.\nThe three components that read blocks back With the sidecar shipping blocks, you need the pieces that read them again.\nQuerier apiVersion: apps/v1 kind: Deployment metadata: name: thanos-querier namespace: monitoring spec: replicas: 2 template: spec: containers: - name: thanos-query image: quay.io/thanos/thanos:v0.34.0 args: - query - --http-address=0.0.0.0:9090 - --grpc-address=0.0.0.0:10901 # Connect to sidecars - --store=dnssrv+_grpc._tcp.prometheus-operated.monitoring.svc # Connect to store gateway - --store=dnssrv+_grpc._tcp.thanos-store.monitoring.svc # Deduplication - --query.replica-label=replica ports: - name: http containerPort: 9090 - name: grpc containerPort: 10901 Store gateway apiVersion: apps/v1 kind: StatefulSet metadata: name: thanos-store namespace: monitoring spec: replicas: 2 template: spec: containers: - name: thanos-store image: quay.io/thanos/thanos:v0.34.0 args: - store - --http-address=0.0.0.0:10902 - --grpc-address=0.0.0.0:10901 - --data-dir=/var/thanos/store - --objstore.config-file=/etc/thanos/objstore.yaml volumeMounts: - name: objstore-config mountPath: /etc/thanos - name: data mountPath: /var/thanos/store volumes: - name: objstore-config secret: secretName: thanos-objstore-secret volumeClaimTemplates: - metadata: name: data spec: storageClassName: longhorn resources: requests: storage: 10Gi Compactor apiVersion: apps/v1 kind: StatefulSet metadata: name: thanos-compactor namespace: monitoring spec: replicas: 1 # Only one compactor! template: spec: containers: - name: thanos-compact image: quay.io/thanos/thanos:v0.34.0 args: - compact - --http-address=0.0.0.0:10902 - --data-dir=/var/thanos/compact - --objstore.config-file=/etc/thanos/objstore.yaml - --retention.resolution-raw=30d - --retention.resolution-5m=90d - --retention.resolution-1h=1y - --wait volumeMounts: - name: objstore-config mountPath: /etc/thanos - name: data mountPath: /var/thanos/compact That replicas: 1 on the compactor isn’t a typo. Run two and they fight over the same blocks and corrupt your data. I put a loud comment on that line so future me doesn’t get clever with it.\nThe retention flags are where the cost comes down. Raw data stays 30 days at full resolution, the 5m downsampled series stay 90 days, and the 1h series stay 1 year. Older data takes less space because there’s less of it per hour. You trade resolution for history, and for anything older than a couple of weeks that’s the trade I want anyway.\nGitOps deployment I don’t click any of this together by hand. It goes through ArgoCD like everything else, so the cluster state lives in Git where I can read it, review it and roll it back:\napiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: thanos namespace: argocd spec: project: default source: repoURL: https://charts.bitnami.com/bitnami chart: thanos targetRevision: 12.20.0 helm: values: | objstoreConfig: |- type: s3 config: bucket: thanos-metrics endpoint: minio.storage:9000 insecure: true query: enabled: true replicaCount: 2 stores: - dnssrv+_grpc._tcp.prometheus-operated.monitoring.svc storegateway: enabled: true replicaCount: 2 persistence: size: 20Gi compactor: enabled: true retentionResolutionRaw: 30d retentionResolution5m: 90d retentionResolution1h: 1y persistence: size: 50Gi ruler: enabled: false # Use Prometheus rules instead receive: enabled: false # Using sidecar mode destination: server: https://kubernetes.default.svc namespace: monitoring Two Prometheus replicas, one set of graphs Two Prometheus replicas scraping the same targets only help if something stitches their results back together. Thanos is that something.\n# Run two Prometheus instances apiVersion: monitoring.coreos.com/v1 kind: Prometheus spec: replicas: 2 externalLabels: replica: $(POD_NAME) # Different for each replica Both instances scrape everything. The querier deduplicates on the replica label so you don’t get every line twice:\n# Querier configuration args: - --query.replica-label=replica - --query.replica-label=prometheus_replica Lose a node, or just one Prometheus pod, and the dashboards keep drawing from the replica that’s left. That’s what the second one is for.\nOne query across every cluster Point several clusters at the same Thanos deployment and give each one its own cluster label. The first one:\nexternalLabels: cluster: production-eu replica: $(POD_NAME) And the second:\nexternalLabels: cluster: production-us replica: $(POD_NAME) Now the querier aggregates across both:\n# Total requests across all clusters sum(rate(http_requests_total[5m])) # Requests by cluster sum by (cluster) (rate(http_requests_total[5m])) One PromQL query, every cluster, one answer. No more three Grafana tabs open, one per cluster, adding the numbers up in your head.\nGrafana doesn’t need to know Grafana doesn’t need to know Thanos exists. Point it at the querier and it thinks it’s talking to a plain Prometheus:\napiVersion: v1 kind: ConfigMap metadata: name: grafana-datasources data: thanos.yaml: | apiVersion: 1 datasources: - name: Thanos type: prometheus url: http://thanos-query.monitoring:9090 access: proxy isDefault: true jsonData: timeInterval: \"15s\" Every dashboard you already built keeps working. On the Grafana side, the change is that one URL.\nRecording rules for the expensive queries Some queries are expensive enough that you don’t want to run them on every dashboard refresh. Compute them once a minute and read the result:\napiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: recording-rules spec: groups: - name: aggregations interval: 1m rules: # Pre-aggregate request rate by service - record: service:http_requests:rate5m expr: sum by (service) (rate(http_requests_total[5m])) # Pre-aggregate error rate - record: service:http_errors:rate5m expr: sum by (service) (rate(http_requests_total{status=~\"5..\"}[5m])) # Pre-compute availability - record: service:availability:ratio expr: | 1 - ( service:http_errors:rate5m / service:http_requests:rate5m ) Dashboards then query the cheap service:* series instead of grinding through raw data on every load.\nAlerting stays with Prometheus Keep alerting close to the data. Run Alertmanager against Prometheus rather than against Thanos:\napiVersion: monitoring.coreos.com/v1 kind: Prometheus spec: alerting: alertmanagers: - namespace: monitoring name: alertmanager port: web ruleSelector: matchLabels: role: alert-rules Thanos Ruler exists and it works. It’s also one more component to understand and one more thing that can be broken while you’re already debugging something else. For most setups Prometheus alerting is plenty, and I’d rather have fewer parts that can surprise me.\nWatching Thanos itself Thanos exposes its own Prometheus metrics, which is just as well, because the thing watching your systems is itself a system that breaks. These are the ones I watch:\n# Sidecar upload success thanos_shipper_uploads_total thanos_shipper_upload_failures_total # Store gateway performance thanos_bucket_store_series_fetch_duration_seconds thanos_bucket_store_block_loads_total # Compactor health thanos_compact_group_compactions_total thanos_compact_group_compaction_failures_total # Querier performance thanos_query_gate_duration_seconds And alert when uploads start failing, because a silent sidecar means you’re losing history without anyone noticing:\n- alert: ThanosSidecarUploadFailing expr: increase(thanos_shipper_upload_failures_total[1h]) \u003e 0 for: 15m labels: severity: warning annotations: summary: \"Thanos sidecar failing to upload blocks\" What the storage costs This is where downsampling pays for itself:\nResolution Data per day 1 year cost (S3) Raw (15s) ~100MB/target ~$4/target 5m downsample ~3MB/target ~$0.12/target 1h downsample ~0.5MB/target ~$0.02/target Keep raw data forever and the storage bill grows a little every month until somebody asks about it. Downsample, and the old data costs close to nothing.\nFor self-hosted object storage, MinIO does the job and keeps the data on hardware I own:\napiVersion: apps/v1 kind: StatefulSet metadata: name: minio spec: template: spec: containers: - name: minio image: minio/minio:latest args: - server - /data - --console-address - \":9001\" env: - name: MINIO_ROOT_USER valueFrom: secretKeyRef: name: minio-credentials key: root-user - name: MINIO_ROOT_PASSWORD valueFrom: secretKeyRef: name: minio-credentials key: root-password What I run, all of it together Put the pieces together and this is roughly what runs in production:\n# Prometheus with sidecar prometheus: replicas: 2 retention: 6h # Very short, Thanos handles long-term thanos: objectStorageConfig: name: thanos-objstore externalLabels: cluster: production environment: prod # Thanos components thanos: query: replicaCount: 2 stores: - dnssrv+_grpc._tcp.prometheus-operated.monitoring.svc - dnssrv+_grpc._tcp.thanos-store.monitoring.svc storegateway: replicaCount: 2 persistence: size: 50Gi compactor: retentionResolutionRaw: 14d retentionResolution5m: 60d retentionResolution1h: 365d persistence: size: 100Gi # Object storage minio: replicas: 4 persistence: size: 500Gi Each of those numbers has a reason. Local retention is 6h because the sidecar uploads often, so there’s no point hoarding on local disk. Raw retention is 14d, which covers any debugging session I’m likely to have. The 1h series stay 1 year, enough trend history for capacity planning. And the bucket is self-hosted MinIO , so the metrics stay on my hardware, with no cloud bill and no third party holding them.\nSame chart as the install at the top. The values are what changed.\nWhat the metrics are for Metrics are how I understand my systems instead of guessing about them. The questions are the same in every incident and every planning session:\nIs this service healthy? What changed right before it broke? Are we meeting our SLOs ? Where is optimization worth the effort? Lose long-term metrics and “compared to when?” has no answer. Lose cross-cluster queries and you only ever see one piece at a time. Prometheus plus Thanos gives you retention measured in years, one view across clusters and an HA pair, behind the same Prometheus interface you already know how to read.\nThat question from the start of this post, “was this slower last month?”, has an answer now. One query against the querier, and the data is there because the sidecar has been uploading it all along.","permalink":"https://tommeurs.nl/en/posts/prometheus-thanos-metrics/","title":"Prometheus and Thanos: Metrics at Scale"},{"content":" TL;DR Kubernetes ships with a permission model that hands every pod a token and lets the default service account collect rights nobody remembers granting. Least privilege means a service account per workload, no token unless the app talks to the API, and roles that name the namespace, the resource and the verb. Below are the four RBAC resources, the five settings that cap the blast radius, the bindings I copy most often, and how I check and enforce all of it.\nThe first cluster I ever ran in anger had exactly one permission model, and it was cluster-admin. My CI pipeline, my monitoring stack, the little webhook receiver I threw together one afternoon. All of it could read every secret, delete every deployment and touch every namespace. It worked fine, right up until I started thinking about what happens when one of those pods gets popped.\nKubernetes RBAC (Role-Based Access Control) answers one question. Who can do what to which resources? On most clusters the answer is “everyone can do everything”, and nobody decided that. It’s what you get when you never write anything down.\nI care about this for the same reason I care about understanding the rest of my stack. When a service account gets compromised, I want to know how far the damage reaches before it gets there. When I fat-finger a kubectl command, I want the damage small enough that I can fix it myself before anyone notices. Least privilege is the ceiling on both.\nThis post goes from the four RBAC resources to the bindings I run in production, and ends with how I check that a binding does what I think it does.\nThe four resources Four resources make up Kubernetes RBAC. Two describe permissions, two hand those permissions to someone.\nA Role describes permissions inside one namespace. This one lets its holder read pods in production and nothing else.\napiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: namespace: production name: pod-reader rules: - apiGroups: [\"\"] resources: [\"pods\"] verbs: [\"get\", \"list\", \"watch\"] A ClusterRole describes permissions across the cluster. Same shape, no namespace in the metadata.\napiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: secret-reader rules: - apiGroups: [\"\"] resources: [\"secrets\"] verbs: [\"get\", \"list\"] A RoleBinding hands a Role to a subject, in one namespace. Here the monitoring service account gets the pod-reader Role from above.\napiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: read-pods namespace: production subjects: - kind: ServiceAccount name: monitoring namespace: monitoring roleRef: kind: Role name: pod-reader apiGroup: rbac.authorization.k8s.io A ClusterRoleBinding does the same with a ClusterRole, cluster-wide. This one gives everyone in the security team read access to every secret there is.\napiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: cluster-secret-reader subjects: - kind: Group name: security-team apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: secret-reader apiGroup: rbac.authorization.k8s.io That’s the whole vocabulary. A Role and a Binding, scoped to a namespace or to the cluster. Everything else in this post is discipline about how you use those four.\nWhat you get out of the box Before the discipline, it helps to see why the defaults are the thing biting you. A deployment that doesn’t say otherwise looks like this:\n# What many deployments look like apiVersion: apps/v1 kind: Deployment spec: template: spec: # Default service account with broad permissions serviceAccountName: default # Token auto-mounted automountServiceAccountToken: true The default service account collects permissions over time. Someone bound a ClusterRole to it once to make a thing work and never walked it back. I’ve done that. The auto-mounted token then sits inside every process in the container, waiting.\nSo when an attacker gets a shell in your app, the chain is short and boring. They read the token at /var/run/secrets/kubernetes.io/serviceaccount/token, ask the API server what it’s allowed to do, and walk through whatever happens to be there.\nThe next five settings each close part of that chain. Apply one and you’re better off than the defaults. Apply all five and a compromised pod is barely worth the attacker’s time.\nA service account per workload Start here. Never run a workload as the default service account.\napiVersion: v1 kind: ServiceAccount metadata: name: api-server namespace: production --- apiVersion: apps/v1 kind: Deployment metadata: name: api-server spec: template: spec: serviceAccountName: api-server automountServiceAccountToken: false # Don't mount unless needed Each application gets its own account, and permissions land on that account and nowhere else. The moment you do this, your audit log starts naming who did what instead of showing a wall of default. That alone made me wish I’d done it a year earlier.\nNo token unless the app needs one Here’s the part most people skip. The bulk of my workloads never talk to the Kubernetes API at all. A web frontend doesn’t list pods. So why hand it a token?\napiVersion: v1 kind: ServiceAccount metadata: name: web-frontend automountServiceAccountToken: false --- apiVersion: apps/v1 kind: Deployment spec: template: spec: serviceAccountName: web-frontend automountServiceAccountToken: false # Belt and suspenders Get a shell in this workload and there’s nothing to steal. No token on disk, no API access, done. This one setting kills the pivot I described above, and it costs nothing for the apps that never needed the API in the first place.\nKeep roles inside the namespace For the workloads that do need the API, the next question is how far that access reaches. Roles are namespace-scoped by default, which is a gift. Take it.\n# Role only works in production namespace apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: namespace: production name: deployment-manager rules: - apiGroups: [\"apps\"] resources: [\"deployments\"] verbs: [\"get\", \"list\", \"watch\", \"create\", \"update\", \"patch\"] A compromised service in production can’t touch staging or anything else. The damage stops at the namespace boundary, which is exactly where I want it to stop.\nReach for a ClusterRole only when you need more than one namespace. Most apps live and die inside one namespace, so keep them there.\nName the resources Namespace scope caps how wide the access goes. This caps how deep. Grant access to named resources, not whole API groups.\n# Bad: access to all core resources rules: - apiGroups: [\"\"] resources: [\"*\"] verbs: [\"*\"] # Good: only specific resources needed rules: - apiGroups: [\"\"] resources: [\"configmaps\"] verbs: [\"get\"] resourceNames: [\"app-config\"] # Even more specific The resourceNames field is the one I see used least and like most. It pins access to specific named objects, so an app that needs one ConfigMap can read that one ConfigMap and nothing else.\nOnly the verbs it uses Last one, and the cheapest. You’ve decided which resources an account can touch. Now decide what it may do to them.\nVerb Meaning get Read single resource list Read multiple resources watch Stream changes create Create new resources update Replace existing patch Modify existing delete Remove resources deletecollection Remove multiple A read-only application needs get, list and watch, and nothing more. A monitoring agent has no business holding delete or deletecollection, so don’t give it the option.\n# Monitoring only needs read access rules: - apiGroups: [\"\"] resources: [\"pods\", \"services\", \"endpoints\"] verbs: [\"get\", \"list\", \"watch\"] The bindings I copy most often Five rules are easy to nod along to and hard to apply under deadline. So here are the ones I copy-paste, each already wearing all five.\nAn application that reads a ConfigMap apiVersion: v1 kind: ServiceAccount metadata: name: config-reader namespace: myapp --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: config-reader namespace: myapp rules: - apiGroups: [\"\"] resources: [\"configmaps\"] verbs: [\"get\", \"watch\"] resourceNames: [\"app-settings\"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: config-reader namespace: myapp subjects: - kind: ServiceAccount name: config-reader namespace: myapp roleRef: kind: Role name: config-reader apiGroup: rbac.authorization.k8s.io An operator that manages its own custom resources apiVersion: v1 kind: ServiceAccount metadata: name: myapp-operator namespace: myapp-system --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: myapp-operator rules: # Own CRDs - apiGroups: [\"myapp.example.com\"] resources: [\"myapps\", \"myapps/status\"] verbs: [\"get\", \"list\", \"watch\", \"create\", \"update\", \"patch\", \"delete\"] # Resources it creates - apiGroups: [\"apps\"] resources: [\"deployments\"] verbs: [\"get\", \"list\", \"watch\", \"create\", \"update\", \"patch\", \"delete\"] - apiGroups: [\"\"] resources: [\"services\", \"configmaps\"] verbs: [\"get\", \"list\", \"watch\", \"create\", \"update\", \"patch\", \"delete\"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: myapp-operator subjects: - kind: ServiceAccount name: myapp-operator namespace: myapp-system roleRef: kind: ClusterRole name: myapp-operator apiGroup: rbac.authorization.k8s.io A CI pipeline that deploys to production apiVersion: v1 kind: ServiceAccount metadata: name: gitlab-deployer namespace: ci-cd --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: deployer namespace: production rules: - apiGroups: [\"apps\"] resources: [\"deployments\"] verbs: [\"get\", \"list\", \"watch\", \"update\", \"patch\"] - apiGroups: [\"\"] resources: [\"configmaps\", \"secrets\"] verbs: [\"get\", \"list\", \"watch\", \"create\", \"update\", \"patch\"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: gitlab-deployer namespace: production subjects: - kind: ServiceAccount name: gitlab-deployer namespace: ci-cd roleRef: kind: Role name: deployer apiGroup: rbac.authorization.k8s.io Note the cross-namespace trick. The ServiceAccount lives in ci-cd, the RoleBinding sits in production, and the binding grants just enough to deploy there. The pipeline can ship to prod without holding admin anywhere.\nAuditing what’s already running Patterns help on new workloads. The harder question is what’s already running with too much power. I can’t trust a permission model I can’t inspect, so the inspection is routine.\nWhat one service account may do kubectl auth can-i --list --as=system:serviceaccount:production:api-server # Output: # Resources Non-Resource URLs Verbs # configmaps [] [get watch] # pods [] [get list watch] Who holds cluster-admin # Find all ClusterRoleBindings to cluster-admin kubectl get clusterrolebindings -o json | jq -r ' .items[] | select(.roleRef.name == \"cluster-admin\") | .metadata.name + \": \" + (.subjects // [] | map(.name) | join(\", \")) ' The first time I ran that on a cluster I’d inherited, the list was longer than I’d like to write down.\nEvery binding in a namespace kubectl get rolebindings -n production -o yaml Enforcing it with Kyverno Auditing tells you where you stand today. It does nothing to stop the next person from shipping a pod on the default account next week. For that the rules have to be checked at admission, not written in a wiki nobody reads. I use Kyverno for that.\napiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: require-dedicated-service-account spec: validationFailureAction: Enforce rules: - name: no-default-serviceaccount match: any: - resources: kinds: - Pod validate: message: \"Using 'default' service account is not allowed\" pattern: spec: serviceAccountName: \"!default\" --- apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: disable-automount-by-default spec: validationFailureAction: Enforce rules: - name: disable-automount match: any: - resources: kinds: - Pod validate: message: \"automountServiceAccountToken must be explicitly set to false unless needed\" pattern: spec: automountServiceAccountToken: false A deployment that forgets these settings now gets rejected before it schedules. I don’t have to remember the rule, and neither does the next person.\nAggregated ClusterRoles Once the basics hold, RBAC has a sharper tool worth knowing. Role aggregation composes permissions from labelled pieces.\napiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: monitoring-base labels: rbac.example.com/aggregate-to-monitoring: \"true\" rules: - apiGroups: [\"\"] resources: [\"pods\", \"services\"] verbs: [\"get\", \"list\", \"watch\"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: monitoring-extended labels: rbac.example.com/aggregate-to-monitoring: \"true\" rules: - apiGroups: [\"apps\"] resources: [\"deployments\", \"replicasets\"] verbs: [\"get\", \"list\", \"watch\"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: monitoring aggregationRule: clusterRoleSelectors: - matchLabels: rbac.example.com/aggregate-to-monitoring: \"true\" rules: [] # Rules are automatically filled by aggregation Drop the right label on a new ClusterRole and it folds into the aggregate on its own. Handy when several teams each own a slice of one shared role.\nHumans Everything so far has been about machines. People need RBAC too, and in my experience they ask for more than they need. The built-in view and edit ClusterRoles cover most of it.\nBind groups, not users # Developers can view most things in their namespace apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: developers-view namespace: development subjects: - kind: Group name: developers apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: view # Built-in read-only role apiGroup: rbac.authorization.k8s.io --- # Platform team has edit in all namespaces apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: platform-team-edit subjects: - kind: Group name: platform-team apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: edit # Built-in edit role apiGroup: rbac.authorization.k8s.io Break-glass # Emergency access that's audited heavily apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: emergency-admin annotations: description: \"Emergency access - all usage is audited and reviewed\" subjects: - kind: Group name: emergency-responders apiGroup: rbac.authorization.k8s.io roleRef: kind: ClusterRole name: cluster-admin apiGroup: rbac.authorization.k8s.io Wire this binding to audit logging and an alert, so the moment someone reaches for cluster-admin the team knows. Without the alert it’s a backdoor that happens to have a YAML file.\nWhat I run Put it together and this is what my clusters look like. The one I opened with had one binding, and it was cluster-admin. This is what replaced it.\n# 1. Disable default service account tokens cluster-wide apiVersion: v1 kind: ServiceAccount metadata: name: default namespace: production automountServiceAccountToken: false # 2. Base read-only role for all applications apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: app-base rules: - apiGroups: [\"\"] resources: [\"configmaps\"] verbs: [\"get\", \"watch\"] # 3. Application-specific additions apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: api-server namespace: production rules: - apiGroups: [\"\"] resources: [\"secrets\"] verbs: [\"get\"] resourceNames: [\"api-credentials\", \"db-credentials\"] # 4. Combined binding apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: api-server namespace: production subjects: - kind: ServiceAccount name: api-server namespace: production roleRef: kind: Role name: api-server apiGroup: rbac.authorization.k8s.io Nothing has access until I grant it. Applications stay inside their namespace. Secret access is pinned to named secrets, and no rule carries a wildcard.\nTest it before you trust it RBAC changes are easy to get subtly wrong, and a broken binding fails closed at whatever moment is worst for you. So I never apply a role to production without checking what it grants first.\n# Dry-run: what would this service account be able to do? kubectl auth can-i --list \\ --as=system:serviceaccount:production:api-server # Test specific permission kubectl auth can-i get secrets \\ --as=system:serviceaccount:production:api-server \\ -n production # Test with impersonation kubectl get pods -n production \\ --as=system:serviceaccount:production:api-server Why I bother Every permission you hand out is something you now have to think about. Every over-privileged account is a bigger blast radius than the job needs.\nWith least privilege in place, that bad day stays small. A compromised app can only touch what you let it touch. A misconfiguration breaks one namespace instead of the cluster. The audit log means something, because not everything was allowed to begin with. And you end up knowing what each component needs, which I’ve come to like on its own.\nThat’s what I mean by resilience through access control. One compromise doesn’t have to become all of them.\nThe trade-off is real. Least privilege is more upfront work than a single cluster-admin binding, and you’ll hit a few “why can’t this pod do X” moments before the model settles. I think it’s worth it every time, but go in knowing you’re trading some convenience for a lot of containment.\nEvery new service account on my clusters starts with zero permissions and gets a rule only when I can point at the call that needs it. I still trip over a missing verb now and then. I’ll take that over the cluster I started with, where everything could delete everything.","permalink":"https://tommeurs.nl/en/posts/kubernetes-rbac-least-privilege/","title":"Kubernetes RBAC: Least Privilege in Practice"},{"content":" TL;DR Trivy and Kyverno check an image before it runs. Falco watches what the container does after that, syscall by syscall, and alerts when the behaviour doesn’t match a rule. This post goes from the one thing Falco does to the config I run, with the noise problem in between.\nI scanned my images with Trivy. I enforced policies with Kyverno. My workloads got cryptographic identity through SPIFFE. Three layers of prevention, all green, and for a while that felt like enough.\nThen I asked myself what happens after a pod is running. My scanner checked the image that went in. My admission controller checked the spec at deploy time. Neither of them is watching once the process is executing. A zero-day that lands in a running container arrives after every one of those controls has finished its job.\nThat bothered me more than I’d like to admit. I can’t claim to understand what my cluster does if I’ve no idea what the processes inside it are doing. Prevention tells me what should be true. I wanted something that tells me what is true, right now.\nFalco is that something. It watches the system calls every container makes, file access, network connections, processes spawning, and alerts when the behaviour doesn’t match the rules I gave it. Here’s how I learned it. The one thing it does, then a rule, then my own rules, then what I run and how much of that is tuning.\nThe gap prevention leaves I think of cluster security as three moments in time. Build time, where Trivy scans the image for known vulnerabilities. Deploy time, where Kyverno and the other admission controllers check the spec. And runtime, where something has to watch what the workload does. Falco is the third moment.\nMost setups stop after the first two, and mine did for a long time. They’re cheaper, they fit in the pipeline, and they make a dashboard go green. The trouble is that an attacker who’s already inside a running container doesn’t replay your pipeline. They work with what’s executing.\nHere’s the kind of thing the first two moments never see:\nA container suddenly executing /bin/bash in production, where nothing should ever spawn a shell A process reading /etc/shadow An outbound connection to a crypto mining pool Something writing into /etc/ Kubernetes service account tokens getting read from a pod that has no business touching them None of that shows up in a scan or an admission check, because none of it is configuration. It’s behaviour, and behaviour is the only thing Falco looks at.\nOne thing, before any rules Falco sits below your containers, reads their system calls, and matches each one against a set of conditions. eBPF (or a kernel module) is how it gets that seat:\nKernel Space\nUser Space\nContainer A\nContainer B\nContainer C\neBPF / Kernel Module\n(intercepts syscalls)\nFalco\n(rules engine)\nAlerts\n(stdout, webhook, Kafka...)\nEvery syscall flows through the rules engine. A match becomes an alert. That’s the whole model. The rest of Falco is detail on top of it, and once this picture is in your head the detail reads easily.\nInstalling it I start with the Helm chart and the eBPF driver. eBPF means no kernel module on every node, which matters on Talos and other locked-down hosts:\nhelm repo add falcosecurity https://falcosecurity.github.io/charts helm repo update helm install falco falcosecurity/falco \\ --namespace falco \\ --create-namespace \\ --set driver.kind=ebpf \\ --set falcosidekick.enabled=true I don’t run helm install by hand, though. Everything in my cluster goes through Git, so the real version is an ArgoCD Application pointing at the same chart:\napiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: falco namespace: argocd spec: project: default source: repoURL: https://falcosecurity.github.io/charts chart: falco targetRevision: 4.0.0 helm: values: | driver: kind: ebpf falcosidekick: enabled: true config: slack: webhookurl: \"https://hooks.slack.com/services/xxx\" tty: true destination: server: https://kubernetes.default.svc namespace: falco syncPolicy: automated: prune: true selfHeal: true Reading a rule Falco is as good as its rules and no better. A rule is a small block of YAML, and once you can read one you can read all of them. Here’s the canonical example, a shell spawning inside a container:\n- rule: Terminal shell in container desc: Detect shell being spawned in a container condition: \u003e spawned_process and container and shell_procs and proc.tty != 0 output: \u003e Shell spawned in container (user=%user.name container=%container.name shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline) priority: WARNING tags: [container, shell, mitre_execution] Four parts do the work. The condition decides when to fire, in Falco’s filter language. The output is what lands in the alert, with the % fields filled in at runtime. The priority is the severity, DEBUG up to CRITICAL. The tags group and filter, here mapping to MITRE technique categories.\nRead the condition out loud and it’s nearly English. A process was spawned, inside a container, the process is a shell, and it has a TTY attached. Every other rule reuses that grammar.\nThe rules that ship with it You don’t start from a blank file. Falco ships a large set of community rules that already cover the obvious attacks:\n# View loaded rules kubectl exec -n falco -it falco-xxx -- falco --list Container escapes, privilege escalation, sensitive file reads, suspicious network activity, Kubernetes API abuse. A few of them, so you can see they read the same way as the one above:\n# Detect container escape via mount - rule: Launch Sensitive Mount Container condition: \u003e spawned_process and container and sensitive_mount # Detect reading sensitive files - rule: Read sensitive file untrusted condition: \u003e open_read and sensitive_files and not trusted_containers # Detect crypto mining - rule: Detect outbound connections to crypto mining pools condition: \u003e outbound and fd.sip.name in (cryptomining_pool_domains) Rules that know your app The defaults catch generic attacks. They’ve no idea what your application is supposed to do. That knowledge is yours, and writing it down is where Falco gets interesting:\n# custom-rules.yaml customRules: rules-custom.yaml: |- # Alert when our API server spawns unexpected processes - rule: Unexpected process in api-server desc: Detect processes other than the main app in api-server containers condition: \u003e spawned_process and container.image.repository contains \"api-server\" and not proc.name in (api-server, node, npm) output: \u003e Unexpected process in api-server (command=%proc.cmdline container=%container.name image=%container.image.repository) priority: WARNING # Alert on database connection from unexpected pods - rule: Database connection from non-backend pod desc: Detect connections to PostgreSQL from pods that shouldn't connect condition: \u003e outbound and fd.sport = 5432 and not k8s.pod.label.app in (api-server, worker, migration-job) output: \u003e Unexpected database connection (pod=%k8s.pod.name namespace=%k8s.ns.name dest=%fd.sip) priority: ERROR Custom rules go in through Helm values:\n# values.yaml customRules: rules-custom.yaml: |- - rule: My custom rule ... The filter language I don’t like copying YAML I can’t read, and I’d rather you didn’t either. There’s a handful of field classes, plus macros and lists to glue them together. Know those and every rule above stops being a black box.\nFields # Process fields proc.name # Process name proc.cmdline # Full command line proc.pname # Parent process name proc.exepath # Executable path # Container fields container.name # Container name container.id # Container ID container.image.repository # Image name # File fields fd.name # File descriptor name (file path) fd.directory # Directory of the file # Network fields fd.sip # Server IP fd.sport # Server port fd.cip # Client IP # Kubernetes fields k8s.pod.name # Pod name k8s.ns.name # Namespace k8s.pod.label.app # Pod label Macros Macros are named condition fragments you reuse. Every rule so far leaned on these without saying so:\n- macro: container condition: container.id != host - macro: shell_procs condition: proc.name in (bash, sh, zsh, dash, ksh) - macro: spawned_process condition: evt.type = execve and evt.dir = \u003c - macro: open_read condition: evt.type in (open, openat) and evt.is_open_read = true Lists Lists are named sets of values, referenced instead of repeated:\n- list: sensitive_files items: [/etc/shadow, /etc/sudoers, /etc/pam.d, /root/.ssh] - list: package_managers items: [apt, apt-get, yum, dnf, apk, pip, npm] - list: shell_binaries items: [bash, sh, zsh, csh, tcsh, ksh, dash] Three rules worth their alert With the grammar in hand, here are the three detections I find worth the noise. They map to what an attacker does once they’re in.\nReverse shells - rule: Reverse shell detected desc: Detect reverse shells via common patterns condition: \u003e spawned_process and container and ((proc.name = bash and proc.args contains \"\u003e\u0026\" and proc.args contains \"/dev/tcp\") or (proc.name = nc and proc.args contains \"-e\") or (proc.name = python and proc.args contains \"socket\" and proc.args contains \"subprocess\")) output: \u003e Reverse shell detected (user=%user.name command=%proc.cmdline container=%container.name) priority: CRITICAL tags: [mitre_execution, reverse_shell] Kubernetes secret access - rule: K8s secret accessed from unexpected namespace desc: Detect when secrets are read from non-standard locations condition: \u003e open_read and fd.name startswith \"/var/run/secrets/kubernetes.io\" and not k8s.ns.name in (kube-system, monitoring, vault) output: \u003e K8s secret accessed (file=%fd.name pod=%k8s.pod.name namespace=%k8s.ns.name) priority: WARNING Package installs at runtime - rule: Package manager in container desc: Detect package installations in running containers condition: \u003e spawned_process and container and proc.name in (apt, apt-get, yum, dnf, apk, pip, npm) and not container.image.repository in (allowed_builder_images) output: \u003e Package manager run in container (command=%proc.cmdline container=%container.name image=%container.image.repository) priority: ERROR tags: [mitre_persistence, package_install] A package manager starting up in a running container means someone is building a foothold, or at best a developer debugging in production. Either way I want to know, so it’s an ERROR and not a warning.\nGetting the alerts somewhere you’ll see them Detecting something is pointless if the alert dies in a pod’s stdout, and by default that’s where Falco writes. Falcosidekick fans the alerts out to wherever you look:\nfalcosidekick: enabled: true config: slack: webhookurl: \"https://hooks.slack.com/services/xxx\" minimumpriority: warning prometheus: enabled: true elasticsearch: hostport: \"elasticsearch.logging:9200\" index: \"falco\" alertmanager: hostport: \"http://alertmanager.monitoring:9093\" webhook: address: \"http://security-responder.security:8080/falco\" That config sends warnings and up to Slack, every alert into Prometheus as a metric, everything into Elasticsearch for history, criticals to Alertmanager for paging, and the full stream to a webhook for automated response. One detection, and each destination gets the severity it cares about.\nFolding Falco into Prometheus I already run Prometheus and Grafana for everything else, so I want security events where I watch the rest of the cluster. Falcosidekick exposes metrics, and a ServiceMonitor scrapes them:\n# ServiceMonitor apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: falco-sidekick namespace: monitoring spec: selector: matchLabels: app.kubernetes.io/name: falcosidekick endpoints: - port: http From there a PrometheusRule turns a Falco detection into the same kind of alert as a full disk or a hot CPU:\n# PrometheusRule apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: falco-alerts spec: groups: - name: falco rules: - alert: FalcoCriticalAlert expr: increase(falco_events{priority=\"Critical\"}[5m]) \u003e 0 for: 0m labels: severity: critical annotations: summary: \"Falco critical security event detected\" description: \"{{ $labels.rule }} triggered in {{ $labels.k8s_ns_name }}/{{ $labels.k8s_pod_name }}\" The first week is noise Here’s the trade-off nobody puts on the landing page. Out of the box, the default rules page you for things that are normal in your cluster. The first week I ran Falco, my own config-reload sidecar woke me up. A tool you’ve learned to ignore is a tool you’ve switched off without noticing, so tuning isn’t a step after the install. It’s most of the work.\nTwo moves. Switch off rules that don’t fit, and narrow the macros the rest depend on.\nDisabling and overriding rules # values.yaml falco: rules_file: - /etc/falco/falco_rules.yaml - /etc/falco/falco_rules.local.yaml # Overrides - /etc/falco/rules.d # Custom rules customRules: falco_rules.local.yaml: |- # Disable noisy rule - rule: Terminal shell in container enabled: false # Modify existing rule to exclude certain containers - rule: Read sensitive file untrusted condition: \u003e open_read and sensitive_files and not trusted_containers and not container.image.repository in (my-trusted-app) Tuning macros customRules: falco_rules.local.yaml: |- # Add to trusted containers - macro: trusted_containers condition: \u003e (container.image.repository in ( my-company/trusted-app, my-company/another-app )) # Exclude namespaces from monitoring - macro: user_namespace condition: \u003e k8s.ns.name not in (kube-system, falco, monitoring) Editing trusted_containers beats disabling the rule, because the rule still protects every other container. You’re teaching Falco what normal looks like here instead of blinding it.\nWhat I run Everything stacked together. The eBPF driver, JSON output, tiered alerting, and two rules that say what my own workloads may do. Compare it with the helm install near the top and you can see how much of it is tuning:\ndriver: kind: ebpf ebpf: hostNetwork: true falco: grpc: enabled: true grpc_output: enabled: true json_output: true log_level: info rules_file: - /etc/falco/falco_rules.yaml - /etc/falco/falco_rules.local.yaml - /etc/falco/rules.d falcosidekick: enabled: true replicaCount: 2 config: slack: webhookurl: \"${SLACK_WEBHOOK}\" minimumpriority: error prometheus: enabled: true alertmanager: hostport: \"http://alertmanager.monitoring:9093\" minimumpriority: critical customRules: rules-custom.yaml: |- # Our application-specific rules - rule: Unexpected outbound connection from backend desc: Backend pods should only connect to known services condition: \u003e outbound and k8s.pod.label.app = \"backend\" and not fd.sip.name in (postgres.db, redis.cache, api.internal) output: \u003e Backend made unexpected outbound connection (pod=%k8s.pod.name dest=%fd.sip:%fd.sport) priority: WARNING falco_rules.local.yaml: |- # Tune default rules for our environment - macro: user_known_write_etc_conditions condition: \u003e (container.image.repository = \"my-company/config-manager\") Why each choice. eBPF over the kernel module because it performs better and keeps the host cleaner. JSON output because downstream tools parse it without regex. Tiered alerting so an error nudges Slack and a critical pages someone. And custom rules because only I know what my backend should be talking to.\nAutomated response Falco only detects. What happens next is your call. For most rules I want a human to look. For a CRITICAL I’m fine letting the cluster defend itself, and the webhook receiver from the routing config is where that lives:\n# security-responder deployment apiVersion: apps/v1 kind: Deployment metadata: name: security-responder spec: template: spec: containers: - name: responder image: my-company/security-responder env: - name: SLACK_WEBHOOK valueFrom: secretKeyRef: name: security-responder key: slack-webhook The receiver can be tiny. Kill the pod, let the Deployment recreate it, and post to Slack so a human knows:\n@app.route('/falco', methods=['POST']) def handle_falco_alert(): alert = request.json if alert['priority'] == 'Critical': # Kill the pod immediately pod = alert['output_fields']['k8s.pod.name'] namespace = alert['output_fields']['k8s.ns.name'] # Delete pod (Deployment will recreate it) v1.delete_namespaced_pod(pod, namespace) # Notify send_slack(f\"Killed pod {namespace}/{pod} due to: {alert['rule']}\") return 'ok' Killing a pod off a single alert is a sharp tool and I’m careful with it. Wire a noisy rule to it and you’ve built something that takes down your own cluster. So automation only goes on rules I’ve watched in production long enough to trust. Everything else pages a person.\nShould and is Step back from the YAML and the reason I run Falco is the reason I self-host the rest of my stack. I want to know what my infrastructure is doing, not assume it. Trivy and Kyverno tell me what my cluster should be. Falco tells me what it is. Without it, my honest answer to “what’s running in your cluster right now” was “whatever the image said three weeks ago, probably.”\nThat gap between should and is, between the spec and the running process, is where an attacker works. The three layers I opened with are still green. Falco is the one that’s had something to say since the pods started, and for the first week that was mostly about my own sidecar.","permalink":"https://tommeurs.nl/en/posts/falco-runtime-security/","title":"Runtime Security with Falco: Detect Suspicious Behavior in Your Cluster"},{"content":"How does Service A know that Service B is actually Service B?\nI keep coming back to that question because the usual answer is uncomfortable. For years we trusted network location. Traffic from the right IP was legitimate, end of story. Zero trust took that assumption out back and shot it. Now every service has to prove who it is, every single request, no matter where it sits on the network.\nThat sounds great on a slide. The hard part is the proof. How does a pod actually demonstrate its identity to another pod, without me hand-distributing secrets to everything and then babysitting their rotation forever?\nSPIFFE (Secure Production Identity Framework for Everyone) is a standard for service identity. SPIRE is its production-ready implementation. Together they hand every workload a cryptographic identity automatically, with no static secrets to leak. This post builds up from the smallest idea (a name) to the full thing I run in my own cluster.\nThe thing we keep getting wrong Before SPIFFE , every approach I tried to service authentication had the same shape: bolt on a credential, then spend the rest of my life managing it. Walk through the options and the pattern shows up every time.\nShared secrets\n# Every pod knows the same password env: - name: API_KEY valueFrom: secretKeyRef: name: shared-secret key: api-key Problems:\nOne compromised service exposes all services Rotation requires coordinated updates No identity distinction between services Service accounts with tokens\n# Kubernetes ServiceAccount tokens automountServiceAccountToken: true Better, but:\nTokens are long-lived Work only within Kubernetes Limited attestation capabilities mTLS with static certificates\n# Manual certificate management volumeMounts: - name: certs mountPath: /certs Secure, but:\nCertificate management is operational overhead Rotation requires restarts Cross-cluster identity is complex Every row in that list is me doing work a machine should do. That is the friction I want gone. So let me build the alternative up from nothing, one layer at a time.\nSPIFFE: The Standard The simplest version of identity is a name. SPIFFE starts there and adds exactly two more things on top. Three concepts total.\n1. SPIFFE ID A URI that uniquely identifies a workload:\nspiffe://trust-domain/path/to/workload Examples:\nspiffe://example.com/ns/production/sa/api-server spiffe://example.com/k8s/cluster-a/ns/default/pod/frontend-abc123 spiffe://example.com/aws/account-123/region/eu-west-1/instance/i-abc123 The format stays the same everywhere: Kubernetes, VMs, cloud instances, bare metal. One naming scheme for the whole estate, which already removes a category of “wait, how do we identify things over here” problems.\n2. SVID (SPIFFE Verifiable Identity Document) A name on its own proves nothing. Anyone can claim to be api-server. The SVID is the short-lived credential that backs the claim with crypto. It comes in two flavours:\nX.509 SVID is a certificate with the SPIFFE ID in the SAN\nSubject Alternative Name: URI: spiffe://example.com/ns/production/sa/api-server JWT SVID is a signed JWT token\n{ \"sub\": \"spiffe://example.com/ns/production/sa/api-server\", \"aud\": [\"spiffe://example.com/ns/production/sa/database\"], \"exp\": 1690000000 } 3. Workload API A local API (usually a Unix socket) where workloads fetch their SVIDs:\n/run/spire/sockets/agent.sock This is the part I like most. Workloads never touch private keys directly. They ask the Workload API for an SVID, and key generation plus rotation happens behind that socket. The application code stops caring about certificates entirely, which is one fewer thing for me to get wrong.\nSo: a name, a credential that proves the name, and a socket to fetch it from. That is the whole standard. Now for the machinery that makes it real.\nSPIRE: The Implementation SPIRE is the engine that hands out those SVIDs. Two components, and the split mirrors how trust actually flows.\nSPIRE Server\nCentral authority that issues SVIDs Stores registration entries (which workloads get which identities) Manages trust bundles SPIRE Agent\nRuns on every node Attests workloads (proves they are who they claim) Exposes the Workload API Caches and rotates SVIDs SPIRE Server\nRegistration Entries\nTrust Bundles\nAgent\n(Node1)\nAgent\n(Node2)\nAgent\n(Node3)\nWorkload API\nWorkload API\nWorkload API\nThe server is the brain, the agents are the hands on each node. The agent is also the thing that physically vouches for a workload, which matters when we get to attestation later. For now, install it.\nInstalling SPIRE on Kubernetes The minimum viable install is two Helm commands.\nhelm repo add spiffe https://spiffe.github.io/helm-charts-hardened helm repo update # Install SPIRE CRDs helm install spire-crds spiffe/spire-crds \\ --namespace spire-system \\ --create-namespace # Install SPIRE helm install spire spiffe/spire \\ --namespace spire-system \\ --set global.spire.trustDomain=example.com That gets you running, but I would never leave it as imperative Helm commands. Anything in my cluster lives in Git and gets reconciled, because I want the cluster to be a function of the repo and not of whatever I typed last Tuesday. Here it is as ArgoCD applications:\napiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: spire-crds namespace: argocd spec: project: default source: repoURL: https://spiffe.github.io/helm-charts-hardened chart: spire-crds targetRevision: 0.4.0 destination: server: https://kubernetes.default.svc namespace: spire-system syncPolicy: automated: prune: true --- apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: spire namespace: argocd spec: project: default source: repoURL: https://spiffe.github.io/helm-charts-hardened chart: spire targetRevision: 0.21.0 helm: values: | global: spire: trustDomain: example.com clusterName: production spire-server: replicaCount: 3 dataStore: sql: databaseType: postgres spire-agent: socketPath: /run/spire/sockets/agent.sock destination: server: https://kubernetes.default.svc namespace: spire-system syncPolicy: automated: prune: true Workload Registration SPIRE will not hand an identity to a workload it has never heard of. Something has to say “a pod matching this shape should get this SPIFFE ID.” You can do that by hand, but the Kubernetes workload registrar turns it into a label-driven rule, which is the version I run:\napiVersion: spire.spiffe.io/v1alpha1 kind: ClusterSPIFFEID metadata: name: api-server spec: spiffeIDTemplate: \"spiffe://{{ .TrustDomain }}/ns/{{ .PodMeta.Namespace }}/sa/{{ .PodSpec.ServiceAccountName }}\" podSelector: matchLabels: app: api-server namespaceSelector: matchLabels: spire-enabled: \"true\" Any pod matching the selector in a labelled namespace gets the SPIFFE ID. New deployment, same labels, instant identity, no ticket to me. That is the friction reduction I was after at the start.\nUsing SVIDs in Applications Now the practical question: your app has an identity available on a socket, so how does it actually use it? There are three ways in, sorted roughly by how much you want to touch your application code.\nOption 1: SPIFFE Helper Sidecar The least invasive option. A sidecar fetches SVIDs and writes them to disk as plain PEM files, and your app reads certificates the way it already does:\napiVersion: apps/v1 kind: Deployment metadata: name: api-server spec: template: spec: containers: - name: app image: my-app:v1.0.0 volumeMounts: - name: spiffe mountPath: /spiffe readOnly: true env: - name: TLS_CERT value: /spiffe/svid.pem - name: TLS_KEY value: /spiffe/svid_key.pem - name: CA_BUNDLE value: /spiffe/bundle.pem - name: spiffe-helper image: ghcr.io/spiffe/spiffe-helper:latest volumeMounts: - name: spiffe mountPath: /spiffe - name: spire-agent-socket mountPath: /run/spire/sockets volumes: - name: spiffe emptyDir: {} - name: spire-agent-socket hostPath: path: /run/spire/sockets type: Directory Option 2: Native SPIFFE Library If you own the code, talking to the Workload API directly is cleaner. The library handles rotation in memory, so there is no file on disk to leak or to go stale:\nimport ( \"github.com/spiffe/go-spiffe/v2/workloadapi\" ) func main() { ctx := context.Background() // Connect to Workload API source, err := workloadapi.NewX509Source(ctx) if err != nil { log.Fatal(err) } defer source.Close() // Get SVID svid, err := source.GetX509SVID() if err != nil { log.Fatal(err) } fmt.Printf(\"SPIFFE ID: %s\\n\", svid.ID) // Use for mTLS tlsConfig := tlsconfig.MTLSClientConfig(source, source, tlsconfig.AuthorizeID( spiffeid.RequireID(spiffeid.MustParseSpiffeID(\"spiffe://example.com/ns/production/sa/database\")), )) } Option 3: Envoy Sidecar with SDS When you cannot change the app and you want mTLS handled entirely outside it, put Envoy in front. It pulls SVIDs over the Secret Discovery Service and terminates TLS for you:\napiVersion: apps/v1 kind: Deployment spec: template: spec: containers: - name: app image: my-app:v1.0.0 ports: - containerPort: 8080 - name: envoy image: envoyproxy/envoy:v1.28-latest volumeMounts: - name: envoy-config mountPath: /etc/envoy - name: spire-agent-socket mountPath: /run/spire/sockets Envoy config for SDS:\nstatic_resources: clusters: - name: spire_agent connect_timeout: 1s type: STATIC http2_protocol_options: {} load_assignment: cluster_name: spire_agent endpoints: - lb_endpoints: - endpoint: address: pipe: path: /run/spire/sockets/agent.sock - name: backend connect_timeout: 1s type: STRICT_DNS transport_socket: name: envoy.transport_sockets.tls typed_config: \"@type\": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext common_tls_context: tls_certificate_sds_secret_configs: - name: \"spiffe://example.com/ns/default/sa/api-server\" sds_config: api_config_source: api_type: GRPC grpc_services: - envoy_grpc: cluster_name: spire_agent validation_context_sds_secret_config: name: \"spiffe://example.com\" sds_config: api_config_source: api_type: GRPC grpc_services: - envoy_grpc: cluster_name: spire_agent Service Mesh Integration If you already run a service mesh, you do not have to bolt SPIRE on beside it. The mesh can use SPIRE as the thing that issues identities, so you get one source of truth instead of two overlapping certificate systems.\nIstio Istio can use SPIRE as its identity provider:\napiVersion: install.istio.io/v1alpha1 kind: IstioOperator spec: meshConfig: trustDomain: example.com values: global: caAddress: spiffe-csi-driver:443 Linkerd Linkerd’s identity system is SPIFFE-compatible. Integration is straightforward with the identity controller.\nCross-Cluster Federation Here is where the single naming scheme pays off. Two clusters with separate trust domains can federate, exchange trust bundles, and then verify each other’s workloads. No shared CA, no copying certs around by hand:\nCluster A (production)\nglobal: spire: trustDomain: production.example.com federation: enabled: true bundleEndpoint: address: spire-bundle.production.example.com port: 8443 trustDomainBundles: - endpointURL: https://spire-bundle.staging.example.com:8443 trustDomain: staging.example.com Cluster B (staging)\nglobal: spire: trustDomain: staging.example.com federation: enabled: true bundleEndpoint: address: spire-bundle.staging.example.com port: 8443 trustDomainBundles: - endpointURL: https://spire-bundle.production.example.com:8443 trustDomain: production.example.com Now services in staging can verify identities from production and the other way round.\nAttestation Methods Time to answer the question I parked earlier: how does SPIRE know a pod claiming to be api-server actually is api-server? It cannot just take the pod’s word for it, or the whole thing is theatre. The answer is attestation, and it happens in two stages.\nNode Attestation\n# How nodes prove their identity to the server nodeAttestor: k8sPsat: enabled: true cluster: production The node proves it’s a legitimate Kubernetes node using projected service account tokens.\nWorkload Attestation\n# How workloads prove their identity to the agent workloadAttestors: k8s: enabled: true The agent verifies the workload’s container using the Kubernetes API. Two checks, chained: the node proves itself to the server, then the agent on that node proves the workload to itself. No step relies on a secret the workload could have copied from somewhere else.\nAuthorization with OPA One honest caveat. SPIRE tells you who is calling. It does not tell you whether they are allowed to. Identity and authorization are different jobs, and SPIRE only does the first. For the second I reach for Kyverno or OPA , feeding the SPIFFE ID straight into the policy:\n# OPA policy using SPIFFE IDs package authz default allow = false allow { # Allow api-server to access database input.source == \"spiffe://example.com/ns/production/sa/api-server\" input.destination == \"spiffe://example.com/ns/production/sa/database\" } allow { # Allow frontend to access api-server input.source == \"spiffe://example.com/ns/production/sa/frontend\" input.destination == \"spiffe://example.com/ns/production/sa/api-server\" } My Production Setup Pulling all the layers together, here is the actual config I run. Nothing exotic, just the defaults tightened to my taste:\nglobal: spire: trustDomain: infrastructure.internal clusterName: production spire-server: replicaCount: 3 dataStore: sql: databaseType: postgres connectionString: \"postgres://spire:password@postgres:5432/spire?sslmode=require\" nodeAttestor: k8sPsat: enabled: true serviceAccountAllowList: - spire-system:spire-agent ca: keyType: ec-p256 ttl: 24h spire-agent: socketPath: /run/spire/sockets/agent.sock workloadAttestors: k8s: enabled: true sds: enabled: true defaultBundleName: \"null\" defaultAllBundlesName: ROOTCA spiffe-csi-driver: enabled: true Key decisions, and why:\nPostgreSQL backend keeps server state persistent so I can run three replicas and survive a node dying EC-P256 keys sit at a sensible point between security and CPU cost 24h TTL means SVIDs rotate often, so a leaked credential is worthless by tomorrow CSI driver delivers the SVID as a volume, which is cleaner than poking at host paths Troubleshooting When something goes quiet, these are the three commands I run before anything else.\nCheck agent health kubectl exec -n spire-system -it spire-agent-xxx -- \\ /opt/spire/bin/spire-agent healthcheck List registered workloads kubectl exec -n spire-system -it spire-server-0 -- \\ /opt/spire/bin/spire-server entry show Verify workload identity kubectl exec -it my-pod -- \\ cat /run/spire/sockets/agent.sock # Check socket exists kubectl exec -it my-pod -- \\ openssl x509 -in /spiffe/svid.pem -text -noout | grep URI Why This Matters Go back to the start: a name, a credential, a socket. We built up from that to a system where every service in my cluster proves who it is with crypto, every request, and I never touch a certificate by hand.\nStatic secrets were always a liability. They leak, they are painful to rotate, and worst of all they never answer the only question that matters in zero trust: who is making this request? SPIFFE and SPIRE answer it with evidence instead of a password.\nWhat I get out of it:\nAutomatic identity, so workloads are named without me handing out secrets Short-lived credentials, so a stolen SVID expires before it is useful Cryptographic proof, so an identity cannot be forged or replayed One model everywhere, so a cluster, a VM, and a cloud instance all speak the same language This matters to me for the same reason self-hosting does. I want to understand the mechanism I am trusting, all the way down, and I want it to keep working when one component falls over. Workload identity that I can reason about is the difference between a breach staying contained and a breach owning everything. That is worth the setup cost.\nIdentity should be automatic and cryptographically verifiable. SPIFFE and SPIRE give every workload an identity it never has to manage, and that an attacker cannot steal.","permalink":"https://tommeurs.nl/en/posts/spiffe-spire-zero-trust/","title":"SPIFFE and SPIRE: Zero Trust Service Identity"},{"content":"I used to keep a wiki page titled “Cluster conventions”. Resource limits on everything. No :latest tags. No deploys in the default namespace. It was a good page. Nobody read it. Six months in, half the cluster broke those rules and I only found out when something fell over.\nA rule that lives in a doc is a suggestion. A rule the API server refuses to accept is governance. That gap is the whole reason this post exists.\nKubernetes hands every team enormous freedom. Deploy anything, configure it however you like. Without guardrails that freedom turns into a slow accumulation of mess you discover at the worst possible moment. I want the cluster to enforce my standards so I don’t have to remember them, and so I can actually understand what’s running instead of hoping people followed the wiki.\nKyverno is a policy engine for Kubernetes. It validates, mutates, and generates resources based on policies you write as plain Kubernetes-native YAML. The rest of this post builds up from the smallest possible policy to the full set I run on every cluster, so you can stop wherever you have enough.\nWhy Kyverno and not OPA You have options here: Open Policy Agent (OPA) with Gatekeeper, Kyverno, and Kubewarden. I landed on Kyverno for a few honest reasons:\nNative YAML. No Rego, no second language to keep in my head. Validation, mutation, and generation in one tool. I don’t want three different things for three different jobs. Declarative. Policies are just Kubernetes resources, so they live in Git next to everything else. GitOps-friendly. ArgoCD syncs them like any other manifest. OPA /Gatekeeper is genuinely powerful, and Rego buys you expressiveness Kyverno can’t match for the gnarliest rules. The trade-off is real: you learn a new language before you write your first useful policy. For my homelab and most clusters I touch, that cost isn’t worth it. If you already know YAML, Kyverno lets you write something useful in the next five minutes.\nThe three things a policy can do Before any YAML, hold onto these three verbs. Everything Kyverno does is one of them.\nValidate. Block resources that don’t meet your criteria.\n\"You can't deploy without resource limits\" Mutate. Modify resources on the way in.\n\"All pods get this label automatically\" Generate. Create new resources when others appear.\n\"Every namespace gets a default NetworkPolicy\" That’s the whole mental model. Validate is where almost everyone starts, so that’s where we’ll start too.\nInstalling Kyverno helm repo add kyverno https://kyverno.github.io/kyverno/ helm repo update helm install kyverno kyverno/kyverno \\ --namespace kyverno \\ --create-namespace For GitOps with ArgoCD:\napiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: kyverno namespace: argocd spec: project: default source: repoURL: https://kyverno.github.io/kyverno/ chart: kyverno targetRevision: 3.1.4 helm: values: | admissionController: replicas: 3 backgroundController: replicas: 2 cleanupController: replicas: 2 reportsController: replicas: 2 destination: server: https://kubernetes.default.svc namespace: kyverno syncPolicy: automated: prune: true selfHeal: true The simplest possible policy: require a label Here’s the smallest useful thing you can write. It requires every pod to carry a team label, and nothing more:\napiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: require-labels spec: validationFailureAction: Enforce rules: - name: require-team-label match: any: - resources: kinds: - Pod validate: message: \"The label 'team' is required.\" pattern: metadata: labels: team: \"?*\" Try to create a pod without the label and the API server slams the door:\nkubectl run test --image=nginx # Error: The label 'team' is required. That’s it. One CRD, and the convention is now load-bearing instead of decorative. Notice nobody had to remember anything.\nEnforce versus Audit: don’t break prod on day one Before you go further, the one knob that saves you from yourself. validationFailureAction has two settings.\nEnforce blocks non-compliant resources outright. This is what you want in production, eventually.\nvalidationFailureAction: Enforce Audit lets the resource through but records the violation in a report.\nvalidationFailureAction: Audit I learned this one the hard way. Roll out a strict policy in Enforce mode against a cluster that’s been ignoring your wiki for months, and you’ll wedge half your deploys at once. Start every new policy in Audit, look at what would have failed, fix the real offenders, then flip to Enforce. Graceful degradation applies to your own rollout too.\nLayer 1: the validation policies that earn their keep A label requirement is a warm-up. These are the rules that actually keep a cluster sane. Each one is still a single ClusterPolicy, just pointed at a more useful thing.\nRequire resource limits apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: require-resource-limits spec: validationFailureAction: Enforce rules: - name: require-limits match: any: - resources: kinds: - Pod validate: message: \"CPU and memory limits are required.\" pattern: spec: containers: - resources: limits: memory: \"?*\" cpu: \"?*\" Disallow privileged containers apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: disallow-privileged spec: validationFailureAction: Enforce rules: - name: deny-privileged match: any: - resources: kinds: - Pod validate: message: \"Privileged containers are not allowed.\" pattern: spec: containers: - securityContext: privileged: false Disallow the latest tag apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: disallow-latest-tag spec: validationFailureAction: Enforce rules: - name: require-image-tag match: any: - resources: kinds: - Pod validate: message: \"Using ':latest' tag is not allowed. Specify a version tag.\" pattern: spec: containers: - image: \"!*:latest\" Restrict image registries apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: restrict-registries spec: validationFailureAction: Enforce rules: - name: allowed-registries match: any: - resources: kinds: - Pod validate: message: \"Images must come from approved registries.\" pattern: spec: containers: - image: \"registry.example.com/* | gcr.io/my-project/*\" Layer 2: mutation, or fixing instead of nagging Validation is the bouncer who turns people away. Mutation is the host who quietly fixes their outfit at the door. Instead of rejecting a resource for missing a label or a security setting, Kyverno can just add it. Less friction for whoever’s deploying, same end state for the cluster.\nAdd default labels apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: add-default-labels spec: rules: - name: add-managed-by match: any: - resources: kinds: - Pod - Deployment - Service mutate: patchStrategicMerge: metadata: labels: managed-by: kyverno environment: \"{{request.namespace}}\" Add a default security context apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: add-security-context spec: rules: - name: add-run-as-nonroot match: any: - resources: kinds: - Pod mutate: patchStrategicMerge: spec: securityContext: runAsNonRoot: true seccompProfile: type: RuntimeDefault Add image pull secrets apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: add-imagepullsecret spec: rules: - name: add-registry-secret match: any: - resources: kinds: - Pod mutate: patchStrategicMerge: spec: imagePullSecrets: - name: registry-credentials Layer 3: generation, the one that surprised me This is the verb I underused for the longest time, and it’s the one that changed how I think about cluster defaults. Generation creates brand-new resources in response to other resources appearing. Make a namespace, and Kyverno can drop a NetworkPolicy and a ResourceQuota into it before anyone deploys a single pod.\nA default-deny NetworkPolicy in every namespace apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: generate-default-networkpolicy spec: rules: - name: generate-deny-all match: any: - resources: kinds: - Namespace exclude: any: - resources: namespaces: - kube-system - kyverno generate: apiVersion: networking.k8s.io/v1 kind: NetworkPolicy name: default-deny namespace: \"{{request.object.metadata.name}}\" data: spec: podSelector: {} policyTypes: - Ingress - Egress Every new namespace starts locked down. Teams have to explicitly allow the traffic they need. That’s zero trust as a default rather than a checklist item someone forgets, and it shrinks the blast radius when something does get popped.\nA default ResourceQuota per namespace apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: generate-resourcequota spec: rules: - name: generate-quota match: any: - resources: kinds: - Namespace exclude: any: - resources: namespaces: - kube-system - kyverno generate: apiVersion: v1 kind: ResourceQuota name: default-quota namespace: \"{{request.object.metadata.name}}\" data: spec: hard: requests.cpu: \"4\" requests.memory: 8Gi limits.cpu: \"8\" limits.memory: 16Gi persistentvolumeclaims: \"10\" Variables: making policies context-aware Hardcoded values get you far, but the good stuff comes from reading the request. Kyverno supports JMESPath expressions so a policy can react to who’s deploying, into which namespace, at what time:\napiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: add-namespace-label spec: rules: - name: add-ns-to-pods match: any: - resources: kinds: - Pod mutate: patchStrategicMerge: metadata: labels: namespace: \"{{request.namespace}}\" created-at: \"{{time_now_utc()}}\" Useful variables:\n{{request.namespace}} is the namespace of the resource {{request.object.metadata.name}} is the name of the resource {{request.userInfo.username}} is the user who made the request {{time_now_utc()}} is the current timestamp Excluding the resources you can’t touch One gotcha that bites everyone: apply a strict policy cluster-wide and you’ll break kube-system or Kyverno itself. System components don’t follow your rules, and they shouldn’t have to. Exclude them explicitly:\napiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: require-labels spec: rules: - name: require-team-label match: any: - resources: kinds: - Pod exclude: any: - resources: namespaces: - kube-system - kube-public - kyverno - subjects: - kind: ServiceAccount name: system:* Policy reports: seeing the state of the cluster Enforcement is half the value. The other half is visibility. Kyverno writes policy reports that show exactly what’s passing and what isn’t, which means I can answer “is this cluster compliant right now?” without trusting anyone’s memory:\nkubectl get policyreport -A kubectl get clusterpolicyreport Example report:\napiVersion: wgpolicyk8s.io/v1alpha2 kind: PolicyReport metadata: name: polr-ns-default namespace: default results: - message: \"validation rule 'require-team-label' passed.\" policy: require-labels result: pass source: kyverno - message: \"validation rule 'require-limits' failed.\" policy: require-resource-limits result: fail source: kyverno Integrate with Prometheus for alerting on policy violations:\n# ServiceMonitor for Kyverno metrics apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: kyverno spec: selector: matchLabels: app.kubernetes.io/name: kyverno endpoints: - port: metrics Don’t write everything yourself: curated policy sets You don’t have to reinvent the security baseline. Kyverno maintains curated policy collections, including ready-made implementations of the Kubernetes Pod Security Standards:\n# Install Pod Security Standards (Baseline) kubectl apply -f https://raw.githubusercontent.com/kyverno/policies/main/pod-security/baseline/ # Install Pod Security Standards (Restricted) kubectl apply -f https://raw.githubusercontent.com/kyverno/policies/main/pod-security/restricted/ These give you the Baseline and Restricted Pod Security Standards as Kyverno policies, no Rego, no hand-rolling.\nThe full picture: what I actually run Remember that single label policy at the top? Here’s where it ends up. This is the core set I drop on every cluster I build, all in Enforce mode after an Audit shakedown: readiness probes required, no deploys in default, resource requests mandatory.\n# require-probes.yaml apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: require-probes spec: validationFailureAction: Enforce rules: - name: require-readiness-probe match: any: - resources: kinds: - Deployment exclude: any: - resources: namespaces: - kube-system - kyverno validate: message: \"Readiness probe is required for all deployments.\" pattern: spec: template: spec: containers: - readinessProbe: \"?*\": \"?*\" --- # disallow-default-namespace.yaml apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: disallow-default-namespace spec: validationFailureAction: Enforce rules: - name: deny-default match: any: - resources: kinds: - Pod - Deployment - Service namespaces: - default validate: message: \"Using the 'default' namespace is not allowed.\" deny: {} --- # require-requests.yaml apiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: require-requests spec: validationFailureAction: Enforce rules: - name: require-cpu-memory-requests match: any: - resources: kinds: - Pod exclude: any: - resources: namespaces: - kube-system validate: message: \"CPU and memory requests are required.\" pattern: spec: containers: - resources: requests: memory: \"?*\" cpu: \"?*\" Test policies before they touch the cluster Policies are code, so I treat them like code. The Kyverno CLI lets me check a policy against a sample resource on my laptop before anything reaches a real API server:\n# Install CLI brew install kyverno # Test policy against a resource kyverno apply policy.yaml --resource pod.yaml # Test all policies in a directory kyverno apply ./policies/ --resource ./manifests/ Create a test suite:\n# test/values.yaml policies: - policy.yaml resources: - resources/valid-pod.yaml - resources/invalid-pod.yaml results: - policy: require-labels resource: valid-pod.yaml result: pass - policy: require-labels resource: invalid-pod.yaml result: fail Run tests:\nkyverno test ./test/ Background scanning: catching what’s already there Admission control only sees new requests. But your cluster is full of pods that were created before you wrote any of this. Turn on background scanning and Kyverno audits the resources already running:\napiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: audit-existing spec: validationFailureAction: Audit background: true # Enable background scanning rules: - name: check-labels match: any: - resources: kinds: - Pod validate: message: \"Label 'team' is missing.\" pattern: metadata: labels: team: \"?*\" Now you get reports for every pod in the cluster, not just the ones created after you showed up. It’s the difference between guarding the door and actually knowing what’s in the building.\nExceptions, and why they should sting a little Sometimes a workload genuinely needs to break a rule. node-exporter wants a privileged container, and that’s fine. Kyverno has a first-class object for it so you don’t water down the policy itself:\napiVersion: kyverno.io/v1 kind: PolicyException metadata: name: allow-privileged-monitoring namespace: monitoring spec: exceptions: - policyName: disallow-privileged ruleNames: - deny-privileged match: any: - resources: kinds: - Pod namespaces: - monitoring names: - node-exporter* Write down why each exception exists, ideally in the manifest itself. A PolicyException is a named, version-controlled hole in your defenses, which is exactly how I want my holes: visible. If you find yourself adding a third one this month, the policy is probably wrong, not the workloads.\nCatch violations at merge, not at deploy The best place to reject a bad manifest is in a merge request, long before it reaches a cluster. Run the same CLI in your pipeline:\n# .gitlab-ci.yml validate-policies: stage: test image: ghcr.io/kyverno/kyverno-cli:latest script: - kyverno apply ./policies/ --resource ./k8s/ rules: - if: $CI_MERGE_REQUEST_ID Now a developer gets the feedback in their MR pipeline, with the rest of their tests, while the change is still cheap to fix.\nWhat this buys me Go back to that wiki page I started with. Here’s what changed when the rules moved out of the doc and into Kyverno.\nWithout a policy engine, standards live in wikis nobody reads, reviews catch problems inconsistently depending on who’s looking, and non-compliance piles up quietly until something breaks. With Kyverno, the standards are code in Git, enforcement is automatic and immediate, and every violation shows up in a report I can actually query.\nThis is low-friction governance. The point isn’t that it’s stricter, it’s that the strictness costs nobody any attention. I don’t ask developers to memorise a convention page, and I don’t have to police it myself. The cluster remembers the rules so my brain doesn’t have to, which is the whole reason I build systems this way.\nThe best policies are the ones you stop thinking about. Mine have been running quietly for months, and the only time I notice them is when one stops a mistake I would otherwise have shipped.","permalink":"https://tommeurs.nl/en/posts/kyverno-policies-governance/","title":"Kyverno Policies: Governance as Code for Kubernetes"},{"content":"The first time I ran kubectl get secret myapp -o yaml and base64-decoded the value, I felt my stomach drop. There was my database password, sitting in etcd, readable by anyone who could reach the API with the right RBAC. Kubernetes Secrets are not secrets. They’re base64-encoded plain text with a fancy name. That’s the default, and it’s the thing nobody warns you about on day one.\nEvery cloud provider has a fix for sale. AWS has Secrets Manager, Google has Secret Manager, Azure has Key Vault. They work. The catch shows up later: the day you need to migrate, the day you want to know exactly what happens to a secret after you write it, the day you realise your most sensitive data lives in a system you can’t inspect.\nHashiCorp Vault is the self-hosted answer. You run it, you control it, and when it breaks at 2am you can actually reason about why.\nThis post builds Vault up one layer at a time. Start with the smallest thing that works, then add real-world usage, then the advanced bits. Stop reading wherever you have enough.\nWhy Self-Hosted Secrets? A cloud KMS hands you a few problems wrapped as features. Your secrets live in a system you don’t operate, so you can’t audit what happens inside it. Migration is brutal because secrets are the stickiest lock-in there is. And you pay per secret, per request, forever.\nSelf-hosted Vault flips each of those. You see exactly how secrets are stored and accessed. There’s no vendor lock-in on your most critical data. It runs identically across clouds, on-prem, and the edge, so the same config works everywhere. Your secrets never leave infrastructure you control.\nThat’s the sovereignty angle. As I wrote in Sovereign Infrastructure, black boxes feel like splinters, and secrets are the one place where a splinter can sink your whole platform. The trade-off is real: you now operate Vault, including unsealing and backups. I’ll be honest about that cost throughout.\nVault Architecture Basics Before any commands, three concepts. Once these click, the rest of Vault stops feeling like magic.\nSecrets Engines are where secrets live.\nkv (key-value) for static secrets database for dynamic database credentials pki for certificate generation Auth Methods are how clients prove who they are.\nkubernetes for pods token for direct access userpass, ldap, oidc for humans Policies decide what an authenticated client is allowed to touch.\nVault\nSecrets Engines\nAuth Methods\nKV\nkubernetes\nPolicies\nuserpass\nPKI\nDB\nThe Simplest Version: Installing Vault Start with the smallest install that runs. The official Helm chart gets you there.\nhelm repo add hashicorp https://helm.releases.hashicorp.com helm repo update helm install vault hashicorp/vault \\ --namespace vault \\ --create-namespace \\ --set server.ha.enabled=true \\ --set server.ha.replicas=3 I don’t actually install anything by hand though. Everything goes through GitOps with ArgoCD, so Vault is declared once and the cluster reconciles it:\napiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: vault namespace: argocd spec: project: default source: repoURL: https://helm.releases.hashicorp.com chart: vault targetRevision: 0.27.0 helm: values: | server: ha: enabled: true replicas: 3 raft: enabled: true dataStorage: size: 10Gi storageClass: longhorn injector: enabled: true destination: server: https://kubernetes.default.svc namespace: vault syncPolicy: automated: prune: true selfHeal: true Initializing and Unsealing Here’s the part that trips up everyone new to Vault. Vault starts sealed. Even with the pods running, it won’t serve a single secret until you unseal it. This is on purpose: the encryption key that protects everything is itself encrypted, and you have to provide enough key shares to reconstruct it. Initialize first:\nkubectl exec -it vault-0 -n vault -- vault operator init This spits out two things: 5 unseal keys, and an initial root token. The keys are Shamir shares, so by default you need any 3 of the 5 to unseal.\nStore these somewhere safe and offline. Lose the unseal keys and you lose access to every secret Vault holds. There’s no recovery, no support ticket, no reset link. I keep mine split across separate offline locations, because the whole point is that no single compromise hands someone the keys to everything.\nUnseal each pod, feeding it 3 of the 5 keys one at a time:\nkubectl exec -it vault-0 -n vault -- vault operator unseal \u003ckey1\u003e kubectl exec -it vault-0 -n vault -- vault operator unseal \u003ckey2\u003e kubectl exec -it vault-0 -n vault -- vault operator unseal \u003ckey3\u003e Repeat for vault-1 and vault-2. Yes, by hand, three times.\nLayer 1: Auto-Unseal The manual dance is fine for a demo. In production it’s a trap. Every pod restart, every node reboot, every chaos event leaves Vault sealed and your apps unable to read secrets until a human shows up with the keys. For a homelab that’s supposed to survive me being asleep, that’s a non-starter.\nAuto-unseal solves it by handing the unsealing job to a separate key, either another Vault’s transit engine or a cloud KMS:\n# Using another Vault for transit auto-unseal server: ha: enabled: true config: | seal \"transit\" { address = \"https://vault-primary.example.com:8200\" disable_renewal = \"false\" key_name = \"autounseal\" mount_path = \"transit/\" tls_skip_verify = \"false\" } Or with a cloud KMS. Using cloud here is a deliberate, narrow exception: one tiny key whose only job is to wrap Vault’s master key, not your actual secrets. The blast radius is small and the resilience win is large, so I’ll take that trade:\nserver: config: | seal \"awskms\" { region = \"eu-west-1\" kms_key_id = \"alias/vault-unseal\" } Enabling the KV Secrets Engine With Vault unsealed, you can finally store something. The kv engine is the simplest one and a good place to start. Log in, enable it, write a secret:\n# Login with root token kubectl exec -it vault-0 -n vault -- vault login # Enable KV v2 secrets engine kubectl exec -it vault-0 -n vault -- vault secrets enable -path=secret kv-v2 # Create a secret kubectl exec -it vault-0 -n vault -- vault kv put secret/myapp/config \\ database_url=\"postgres://user:pass@db:5432/myapp\" \\ api_key=\"super-secret-key\" Kubernetes Authentication A secret in Vault is useless if your pods can’t reach it. The clean way to grant access is the Kubernetes auth method, which lets a pod prove its identity using its own service account token. No long-lived static credential baked into an image, no shared password. Vault verifies the token against the API server and decides whether to issue a short-lived Vault token in return.\nEnable and configure it:\n# Enable Kubernetes auth kubectl exec -it vault-0 -n vault -- vault auth enable kubernetes # Configure it kubectl exec -it vault-0 -n vault -- vault write auth/kubernetes/config \\ kubernetes_host=\"https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT\" Then bind a role to a specific service account in a specific namespace. This is where least privilege starts: only the myapp service account in default gets this role, and only for an hour at a time.\nkubectl exec -it vault-0 -n vault -- vault write auth/kubernetes/role/myapp \\ bound_service_account_names=myapp \\ bound_service_account_namespaces=default \\ policies=myapp-policy \\ ttl=1h Creating Policies The role above referenced myapp-policy, which doesn’t exist yet. Policies are where you spell out exactly what a client may read or list, path by path. Default-deny is baked in: if a path isn’t granted, it’s forbidden. This one lets the app read its own config and nothing else.\n# myapp-policy.hcl path \"secret/data/myapp/*\" { capabilities = [\"read\"] } path \"secret/metadata/myapp/*\" { capabilities = [\"list\"] } Apply it:\nkubectl exec -it vault-0 -n vault -- vault policy write myapp-policy - \u003c\u003cEOF path \"secret/data/myapp/*\" { capabilities = [\"read\"] } path \"secret/metadata/myapp/*\" { capabilities = [\"list\"] } EOF Layer 2: Injecting Secrets into Pods So far you’ve been running vault commands by hand. Real workloads need secrets delivered automatically. The Vault Agent Injector does this with nothing but pod annotations: it watches for them, adds an init container that authenticates, and drops the rendered secret onto a file your app reads. No SDK, no Vault-aware code in your application.\napiVersion: v1 kind: ServiceAccount metadata: name: myapp --- apiVersion: apps/v1 kind: Deployment metadata: name: myapp spec: selector: matchLabels: app: myapp template: metadata: labels: app: myapp annotations: vault.hashicorp.com/agent-inject: \"true\" vault.hashicorp.com/role: \"myapp\" vault.hashicorp.com/agent-inject-secret-config: \"secret/data/myapp/config\" vault.hashicorp.com/agent-inject-template-config: | {{- with secret \"secret/data/myapp/config\" -}} export DATABASE_URL=\"{{ .Data.data.database_url }}\" export API_KEY=\"{{ .Data.data.api_key }}\" {{- end }} spec: serviceAccountName: myapp containers: - name: app image: myapp:v1.0.0 command: [\"/bin/sh\", \"-c\"] args: - source /vault/secrets/config \u0026\u0026 ./myapp The injector:\nAdds an init container that authenticates with Vault Writes secrets to /vault/secrets/config Your app reads them at startup External Secrets Operator Alternative The injector is great, but it couples your pods to Vault’s annotation dance and mounts secrets as files, which not every app wants. External Secrets Operator takes a different route. It syncs Vault secrets into native Kubernetes Secrets, so your workloads consume them the boring, standard way through envFrom or valueFrom. I reach for this when an app expects plain environment variables and I don’t want to teach it anything about Vault.\napiVersion: external-secrets.io/v1beta1 kind: SecretStore metadata: name: vault-backend spec: provider: vault: server: \"http://vault.vault:8200\" path: \"secret\" version: \"v2\" auth: kubernetes: mountPath: \"kubernetes\" role: \"myapp\" --- apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: myapp-secrets spec: refreshInterval: 1h secretStoreRef: name: vault-backend kind: SecretStore target: name: myapp-secrets creationPolicy: Owner data: - secretKey: DATABASE_URL remoteRef: key: myapp/config property: database_url - secretKey: API_KEY remoteRef: key: myapp/config property: api_key This creates a regular Kubernetes Secret that your pods can use normally:\napiVersion: apps/v1 kind: Deployment spec: template: spec: containers: - name: app envFrom: - secretRef: name: myapp-secrets Dynamic Database Credentials This is where Vault stops being a fancy password store and starts being something you can’t easily replicate with a cloud KMS. Static secrets sit there until someone rotates them, which is to say, until someone forgets. Dynamic secrets flip the model: Vault creates a brand new database user on demand, hands it to one pod, and deletes it when the lease expires. Nothing is shared, nothing is permanent.\nEnable the database engine, point it at Postgres, and define a role that knows how to mint users:\n# Enable database secrets engine kubectl exec -it vault-0 -n vault -- vault secrets enable database # Configure PostgreSQL connection kubectl exec -it vault-0 -n vault -- vault write database/config/mydb \\ plugin_name=postgresql-database-plugin \\ connection_url=\"postgresql://{{username}}:{{password}}@postgres.default:5432/myapp\" \\ allowed_roles=\"myapp-db\" \\ username=\"vault_admin\" \\ password=\"vault_admin_password\" # Create role that generates credentials kubectl exec -it vault-0 -n vault -- vault write database/roles/myapp-db \\ db_name=mydb \\ creation_statements=\"CREATE ROLE \\\"{{name}}\\\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO \\\"{{name}}\\\";\" \\ default_ttl=\"1h\" \\ max_ttl=\"24h\" Now a pod requests its own credentials through the same injector annotations as before:\nannotations: vault.hashicorp.com/agent-inject-secret-db: \"database/creds/myapp-db\" vault.hashicorp.com/agent-inject-template-db: | {{- with secret \"database/creds/myapp-db\" -}} export PGUSER=\"{{ .Data.username }}\" export PGPASSWORD=\"{{ .Data.password }}\" {{- end }} Each pod gets unique credentials that:\nAuto-rotate Can be revoked individually Create an audit trail per pod PKI: Certificate Authority The same dynamic-secret idea works for certificates. Vault can run as your internal CA and issue short-lived certs on demand, which beats the alternative of long-lived certificates you forget about until they expire on a Sunday. Set up a root CA and a role that constrains what it’ll sign:\n# Enable PKI kubectl exec -it vault-0 -n vault -- vault secrets enable pki # Configure max TTL kubectl exec -it vault-0 -n vault -- vault secrets tune -max-lease-ttl=87600h pki # Generate root CA kubectl exec -it vault-0 -n vault -- vault write -field=certificate pki/root/generate/internal \\ common_name=\"My Org Root CA\" \\ ttl=87600h \u003e CA_cert.crt # Create role for issuing certs kubectl exec -it vault-0 -n vault -- vault write pki/roles/internal-certs \\ allowed_domains=\"internal,svc.cluster.local\" \\ allow_subdomains=true \\ max_ttl=72h Wire this into cert-manager and certificates rotate themselves across the cluster without you touching anything.\nHigh Availability A secrets store that goes down takes every dependent app with it, so Vault is the last place you want a single point of failure. Run it in HA mode with the built-in Raft storage, which keeps the data inside Vault itself instead of leaning on an external database you’d also have to make resilient.\nserver: ha: enabled: true replicas: 3 raft: enabled: true setNodeId: true config: | cluster_name = \"vault-cluster\" storage \"raft\" { path = \"/vault/data\" retry_join { leader_api_addr = \"http://vault-0.vault-internal:8200\" } retry_join { leader_api_addr = \"http://vault-1.vault-internal:8200\" } retry_join { leader_api_addr = \"http://vault-2.vault-internal:8200\" } } This creates a 3-node Raft cluster:\nOne leader, two followers Automatic leader election on failure Data replicated across all nodes Backup Strategy HA protects you from a node dying. It does nothing for a fat-fingered vault policy delete or a corrupted Raft log. For that you need snapshots, and you need them automated, because a backup you have to remember to run is a backup that won’t exist when you need it. Take a manual snapshot to see how it works:\n# Create snapshot kubectl exec -it vault-0 -n vault -- vault operator raft snapshot save /tmp/vault-snapshot.snap # Copy to local kubectl cp vault/vault-0:/tmp/vault-snapshot.snap ./vault-snapshot.snap Then make it boring and automatic with a CronJob, so snapshots happen whether or not I’m paying attention:\napiVersion: batch/v1 kind: CronJob metadata: name: vault-backup spec: schedule: \"0 */6 * * *\" # Every 6 hours jobTemplate: spec: template: spec: serviceAccountName: vault-backup containers: - name: backup image: hashicorp/vault:1.15 command: - /bin/sh - -c - | vault operator raft snapshot save /backup/vault-$(date +%Y%m%d-%H%M%S).snap volumeMounts: - name: backup mountPath: /backup volumes: - name: backup persistentVolumeClaim: claimName: vault-backup restartPolicy: OnFailure Audit Logging If you can’t tell who read which secret and when, you don’t really control your secrets, you just store them. Audit logging closes that gap by recording every single request. Turn it on:\nkubectl exec -it vault-0 -n vault -- vault audit enable file file_path=/vault/audit/audit.log Every access lands in the log as structured JSON, ready to ship into Loki or whatever you query with:\n{ \"time\": \"2025-07-02T10:00:00Z\", \"type\": \"response\", \"auth\": { \"token_type\": \"service\", \"policies\": [\"myapp-policy\"] }, \"request\": { \"path\": \"secret/data/myapp/config\", \"operation\": \"read\" } } The Full Picture: My Actual Setup Remember the three-line Helm install from the start? Here’s where all the layers land in practice. This is the real config running in my homelab, HA Raft, audit storage, sane resource limits, and the UI for the rare moment I need to click something:\nserver: ha: enabled: true replicas: 3 raft: enabled: true dataStorage: size: 10Gi storageClass: longhorn auditStorage: enabled: true size: 10Gi storageClass: longhorn resources: requests: memory: 256Mi cpu: 250m limits: memory: 512Mi cpu: 500m extraEnvironmentVars: VAULT_CACERT: /vault/userconfig/vault-tls/ca.crt injector: enabled: true resources: requests: memory: 64Mi cpu: 50m limits: memory: 128Mi cpu: 100m ui: enabled: true A few choices worth calling out:\nHA with Raft keeps storage inside Vault, so there’s no external database to also keep alive. Audit storage earns its keep the first time you have to answer “who touched this secret?”. Resource limits stay small because Vault genuinely is lightweight. It barely registers on the cluster. UI enabled for the occasional manual poke, even though almost everything runs through GitOps. Migration from Cloud KMS Already on AWS Secrets Manager or similar? You don’t have to flip a switch and pray. Migrate one app at a time. Install Vault alongside your existing KMS, sync the secrets you want to move, repoint apps individually, and decommission the cloud KMS once it’s empty. Nobody gets a big-bang cutover weekend.\nExternal Secrets Operator makes this painless because it can read from both backends at once during the transition:\napiVersion: external-secrets.io/v1beta1 kind: ClusterSecretStore metadata: name: aws-secrets-manager spec: provider: aws: service: SecretsManager region: eu-west-1 --- apiVersion: external-secrets.io/v1beta1 kind: ClusterSecretStore metadata: name: vault spec: provider: vault: server: \"http://vault.vault:8200\" Why This Matters We started with a base64 string in etcd and ended with dynamic database credentials, a private CA, an HA cluster, automated snapshots, and an audit log of every access. You can stop at any layer and still come out ahead of where the defaults left you.\nWhat you get from running it yourself is visibility into how secrets are actually stored and accessed, the same workflow whether you’re on a cloud, on-prem, or a Raspberry Pi in island mode, no surprise pricing, and the ability to fix things when they break because you understand the whole stack.\nI won’t pretend the operational cost is zero. You now own unsealing, backups, and upgrades. But secrets are the one thing I refuse to leave inside a black box, because they’re the keys to everything else I run. Owning that complexity is the price of not depending on a system I can’t inspect, and for this particular thing I’ll pay it every time.","permalink":"https://tommeurs.nl/en/posts/vault-beginners-kubernetes/","title":"Vault for Beginners: Secrets Management in Kubernetes"},{"content":"A standard Kubernetes Deployment had served me well for a long time. Push a new image tag, watch the pods roll, done. It was simple, it was declarative, and most of the time nothing went wrong. The rolling update even gave me a warm feeling of safety: old pods only get torn down once new ones are ready.\nThat feeling is a lie. A rolling update protects you from pods that fail to start. It does nothing to protect you from pods that start perfectly and then serve broken responses. The container is healthy, the readiness probe is green, and your new code is quietly returning 500s to every single user. Within seconds, 100% of your traffic is hitting code that nobody validated under real load.\nSo the question I kept coming back to: how do you ship a new version without betting all of your users on it being correct? I want to understand what a release is actually doing before it touches everyone, and I want the system to pull the plug on its own when I’m not watching.\nArgo Rollouts is my answer. It’s a drop-in replacement for a Deployment that shifts traffic to the new version in steps, checks metrics along the way, and rolls back automatically when the numbers go bad. Same declarative spirit, far smaller blast radius.\nWhy Progressive Delivery? Look at what a standard Deployment costs you when a bug ships:\nTime 0:00 - Deploy new version Time 0:02 - All pods running new version Time 0:05 - Errors start appearing Time 0:08 - Alerts fire Time 0:15 - Engineer investigates Time 0:25 - Rollback initiated Time 0:27 - All pods back to old version Blast radius: 100% of users for ~25 minutes With progressive delivery:\nTime 0:00 - Deploy new version (5% traffic) Time 0:05 - Automated analysis detects errors Time 0:06 - Automatic rollback Blast radius: 5% of users for ~6 minutes That gap is the whole point. I’m not trying to prevent failures, because I can’t. Bugs ship no matter how good the tests are. What I can do is make sure that when one slips through, it hits five people for six minutes instead of everyone for half an hour. That’s resilience by design: assume things break, and limit how much they can hurt.\nTwo Strategies: Canary vs Blue-Green Argo Rollouts supports a few strategies. Two cover almost everything I do.\nCanary Traffic shifts gradually from old to new version:\nStep 1: 5% new, 95% old (test the waters) Step 2: 20% new, 80% old (expand if healthy) Step 3: 50% new, 50% old (halfway point) Step 4: 100% new, 0% old (full rollout) Best for stateless services and anything with enough traffic that a small slice gives you real signal. This is what I reach for by default.\nBlue-Green Two complete environments, instant switch:\nBefore: 100% blue (old) 0% green (new) Deploy: 100% blue green ready, not receiving traffic Switch: 0% blue 100% green Best for services that need an instant flip back, for testing the new version against production data before it goes live, and for cases where you want both versions up at the same time to compare them. The cost is double the resources during the switch, since you run two full copies.\nInstalling Argo Rollouts kubectl create namespace argo-rollouts kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml That works, but applying a release manifest by hand is the kind of thing I forget to do consistently. For GitOps with ArgoCD, let the controller manage it as an Application instead:\napiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: argo-rollouts namespace: argocd spec: project: default source: repoURL: https://argoproj.github.io/argo-helm chart: argo-rollouts targetRevision: 2.35.1 helm: values: | dashboard: enabled: true destination: server: https://kubernetes.default.svc namespace: argo-rollouts syncPolicy: automated: prune: true selfHeal: true Canary Rollout Example The nice part is how little changes. A Rollout looks almost identical to a Deployment. Same replicas, same selector, same pod template. You swap the kind and add a strategy block:\napiVersion: argoproj.io/v1alpha1 kind: Rollout metadata: name: my-app spec: replicas: 10 selector: matchLabels: app: my-app template: metadata: labels: app: my-app spec: containers: - name: app image: my-app:v2.0.0 ports: - containerPort: 8080 strategy: canary: steps: - setWeight: 5 - pause: { duration: 5m } - setWeight: 20 - pause: { duration: 5m } - setWeight: 50 - pause: { duration: 5m } - setWeight: 100 This creates a 20-minute gradual rollout:\nSend 5% of traffic to new version, wait 5 minutes If healthy, increase to 20%, wait 5 minutes If healthy, increase to 50%, wait 5 minutes Complete rollout to 100% If you abort at any point, traffic snaps back to the old version. The old ReplicaSet never went away, so there’s nothing to rebuild.\nTraffic Management Here’s a detail that trips people up. By default, Argo Rollouts approximates the traffic weight using replica count. Ask for 5% with 10 replicas and it can’t give you 5%, it gives you the nearest it can manage with whole pods. For most apps that approximation is fine. When you need the weight to actually mean 5%, you wire the Rollout into your ingress so the proxy splits traffic precisely.\nI run Traefik, so that’s the one I use in production. The Nginx and Istio configs are here too for completeness.\nWith Traefik apiVersion: argoproj.io/v1alpha1 kind: Rollout metadata: name: my-app spec: strategy: canary: canaryService: my-app-canary stableService: my-app-stable trafficRouting: traefik: weightedTraefikServiceName: my-app-weighted steps: - setWeight: 10 - pause: { duration: 2m } - setWeight: 50 - pause: { duration: 2m } With supporting TraefikService:\napiVersion: traefik.io/v1alpha1 kind: TraefikService metadata: name: my-app-weighted spec: weighted: services: - name: my-app-stable port: 80 weight: 100 # Managed by Argo Rollouts - name: my-app-canary port: 80 weight: 0 # Managed by Argo Rollouts With Nginx Ingress apiVersion: argoproj.io/v1alpha1 kind: Rollout spec: strategy: canary: canaryService: my-app-canary stableService: my-app-stable trafficRouting: nginx: stableIngress: my-app-ingress With Istio apiVersion: argoproj.io/v1alpha1 kind: Rollout spec: strategy: canary: trafficRouting: istio: virtualService: name: my-app-vsvc routes: - primary Blue-Green Rollout apiVersion: argoproj.io/v1alpha1 kind: Rollout metadata: name: my-app spec: replicas: 5 selector: matchLabels: app: my-app template: metadata: labels: app: my-app spec: containers: - name: app image: my-app:v2.0.0 strategy: blueGreen: activeService: my-app-active previewService: my-app-preview autoPromotionEnabled: false prePromotionAnalysis: templates: - templateName: smoke-tests postPromotionAnalysis: templates: - templateName: load-test This creates:\nmy-app-active: Points to current production version my-app-preview: Points to new version for testing The new version runs in full but sees no production traffic until you promote it. With autoPromotionEnabled: false, that promotion is a deliberate decision you make after poking at the preview service yourself.\nAutomated Analysis Everything so far still leans on a human watching a dashboard and deciding when to continue. That’s where I lose trust in the process, because I’m not always watching, and at 2am I’m definitely not. The part of Argo Rollouts I actually care about is automated analysis: the controller queries your metrics during the rollout and aborts on its own when they go bad.\nAnalysis Template apiVersion: argoproj.io/v1alpha1 kind: AnalysisTemplate metadata: name: success-rate spec: args: - name: service-name metrics: - name: success-rate interval: 1m successCondition: result[0] \u003e= 0.95 failureLimit: 3 provider: prometheus: address: http://prometheus.monitoring:9090 query: | sum(rate(http_requests_total{service=\"{{args.service-name}}\",status=~\"2..\"}[5m])) / sum(rate(http_requests_total{service=\"{{args.service-name}}\"}[5m])) This template queries Prometheus every minute for the HTTP success rate, treats anything at or above 95% as healthy, and fails the rollout after 3 consecutive bad readings. The failureLimit matters: a single scrape blip shouldn’t kill a deploy, but three in a row is a real signal.\nUsing Analysis in Rollout apiVersion: argoproj.io/v1alpha1 kind: Rollout spec: strategy: canary: steps: - setWeight: 5 - pause: { duration: 2m } - analysis: templates: - templateName: success-rate args: - name: service-name value: my-app - setWeight: 50 - pause: { duration: 5m } - analysis: templates: - templateName: success-rate args: - name: service-name value: my-app Now the rollout shifts 5% of traffic, waits two minutes, then runs the analysis. If success rate is below 95% for three readings, it rolls back without me touching anything. If it passes, traffic goes to 50% and the same check runs again before the full rollout completes. The deploy now defends itself.\nMultiple Analysis Metrics Success rate alone misses things. A version can return 200s while taking three seconds to do it, which is its own kind of broken. So I check several signals at once, and any one of them failing aborts the rollout:\napiVersion: argoproj.io/v1alpha1 kind: AnalysisTemplate metadata: name: comprehensive-check spec: args: - name: service-name metrics: # HTTP success rate - name: success-rate interval: 1m successCondition: result[0] \u003e= 0.95 failureLimit: 3 provider: prometheus: address: http://prometheus.monitoring:9090 query: | sum(rate(http_requests_total{service=\"{{args.service-name}}\",status=~\"2..\"}[5m])) / sum(rate(http_requests_total{service=\"{{args.service-name}}\"}[5m])) # P99 latency - name: latency-p99 interval: 1m successCondition: result[0] \u003c 0.5 failureLimit: 3 provider: prometheus: address: http://prometheus.monitoring:9090 query: | histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{service=\"{{args.service-name}}\"}[5m])) by (le)) # Error rate - name: error-rate interval: 1m successCondition: result[0] \u003c 0.01 failureLimit: 2 provider: prometheus: address: http://prometheus.monitoring:9090 query: | sum(rate(http_requests_total{service=\"{{args.service-name}}\",status=~\"5..\"}[5m])) / sum(rate(http_requests_total{service=\"{{args.service-name}}\"}[5m])) Web-Based Job Analysis Not every check fits a Prometheus query. Sometimes you just want to hit a couple of endpoints and confirm they answer. Argo Rollouts can run a Kubernetes Job as an analysis step, so a failed curl aborts the rollout the same way a bad metric does:\napiVersion: argoproj.io/v1alpha1 kind: AnalysisTemplate metadata: name: smoke-test spec: metrics: - name: smoke-test provider: job: spec: backoffLimit: 1 template: spec: containers: - name: smoke image: curlimages/curl command: - /bin/sh - -c - | curl -f http://my-app-canary/health || exit 1 curl -f http://my-app-canary/api/status || exit 1 restartPolicy: Never Dashboard and CLI Monitor rollouts with the Argo Rollouts kubectl plugin:\n# Install plugin curl -LO https://github.com/argoproj/argo-rollouts/releases/latest/download/kubectl-argo-rollouts-darwin-amd64 chmod +x kubectl-argo-rollouts-darwin-amd64 mv kubectl-argo-rollouts-darwin-amd64 /usr/local/bin/kubectl-argo-rollouts # Watch rollout progress kubectl argo rollouts get rollout my-app -w # Manually promote (if autoPromotion is disabled) kubectl argo rollouts promote my-app # Abort and rollback kubectl argo rollouts abort my-app # View dashboard kubectl argo rollouts dashboard The -w watch view is what I keep open during a release. It shows the current weight, which analysis runs are passing, and how far along each step is, all in the terminal where I already live. The web dashboard does the same thing with graphs if you prefer that.\nIntegration with GitOps This is where it clicks into the rest of my workflow. I don’t run any of those kubectl commands during a normal deploy. The image tag lives in Git, ArgoCD syncs it, and the progressive rollout starts on its own.\n# In your GitOps repo apiVersion: argoproj.io/v1alpha1 kind: Rollout metadata: name: my-app spec: template: spec: containers: - name: app image: my-app:v2.1.0 # Update this line The semantic versioning pipeline creates the tag, that bumps the image in the GitOps repo, ArgoCD syncs the change, and the Rollout takes it from there. Git is the only thing I touch, and the controller does the careful part.\nNotifications Since I’m deliberately not watching every rollout, I want a ping when one finishes or aborts:\napiVersion: argoproj.io/v1alpha1 kind: Rollout metadata: name: my-app annotations: notifications.argoproj.io/subscribe.on-rollout-completed.slack: my-channel notifications.argoproj.io/subscribe.on-rollout-aborted.slack: my-channel You configure the notification controller separately, and it shares its config with ArgoCD notifications if you already run those.\nMy Production Setup Here’s the actual Rollout I run for my API, pulling together everything above:\napiVersion: argoproj.io/v1alpha1 kind: Rollout metadata: name: api spec: replicas: 5 revisionHistoryLimit: 3 selector: matchLabels: app: api template: metadata: labels: app: api spec: containers: - name: api image: registry.example.com/api:v1.0.0 ports: - containerPort: 8080 readinessProbe: httpGet: path: /health port: 8080 initialDelaySeconds: 5 periodSeconds: 5 resources: requests: cpu: 100m memory: 128Mi limits: cpu: 500m memory: 512Mi strategy: canary: canaryService: api-canary stableService: api-stable trafficRouting: traefik: weightedTraefikServiceName: api-weighted steps: # Phase 1: Smoke test - setWeight: 5 - pause: { duration: 2m } - analysis: templates: - templateName: smoke-test # Phase 2: Limited exposure - setWeight: 25 - pause: { duration: 5m } - analysis: templates: - templateName: success-rate args: - name: service-name value: api # Phase 3: Majority traffic - setWeight: 75 - pause: { duration: 10m } - analysis: templates: - templateName: success-rate args: - name: service-name value: api # Phase 4: Full rollout - setWeight: 100 rollbackWindow: revisions: 2 A few choices in there are deliberate. The smoke test runs first at 5%, so an obviously broken build dies in two minutes before it reaches real users. The pauses get longer as the weight climbs, because more traffic means I want more signal before trusting the next jump. And rollbackWindow keeps the last two revisions warm so a revert is instant, not a rebuild.\nWhen Not to Use Progressive Delivery I should be honest about the cost. Progressive delivery adds moving parts: extra services, traffic routing config, analysis templates, metrics that have to actually exist. That complexity buys you nothing in a few cases, and forcing it in anyway just adds friction:\nBreaking database schema changes. If the new code can’t coexist with the old schema, running both versions at once is a bug, not a safety feature. You need the whole app on one version. Single-user or internal tools. There’s no meaningful traffic to split, so a canary tells you nothing. A plain Deployment is the honest choice. Tightly coupled services that must upgrade in lockstep. Shifting one of them gradually breaks the contract with the others. For a production service handling real users over HTTP, though, the complexity earns its keep on the first bad deploy it catches.\nWhy This Matters Every deploy is a bet that your new code behaves in production. Tests narrow the odds, but they don’t settle the bet, because the staging environment is never quite the real one.\nWhat progressive delivery changes is the size of the wager. A bad release reaches a slice of users for a couple of minutes, the controller spots it in the metrics, and traffic flows back to the version you know works, all without waking anyone up. I get to understand what a release is doing while it’s happening, and the system protects itself when I’m not looking. For me that’s the whole reason to run my own platform: not chasing deploys that never fail, but building one that fails small and recovers on its own.","permalink":"https://tommeurs.nl/en/posts/argo-rollouts-progressive-delivery/","title":"Progressive Delivery with Argo Rollouts: Canary and Blue-Green Deployments"},{"content":"Pull a base image, copy in your code, push it to production. That’s the loop most of us run on autopilot. The part nobody looks at is the base image itself, which quietly drags in a few hundred packages you never chose. Any one of them could carry a known CVE , and you’d have no idea.\nThat bothers me. I don’t like running things I can’t inspect, and a container image you haven’t scanned is exactly that: a black box you’re trusting because looking inside felt like too much effort.\nTrivy is how I stopped trusting and started looking. It’s an open-source vulnerability scanner from Aqua that tells you what’s actually inside an image and which of those packages have known problems. In this post I’ll build up a GitLab CI setup from the smallest useful version to the one I run in production, one layer at a time.\nWhy Trivy and not the others There are plenty of scanners. Snyk, Anchore, and Clair all do the job. I landed on Trivy for reasons that line up with how I like to run things:\nSingle binary. No database server, no daemon, no setup ritual. Speed. Scans finish in seconds. Slow tools get skipped, and a skipped scan protects nothing. Broad coverage. OS packages, language dependencies, and IaC misconfigurations in one tool. CI-friendly. Sensible exit codes and a handful of output formats. Free. No licence seats to count, no procurement conversation. Snyk, Anchore, and Clair are all fine choices if they fit your situation. For me, Trivy’s lack of moving parts won. Fewer things to run means fewer things to understand when something breaks.\nThe simplest version that works Here’s the smallest GitLab CI config that does something useful: build, scan, and only push if the scan passes.\nstages: - build - scan - push variables: IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA build: stage: build image: docker:24 services: - docker:24-dind script: - docker build -t $IMAGE_TAG . - docker save $IMAGE_TAG -o image.tar artifacts: paths: - image.tar expire_in: 1 hour scan: stage: scan image: aquasec/trivy:latest script: - trivy image --input image.tar --exit-code 1 --severity HIGH,CRITICAL allow_failure: false push: stage: push image: docker:24 services: - docker:24-dind script: - docker load -i image.tar - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY - docker push $IMAGE_TAG needs: - build - scan Three things make this work:\nBuild first, scan second. The image gets saved as a tarball artifact and handed to the scan stage. Exit code 1. Trivy fails the pipeline when it finds a HIGH or CRITICAL vulnerability. Push only after the scan passes. No vulnerable image ever reaches the registry. If you stop reading here, you already have a working gate. Everything below is refinement.\nReading the output When Trivy finds something, it hands you a table like this:\n┌──────────────────┬────────────────┬──────────┬────────────────────────────────────┐ │ Library │ Vulnerability │ Severity │ Installed Version │ ├──────────────────┼────────────────┼──────────┼────────────────────────────────────┤ │ openssl │ CVE-2024-0727 │ HIGH │ 3.0.2-0ubuntu1.10 │ │ libcurl4 │ CVE-2024-2398 │ MEDIUM │ 7.81.0-1ubuntu1.15 │ │ python3.10 │ CVE-2024-0450 │ HIGH │ 3.10.12-1~22.04.3 │ └──────────────────┴────────────────┴──────────┴────────────────────────────────────┘ That’s the whole point. You now know which packages have issues, how bad each one is, and which version is affected. Before this, the honest description of your security posture was “probably fine.” Now you can actually answer the question.\nGoing further: severity thresholds Not every finding deserves a red pipeline. Trivy lets you tune the threshold to match what you’re willing to ship:\n# Fail only on critical - fast but risky trivy image --exit-code 1 --severity CRITICAL # Fail on high and critical - balanced trivy image --exit-code 1 --severity HIGH,CRITICAL # Fail on medium and above - strict trivy image --exit-code 1 --severity MEDIUM,HIGH,CRITICAL Start with HIGH,CRITICAL. You can always tighten later. Go too strict on day one and you’ll drown in MEDIUM findings, people will start clicking through the red without reading it, and the gate becomes theatre.\nGoing further: false positives Sometimes Trivy flags a CVE that genuinely doesn’t apply to you. The code path is never reached, or there’s simply no fix yet. A .trivyignore file handles those:\n# CVE doesn't affect our code path CVE-2024-0727 # No fix available yet, tracking in JIRA-1234 CVE-2024-2398 Point the scan at it:\ntrivy image --input image.tar --ignorefile .trivyignore --exit-code 1 Write down why you’re ignoring each CVE, right there in the file. Six months from now you won’t remember, and an undocumented ignore is just a hole in your gate with no audit trail.\nGoing further: output formats The default table is for humans. Trivy speaks other formats too, depending on who or what is reading:\n# Human readable table (default) trivy image --format table $IMAGE_TAG # JSON for programmatic processing trivy image --format json --output results.json $IMAGE_TAG # SARIF for GitHub/GitLab security dashboards trivy image --format sarif --output trivy.sarif $IMAGE_TAG # HTML report for stakeholders trivy image --format template --template \"@contrib/html.tpl\" --output report.html $IMAGE_TAG For GitLab’s built-in security dashboard, generate its native report format:\nscan: stage: scan image: aquasec/trivy:latest script: - trivy image --format template --template \"@/contrib/gitlab.tpl\" --output gl-container-scanning-report.json $IMAGE_TAG - trivy image --exit-code 1 --severity HIGH,CRITICAL $IMAGE_TAG artifacts: reports: container_scanning: gl-container-scanning-report.json Now the findings show up directly in the merge request, where the person reviewing the change can actually see them.\nGoing further: scanning more than images Trivy doesn’t stop at container images. The same binary handles your source tree, a remote repo, and your Kubernetes or Terraform config:\n# Filesystem (your project directory) trivy fs --exit-code 1 . # Git repository trivy repo https://github.com/your/repo # Kubernetes manifests trivy config ./k8s/ # Infrastructure as Code trivy config ./terraform/ A fuller security stage might run several of these together:\nsecurity-scan: stage: scan image: aquasec/trivy:latest script: # Scan source code dependencies - trivy fs --exit-code 0 --severity HIGH,CRITICAL . --format table # Scan Kubernetes manifests for misconfigurations - trivy config ./k8s/ --exit-code 0 --severity HIGH,CRITICAL # Scan container image (this one fails the build) - trivy image --input image.tar --exit-code 1 --severity HIGH,CRITICAL Going further: caching the database Trivy pulls a vulnerability database on every run. In CI that’s wasted minutes and an external dependency you don’t need on the critical path. Cache it:\nvariables: TRIVY_CACHE_DIR: .trivy-cache scan: stage: scan image: aquasec/trivy:latest cache: key: trivy-db paths: - .trivy-cache script: - trivy image --cache-dir $TRIVY_CACHE_DIR --input image.tar Or split the download from the scan so you control exactly when the network call happens:\nscan: image: aquasec/trivy:latest before_script: - trivy image --download-db-only script: - trivy image --skip-db-update --input image.tar The full picture: my production config Here’s what I actually run. It’s the simple version from the top, plus every layer we just walked through:\nstages: - build - scan - push - deploy variables: IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA TRIVY_CACHE_DIR: .trivy-cache # Don't fail on unfixed vulnerabilities TRIVY_IGNORE_UNFIXED: \"true\" .trivy-template: \u0026trivy-template image: aquasec/trivy:latest cache: key: trivy-db paths: - .trivy-cache scan-image: \u003c\u003c: *trivy-template stage: scan script: # Generate report for GitLab dashboard - trivy image --cache-dir $TRIVY_CACHE_DIR --format template --template \"@/contrib/gitlab.tpl\" --output gl-container-scanning-report.json --input image.tar # Fail on fixable HIGH/CRITICAL - trivy image --cache-dir $TRIVY_CACHE_DIR --exit-code 1 --severity HIGH,CRITICAL --ignore-unfixed --input image.tar artifacts: reports: container_scanning: gl-container-scanning-report.json needs: - build scan-config: \u003c\u003c: *trivy-template stage: scan script: - trivy config --cache-dir $TRIVY_CACHE_DIR --exit-code 1 --severity HIGH,CRITICAL ./k8s/ allow_failure: true # Warning only, don't block Three decisions in there are worth calling out, because they’re trade-offs rather than obvious defaults:\n--ignore-unfixed. Failing on a vulnerability that has no available patch just blocks me with no action to take. I’d rather know about it without it stopping the line. Config scan as a warning. Misconfigurations matter, but I don’t want a stray k8s/ nit blocking every deploy. It nags, it doesn’t gate. Cached database. Faster builds and one fewer external dependency in the hot path. The pipeline is the safety net, not the first line There are two ways to think about where scanning lives. Gate keeping puts it at build time: scan the image, block the bad ones from the registry, enforce the standard before anything deploys. Shift left pushes it earlier, into the developer’s hands with pre-commit hooks and IDE plugins, so feedback arrives in seconds instead of after a pipeline run.\nI run both. The pipeline catches whatever slips through, but a developer should see the problem before they ever commit:\n# Developer workflow trivy fs . # Scan before commit docker build -t myapp . trivy image myapp # Scan before push Think of the pipeline as the seatbelt. Useful exactly when something has already gone wrong, but you’d rather not be relying on it as your primary plan.\nRescanning at promotion time If you deploy with ArgoCD, hang one more scan on the promotion step:\npromote-to-prod: stage: deploy script: # Re-scan the specific image version before promotion - trivy image --exit-code 1 --severity CRITICAL $IMAGE_TAG # Update GitOps repo only if scan passes - | cd gitops-repo kustomize edit set image myapp:${CI_COMMIT_TAG} git commit -am \"Promote ${CI_COMMIT_TAG} to production\" git push rules: - if: $CI_COMMIT_TAG =~ /^v[0-9]+/ An image that was clean at build time isn’t guaranteed to be clean a week later, because new CVEs land all the time. Rescanning right before you write the new tag into the GitOps repo means a vulnerability disclosed after the original build still gets caught before it reaches production.\nBeyond vulnerabilities: the SBOM A Software Bill of Materials is an inventory of everything in your image. Trivy generates one in a single command:\ntrivy image --format spdx-json --output sbom.json $IMAGE_TAG Worth having for a few reasons:\nCompliance. Some industries now require an SBOM by default. Incident response. When the next big CVE drops, you grep your stored SBOMs instead of rebuilding and rescanning every image under pressure. Supply chain visibility. You know your dependencies before someone else points one out to you. Store the SBOM next to the image in your registry so it’s there when you need it.\nCommon pitfalls Scanning the wrong thing # Wrong - scans the trivy image itself trivy image aquasec/trivy:latest # Right - scans your built image trivy image --input image.tar Swallowing the exit code # Wrong - always succeeds trivy image $IMAGE_TAG || true # Right - fail on findings trivy image --exit-code 1 $IMAGE_TAG Alert fatigue Start at MEDIUM with no ignores and you’ll get hundreds of findings on day one. People stop looking, and a gate nobody reads is just latency. Start strict on critical, expand as you clean house.\nWhy I bother This was never about ticking a compliance box. It’s about being able to answer a simple question when the next critical CVE gets announced: which of my images are affected, which services run them, and how fast can I patch?\nWithout scanning, every one of those questions is a manual hunt across every image you’ve ever shipped, done under time pressure with someone watching. With Trivy wired into the pipeline, the answer is already sitting in front of you. That’s the difference between knowing what you run and hoping it’s fine.\nYou can’t protect what you can’t see. Container scanning makes the contents of your images visible, turning black boxes into documented, auditable systems.","permalink":"https://tommeurs.nl/en/posts/trivy-container-scanning/","title":"Container Image Scanning with Trivy in Your CI Pipeline"},{"content":"Picture the end state first. You merge a branch to main, walk away to get coffee, and by the time you’re back the project has a new version tag, a fresh changelog entry, and a GitLab release with notes written from your commits. Nobody decided “is this 1.4.0 or 1.3.5?” because nobody had to. The version fell out of the work itself.\nThat’s where I want to get to in this post. I run a self-hosted GitLab and I do not want versioning to be a thing I think about. A version number that someone picks by hand is a tiny black box: it depends on whether that person remembered the rules, was paying attention, and agreed with everyone else about what counts as a breaking change. I would rather the number be a consequence of the commits, something I can inspect and reproduce.\nSemantic versioning (semver) gives you the rules to make that possible:\nMAJOR: Breaking changes MINOR: New features, backwards compatible PATCH: Bug fixes, backwards compatible Clear enough on paper. The trouble starts when a human has to apply them under deadline pressure. “Is this a minor or a patch?” gets answered differently depending on who’s tired. Let’s hand that decision to a machine that always reads the same rules the same way.\nThe Core Idea: Conventional Commits The piece that makes this work is conventional commits, a standard commit message format that tells tooling what kind of change you made. The commit log stops being just a story for humans and becomes structured input.\n\u003ctype\u003e(\u003cscope\u003e): \u003cdescription\u003e [optional body] [optional footer] Types that matter for versioning:\nfix: → PATCH bump (1.0.0 → 1.0.1) feat: → MINOR bump (1.0.0 → 1.1.0) BREAKING CHANGE: or ! → MAJOR bump (1.0.0 → 2.0.0) Examples:\nfix(auth): correct password validation regex feat(api): add user preferences endpoint feat(core)!: change configuration format BREAKING CHANGE: config.yml no longer supports legacy format That’s the whole concept. Everything below is just teaching tools how to read it.\nThe Simplest Version: semantic-release semantic-release is the tool I reach for. Start with the bare minimum and you’ll understand the moving parts before the config gets busy. It:\nAnalyzes commits since last release Determines the version bump Generates release notes Creates Git tags Publishes (npm, Docker, etc.) Installation This is the smallest config that does something useful. Create a .releaserc.json:\n{ \"branches\": [\"main\"], \"plugins\": [ \"@semantic-release/commit-analyzer\", \"@semantic-release/release-notes-generator\", \"@semantic-release/changelog\", \"@semantic-release/gitlab\", \"@semantic-release/git\" ] } GitLab CI Configuration stages: - test - release variables: GITLAB_TOKEN: $GITLAB_TOKEN # CI/CD variable with api scope release: stage: release image: node:20 before_script: - npm install -g semantic-release @semantic-release/gitlab @semantic-release/changelog @semantic-release/git script: - semantic-release rules: - if: $CI_COMMIT_BRANCH == \"main\" Required GitLab Token Create a Project Access Token with:\nRole: Maintainer Scopes: api, write_repository Add it as CI/CD variable GITLAB_TOKEN (protected, masked).\nThat’s it. With those three files in place you already have automated versioning. If you stop reading here, you have something that works. Everything past this point is making it fit a real project.\nHow It Works When you push to main:\nCommit Analysis\nCommits since v1.2.3\nfix(api): handle null response\nfeat(ui): add dark mode\ndocs: update readme\n→ PATCH\n→ MINOR\n→ (no release)\nHighest bump: MINOR\nNew version: v1.3.0\nThen semantic-release:\nCreates CHANGELOG.md entry Commits the changelog Creates Git tag v1.3.0 Creates GitLab Release with notes Layer 1: A Real Config The three-line version ships, but on a real project you’ll want control over which commit types release and how the changelog reads. Here’s the config I actually run:\n{ \"branches\": [ \"main\", { \"name\": \"beta\", \"prerelease\": true }, { \"name\": \"alpha\", \"prerelease\": true } ], \"plugins\": [ [\"@semantic-release/commit-analyzer\", { \"preset\": \"conventionalcommits\", \"releaseRules\": [ { \"type\": \"docs\", \"release\": false }, { \"type\": \"refactor\", \"release\": \"patch\" }, { \"type\": \"perf\", \"release\": \"patch\" }, { \"type\": \"chore\", \"scope\": \"deps\", \"release\": \"patch\" } ] }], [\"@semantic-release/release-notes-generator\", { \"preset\": \"conventionalcommits\", \"presetConfig\": { \"types\": [ { \"type\": \"feat\", \"section\": \"Features\" }, { \"type\": \"fix\", \"section\": \"Bug Fixes\" }, { \"type\": \"perf\", \"section\": \"Performance\" }, { \"type\": \"refactor\", \"section\": \"Refactoring\" }, { \"type\": \"docs\", \"section\": \"Documentation\", \"hidden\": true }, { \"type\": \"chore\", \"section\": \"Maintenance\", \"hidden\": true } ] } }], [\"@semantic-release/changelog\", { \"changelogFile\": \"CHANGELOG.md\" }], [\"@semantic-release/gitlab\", { \"gitlabUrl\": \"https://gitlab.example.com\" }], [\"@semantic-release/git\", { \"assets\": [\"CHANGELOG.md\", \"package.json\"], \"message\": \"chore(release): ${nextRelease.version} [skip ci]\\n\\n${nextRelease.notes}\" }] ] } The releaseRules block is where I decide that a refactor counts as a patch and a docs commit ships nothing. The presetConfig controls how the changelog groups things. Read it once and you’ll see it’s the same idea as the simple version, just with the knobs exposed.\nEnforcing Conventional Commits The whole machine depends on commits following the convention. If someone writes “fixed stuff” the analyzer has nothing to read, and the version stalls. Garbage in, no release out. So you enforce the format rather than hope for it.\ncommitlint Create .commitlintrc.json:\n{ \"extends\": [\"@commitlint/config-conventional\"] } Pre-commit Hook Using husky:\nnpm install --save-dev husky @commitlint/cli @commitlint/config-conventional npx husky install npx husky add .husky/commit-msg 'npx --no -- commitlint --edit $1' GitLab CI Check For projects without husky:\nlint-commits: stage: test image: node:20 before_script: - npm install -g @commitlint/cli @commitlint/config-conventional script: - | if [ -n \"$CI_MERGE_REQUEST_IID\" ]; then # Check commits in MR git fetch origin $CI_MERGE_REQUEST_TARGET_BRANCH_NAME commitlint --from origin/$CI_MERGE_REQUEST_TARGET_BRANCH_NAME --to HEAD fi rules: - if: $CI_MERGE_REQUEST_ID Layer 2: Versioning Beyond npm So far this looks npm-shaped. It isn’t tied to npm at all. The same commit analysis can drive Docker tags, monorepo releases, and deployments.\nContainer Image Versioning For Docker images, let semantic-release tag the image with the version it just computed:\nrelease: stage: release image: docker:24 services: - docker:24-dind before_script: - apk add --no-cache nodejs npm git - npm install -g semantic-release @semantic-release/exec - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY script: - semantic-release rules: - if: $CI_COMMIT_BRANCH == \"main\" With .releaserc.json:\n{ \"plugins\": [ \"@semantic-release/commit-analyzer\", \"@semantic-release/release-notes-generator\", [\"@semantic-release/exec\", { \"publishCmd\": \"docker tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA $CI_REGISTRY_IMAGE:${nextRelease.version} \u0026\u0026 docker push $CI_REGISTRY_IMAGE:${nextRelease.version}\" }], \"@semantic-release/gitlab\" ] } Now your Docker images get semantic version tags automatically, derived from the exact same commits.\nMonorepo Versioning For monorepos with multiple packages, use semantic-release-monorepo or independent releases per package:\nrelease-api: stage: release script: - cd packages/api \u0026\u0026 semantic-release rules: - if: $CI_COMMIT_BRANCH == \"main\" changes: - packages/api/**/* release-web: stage: release script: - cd packages/web \u0026\u0026 semantic-release rules: - if: $CI_COMMIT_BRANCH == \"main\" changes: - packages/web/**/* Each package maintains its own version, scoped to the files that changed.\nIntegration with GitOps This is the part I care about most, because it closes the loop. After semantic-release creates a tag, that tag can trigger a deployment:\ndeploy: stage: deploy script: - | # Update GitOps repo with new version cd gitops-repo kustomize edit set image myapp:${CI_COMMIT_TAG} git commit -am \"Deploy ${CI_COMMIT_TAG}\" git push rules: - if: $CI_COMMIT_TAG =~ /^v[0-9]+\\.[0-9]+\\.[0-9]+$/ The tag triggers the deploy job, which updates the GitOps repository with the new version. A commit message ends up driving what runs in the cluster, and Git stays the single source of truth from message to deployment.\nHandling Breaking Changes Breaking changes are the one case where I want a human to slow down and write a sentence, so the format makes room for it:\nfeat(api)!: change authentication to OAuth2 BREAKING CHANGE: Basic auth is no longer supported. Users must migrate to OAuth2. See migration guide at /docs/oauth-migration The ! after the scope or BREAKING CHANGE: in the footer triggers a MAJOR bump. Document the breaking change in the commit body and it lands in the changelog, so the person hitting the breakage gets the migration note for free.\nPre-release Versions For beta/alpha releases:\n{ \"branches\": [ \"main\", { \"name\": \"beta\", \"prerelease\": true }, { \"name\": \"alpha\", \"prerelease\": true } ] } Commits to the beta branch create versions like 1.3.0-beta.1, so you can ship something to early testers without bumping the stable line.\nThe Full Picture: My Workflow Develop on feature branch with conventional commits MR to main: commitlint validates messages Merge to main: semantic-release determines version Tag created: triggers deployment pipeline GitOps updated: ArgoCD deploys new version Put the layers together and this is what a release looks like end to end:\nNo manual version bumps. No forgotten changelog entries. The “is this a minor or patch?” debate is gone because the rules live in a config file instead of in someone’s head. That’s the same coffee-break version tag I described at the top, now with the whole chain behind it.\nTroubleshooting “No release” when expecting one Check commit types. Only fix, feat, and breaking changes trigger releases by default.\n“Permission denied” on push Ensure GITLAB_TOKEN has write_repository scope and the associated user can push to protected branches.\nDuplicate releases Add [skip ci] to the release commit message (already in the config above).\nWhy I Bother People sometimes read this setup as laziness, like I just don’t want to type a version number. The real reason is that I trust a config file more than I trust my own memory at the end of a long day. Here’s what that buys me:\nConsistency: every release follows the same rules, whether I’m sharp or half asleep Documentation: the changelog is always current because it’s a byproduct, not a chore Traceability: version maps back to commits maps back to changes, no detective work Speed: no manual steps means a release is a merge, nothing more Low friction: I think about the code, the pipeline thinks about the version There’s a cost, and I want to be straight about it. You’re trading freedom for discipline. Everyone has to write commits in a specific format, the tooling adds moving parts, and the first time the token scope is wrong you’ll spend an afternoon on it. For a quick throwaway script that’s overkill. For anything I want to run for years and understand later, the version number stops being a decision and starts being a record of what actually happened. That trade is worth it to me.","permalink":"https://tommeurs.nl/en/posts/semantic-versioning-gitlab/","title":"Automating Semantic Versioning with GitLab CI"},{"content":"For a long time my CI lived on someone else’s servers. Push code, wait for a hosted runner, watch the minutes tick down against a monthly quota, hope the provider doesn’t change the rules next quarter. It worked, right up until I started asking where my source code actually sat while it was being built, and who else could see it.\nSo I moved GitLab in-house. I run it self-hosted now, on my own hardware, next to the clusters it deploys to. That’s sovereignty applied to CI/CD: my code, my builds, my artifacts, no vendor that can change pricing, deprecate a feature I depend on, or read my repositories without me knowing.\nNote added 2026-09-23. That paragraph was true when I wrote it and is not true now. My own forge is Forgejo these days, and the GitLab in my week belongs to a client. The argument for moving CI in-house is the part I still stand behind, and everything below about GitLab CI still runs, so I am leaving the post rather than rewriting it. Read the first person in it as 2025.\nThe reasons stack up once you start counting:\nData sovereignty: your code, your builds, your artifacts stay on your infrastructure No usage limits: unlimited CI minutes, unlimited storage, unlimited users Network locality: builds run close to your clusters, so artifact transfers are fast Customisation: configure runners exactly how you need them Air-gap capable: works in offline environments None of this is free. You maintain GitLab now. You patch it, back it up, and you own the 2am page when a runner pod gets OOM-killed. For me that overhead buys something I can’t get any other way, so I pay it gladly.\nThat settles where the pipeline runs. The harder question is what it should actually do, and the part most people get wrong is the last step.\nThe complication: how does code get from a commit into a running pod, safely? Building an image is easy. Anyone can write a docker build. The mess starts after the image exists. How do you stop a vulnerable image reaching production? Where do the cluster credentials live, and how many places get to hold them? When a deploy goes sideways, can you tell what changed and roll it back without spelunking through job logs?\nThe naive answer is to give the CI pipeline a kubeconfig and let it run kubectl apply at the end. It works on day one. By day ninety you’ve got cluster admin credentials sitting in a dozen project pipelines, no audit trail of what’s actually running, and a deploy process that only exists as imperative shell commands inside a YAML file.\nI’d rather draw a hard line between what CI is good at and what it has no business touching.\nThe Pipeline Architecture A Kubernetes deployment pipeline has distinct stages:\nGitLab CI Pipeline\nBuild\nCompile, Lint,\nBuild image\nTest\nUnit, Integr.,\nE2E\nScan\nTrivy, SAST,\nSecrets\nPublish\nPush to Registry,\nSign\nDeploy\nUpdate GitOps\nRepo\nEach stage has a clear purpose, and the last one stops at the GitOps repo rather than the cluster. More on why that matters below. Let’s build it.\nThe Complete .gitlab-ci.yml Here’s a production-ready pipeline:\nstages: - build - test - scan - publish - deploy variables: # Container settings DOCKER_HOST: tcp://docker:2376 DOCKER_TLS_CERTDIR: \"/certs\" DOCKER_TLS_VERIFY: 1 DOCKER_CERT_PATH: \"$DOCKER_TLS_CERTDIR/client\" # Image naming IMAGE_NAME: $CI_REGISTRY_IMAGE IMAGE_TAG: $CI_COMMIT_SHORT_SHA # Kubernetes KUBE_NAMESPACE: $CI_PROJECT_NAME # Reusable configurations .docker-base: image: docker:24 services: - docker:24-dind before_script: - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY # ============================================ # BUILD STAGE # ============================================ build: extends: .docker-base stage: build script: - docker build --cache-from $IMAGE_NAME:latest --tag $IMAGE_NAME:$IMAGE_TAG --tag $IMAGE_NAME:latest --build-arg BUILDKIT_INLINE_CACHE=1 . - docker push $IMAGE_NAME:$IMAGE_TAG - docker push $IMAGE_NAME:latest rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH - if: $CI_MERGE_REQUEST_ID lint: stage: build image: golangci/golangci-lint:latest # Adjust for your language script: - golangci-lint run ./... rules: - if: $CI_MERGE_REQUEST_ID # ============================================ # TEST STAGE # ============================================ unit-tests: stage: test image: golang:1.22 # Adjust for your language script: - go test -v -race -coverprofile=coverage.out ./... - go tool cover -func=coverage.out coverage: '/total:\\s+\\(statements\\)\\s+(\\d+\\.\\d+)%/' artifacts: reports: coverage_report: coverage_format: cobertura path: coverage.xml rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH - if: $CI_MERGE_REQUEST_ID integration-tests: stage: test extends: .docker-base script: - docker compose -f docker-compose.test.yml up -d - docker compose -f docker-compose.test.yml run tests - docker compose -f docker-compose.test.yml down rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH # ============================================ # SCAN STAGE # ============================================ container-scan: stage: scan image: name: aquasec/trivy:latest entrypoint: [\"\"] script: - trivy image --exit-code 1 --severity HIGH,CRITICAL --ignore-unfixed $IMAGE_NAME:$IMAGE_TAG rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH - if: $CI_MERGE_REQUEST_ID allow_failure: false sast: stage: scan image: returntocorp/semgrep script: - semgrep --config auto --error . rules: - if: $CI_MERGE_REQUEST_ID secret-detection: stage: scan image: trufflesecurity/trufflehog:latest script: - trufflehog git file://. --only-verified --fail rules: - if: $CI_MERGE_REQUEST_ID # ============================================ # PUBLISH STAGE # ============================================ publish: extends: .docker-base stage: publish script: # Tag with semantic version if tagged - | if [ -n \"$CI_COMMIT_TAG\" ]; then docker pull $IMAGE_NAME:$IMAGE_TAG docker tag $IMAGE_NAME:$IMAGE_TAG $IMAGE_NAME:$CI_COMMIT_TAG docker push $IMAGE_NAME:$CI_COMMIT_TAG fi rules: - if: $CI_COMMIT_TAG # ============================================ # DEPLOY STAGE # ============================================ deploy-staging: stage: deploy image: alpine:latest before_script: - apk add --no-cache git openssh-client - eval $(ssh-agent -s) - echo \"$GITOPS_SSH_KEY\" | tr -d '\\r' | ssh-add - - mkdir -p ~/.ssh \u0026\u0026 chmod 700 ~/.ssh - ssh-keyscan gitlab.com \u003e\u003e ~/.ssh/known_hosts script: - git clone git@gitlab.com:$GITOPS_REPO.git gitops - cd gitops - | # Update image tag in Kustomize cd apps/$CI_PROJECT_NAME/overlays/staging kustomize edit set image $IMAGE_NAME:$IMAGE_TAG - git config user.email \"ci@gitlab.local\" - git config user.name \"GitLab CI\" - git add -A - git commit -m \"Deploy $CI_PROJECT_NAME:$IMAGE_TAG to staging\" || exit 0 - git push environment: name: staging url: https://staging.example.com rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH deploy-production: stage: deploy image: alpine:latest before_script: - apk add --no-cache git openssh-client - eval $(ssh-agent -s) - echo \"$GITOPS_SSH_KEY\" | tr -d '\\r' | ssh-add - - mkdir -p ~/.ssh \u0026\u0026 chmod 700 ~/.ssh - ssh-keyscan gitlab.com \u003e\u003e ~/.ssh/known_hosts script: - git clone git@gitlab.com:$GITOPS_REPO.git gitops - cd gitops - | cd apps/$CI_PROJECT_NAME/overlays/production kustomize edit set image $IMAGE_NAME:$IMAGE_TAG - git config user.email \"ci@gitlab.local\" - git config user.name \"GitLab CI\" - git add -A - git commit -m \"Deploy $CI_PROJECT_NAME:$IMAGE_TAG to production\" || exit 0 - git push environment: name: production url: https://example.com rules: - if: $CI_COMMIT_TAG when: manual # Require manual approval for production That’s a lot of YAML at once. Let me walk through the stages that earn their keep.\nBreaking Down the Pipeline Build Stage: Creating the Container build: extends: .docker-base stage: build script: - docker build --cache-from $IMAGE_NAME:latest --tag $IMAGE_NAME:$IMAGE_TAG --tag $IMAGE_NAME:latest . Key points:\nCache from previous builds: --cache-from speeds up builds dramatically Commit SHA as tag: immutable, traceable image versions. The tag tells you exactly which commit produced the image, which you’ll want the first time a deploy misbehaves Also tag latest: for the cache source in the next build Test Stage: Verify Before Deploy Tests run in parallel where possible:\nunit-tests: # ... coverage: '/total:\\s+\\(statements\\)\\s+(\\d+\\.\\d+)%/' The coverage regex extracts the coverage percentage so GitLab can show it on the merge request. Small thing, but it puts the number in front of you at review time instead of buried in a log nobody opens.\nScan Stage: Security Gates This is where the pipeline starts saying no on your behalf. Three scans, each catching a different class of mistake:\nContainer scanning (Trivy): finds vulnerabilities in container images SAST: static analysis of source code Secret detection: catches credentials someone accidentally committed container-scan: script: - trivy image --exit-code 1 # Fail pipeline on findings --severity HIGH,CRITICAL # Only block on serious issues --ignore-unfixed # Skip vulnerabilities without fixes Deploy Stage: GitOps Integration Here’s the answer to the question I posed up top. The pipeline never touches the cluster. It updates a GitOps repository, and ArgoCD does the actual deployment by reconciling the cluster against what’s now in Git.\ndeploy-staging: script: - git clone git@gitlab.com:$GITOPS_REPO.git gitops - cd gitops/apps/$CI_PROJECT_NAME/overlays/staging - kustomize edit set image $IMAGE_NAME:$IMAGE_TAG - git commit -m \"Deploy $CI_PROJECT_NAME:$IMAGE_TAG to staging\" - git push This pattern:\nClones the GitOps repository Updates the image tag using Kustomize Commits and pushes ArgoCD detects the change and deploys The win is in what’s missing. The application repo never holds cluster credentials. It only needs write access to one GitOps repo. Compromise a project pipeline and the blast radius is a Git commit someone will notice on review, not a kubeconfig with the keys to the cluster. Every deploy is now a commit with a diff, an author, and a timestamp, so the question “what’s running right now and who put it there” has a one-line answer. And when production catches fire, rollback is git revert, which is muscle memory rather than a fresh round of imperative kubectl under pressure.\nThat separation is the whole point. CI builds and proves the artifact; Git records intent; ArgoCD makes reality match. Each part does the thing it’s actually good at.\nGitLab Runners for Kubernetes None of this runs without GitLab Runners to execute the jobs. I run them in Kubernetes too, so the same cluster that hosts my workloads also builds them:\n# values.yaml for gitlab-runner Helm chart gitlabUrl: https://gitlab.example.com runnerRegistrationToken: \"your-token\" runners: config: | [[runners]] [runners.kubernetes] namespace = \"gitlab-runners\" image = \"alpine:latest\" privileged = true # Required for Docker-in-Docker [[runners.kubernetes.volumes.empty_dir]] name = \"docker-certs\" mount_path = \"/certs/client\" medium = \"Memory\" rbac: create: true resources: limits: memory: 256Mi cpu: 250m Install with Helm :\nhelm repo add gitlab https://charts.gitlab.io helm install gitlab-runner gitlab/gitlab-runner -f values.yaml -n gitlab-runners Environment Variables and Secrets Store sensitive values in GitLab CI/CD variables:\nProject Settings → CI/CD → Variables\nVariable Protected Masked Description GITOPS_SSH_KEY ✓ ✓ SSH key for GitOps repo KUBECONFIG ✓ ✓ Kubernetes config (if direct deploy) SONAR_TOKEN ✓ ✓ SonarQube token Protected: only available on protected branches Masked: hidden in job logs\nMark the GitOps SSH key both protected and masked. A feature branch should never be able to push a deploy, and the key should never show up in a log a teammate can scroll back through.\nMerge Request Pipelines You don’t need the full pipeline on every push. Merge requests run a subset so feedback comes back fast:\nlint: rules: - if: $CI_MERGE_REQUEST_ID # Only on MRs unit-tests: rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH - if: $CI_MERGE_REQUEST_ID # Also on MRs Fast feedback on the MR, the heavier jobs reserved for the default branch. A pipeline you wait ten minutes for on every push is a pipeline people start skipping.\nCaching for Speed The other thing that keeps a pipeline from getting ignored is speed, and caching does most of the work:\nvariables: GOPATH: $CI_PROJECT_DIR/.go cache: key: ${CI_COMMIT_REF_SLUG} paths: - .go/pkg/mod/ - node_modules/ - .npm/ For Docker layer caching, use BuildKit:\nbuild: variables: DOCKER_BUILDKIT: 1 script: - docker build --build-arg BUILDKIT_INLINE_CACHE=1 ... Monitoring Pipeline Health A pipeline is infrastructure, and infrastructure you don’t watch is infrastructure you don’t understand. Scrape GitLab’s own metrics:\n# In your monitoring stack - job_name: 'gitlab' static_configs: - targets: ['gitlab.example.com'] metrics_path: '/-/metrics' Key metrics to watch:\nPipeline duration (creeping up means caching is failing somewhere) Success/failure rate Queue time (climbing queue time means runners are saturated) Cache hit rate The Complete Flow Developer\npushes code\nGitLab\ntriggers CI\nPipeline\nBuild → Test → Scan → Publish → Update GitOps\nGitOps Repo\n(updated)\nArgoCD\ndetects \u0026 deploys\nKubernetes\n(running)\nClean separation of concerns:\nGitLab CI: build, test, scan, publish GitOps repo: desired state ArgoCD: reconciliation Kubernetes: runtime Each box knows nothing about the others’ internals. CI doesn’t know how the cluster applies manifests. The cluster doesn’t know how the image got built. They meet at a Git commit, which is the most boring and most inspectable interface I could ask for.\nMy Recommendations Self-host GitLab if you care about sovereignty. The operational overhead is real, and it’s worth it.\nNever deploy directly from CI to Kubernetes. Update GitOps repos and let a reconciler do the applying.\nFail fast on security: container scans with --exit-code 1 block vulnerable images before they ship.\nUse protected variables: secrets should only be reachable from protected branches.\nManual production deploys: keep a human in the loop for production.\nThe pipeline I trust most is the one I’d be happy to explain to a stranger line by line. Build, prove, record, reconcile. Nothing magic, nothing hidden, and a Git history that tells you exactly what happened when something breaks.","permalink":"https://tommeurs.nl/en/posts/gitlab-ci-kubernetes/","title":"GitLab CI for Kubernetes: From Commit to Deployment"},{"content":"The whole pitch of GitOps is that Git is the source of truth. That promise holds right up until someone runs kubectl edit on a deployment at 2am to stop an incident, a mutating webhook quietly rewrites a resource, or a half-finished sync leaves your cluster somewhere between what Git wanted and what it got. Now Git says one thing and the cluster does another, and nobody told you.\nThat gap is configuration drift, and it is the part of GitOps people forget to defend. The good news: ArgoCD already watches for it. The catch is that the defaults don’t do what you probably assume, and a few of them will bite you. This post walks from the simplest possible drift check up to the setup I actually run, one layer at a time. Stop wherever you have enough.\nWhat Drift Actually Is Drift is the moment the real state of your cluster stops matching the desired state in Git.\n!=\nCluster has (Actual state)\nreplicas: 5\nimage: v1.2.3\ncpu: 200m\nGit says (Source of truth)\nreplicas: 3\nimage: v1.2.3\ncpu: 100m\nGit says three replicas, the cluster runs five. How does that happen? A few usual suspects:\nManual changes: someone ran kubectl scale or kubectl edit and meant to fix it in Git later Horizontal Pod Autoscaler: HPA bumped replicas on its own Mutating webhooks: an admission controller rewrote the resource on the way in Controller side effects: an operator changed a field it manages Partial syncs: a sync died halfway through Some of that drift is wanted (HPA doing its job). Most of it is not. The real problem is that from the outside you can’t tell the wanted kind from the dangerous kind, so you have to treat all of it as a signal.\nWhy I Care About This Undetected drift quietly cancels the reason you adopted GitOps in the first place. Four things break:\nAudit trail: “what’s deployed?” stops being “read the Git history” and becomes “go poke the live cluster” Disaster recovery: rebuild from Git and you get a cluster that doesn’t match the one you lost Security: an unauthorised change to a Deployment sits there unnoticed Reproducibility: two clusters built from the same repo end up subtly different This connects straight to the sovereignty thing I keep banging on about: I want to understand what I’m running, and a cluster that has silently wandered off from its declared state is a black box I built for myself. Drift detection is how I keep Git honest.\nThe Simplest Version: Sync Status You don’t need any extra tooling to start. ArgoCD already compares Git to the live cluster on a loop and reports one of three states:\nSynced: cluster matches Git exactly OutOfSync: it found differences Unknown: it can’t figure out the state Application Sync Status Health frontend Synced Healthy backend OutOfSync Healthy database Synced Healthy cache Synced Degraded OutOfSync is your drift signal. That’s the entire basic version: glance at the dashboard, anything not green is drift. It works, but staring at a dashboard is a terrible detection strategy, and the default sync behaviour might not be what you expect. So let’s add a layer.\nLayer 1: Self-Heal, and Why It’s a Double-Edged Tool ArgoCD can revert drift for you automatically:\napiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: frontend spec: syncPolicy: automated: selfHeal: true # Revert manual changes prune: true # Delete orphaned resources With selfHeal: true, the moment someone runs kubectl scale deployment frontend --replicas=5, ArgoCD notices the gap and puts it back to whatever Git says, usually within seconds. This is the self-healing infrastructure ideal in action: the system corrects itself instead of waiting for a human to notice.\nIt also has sharp edges. That same automation will revert an intentional change, stomp on an HPA adjustment, and make a genuine emergency hotfix bounce right back. For most apps I leave selfHeal on anyway, because the cost of letting drift accumulate is higher than the cost of the occasional reverted change. Just go in knowing the trade-off rather than discovering it during an incident.\nLayer 1b: Telling ArgoCD What to Ignore Some fields are supposed to live outside Git, and self-heal will fight you over them unless you say so. The classic case is replica count when HPA owns it:\napiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: autoscaled-app spec: ignoreDifferences: - group: apps kind: Deployment jsonPointers: - /spec/replicas - group: \"\" kind: Service jsonPointers: - /spec/clusterIP This tells ArgoCD to stop reporting drift on those specific fields. The ones I reach for most:\n/spec/replicas when HPA is in charge /spec/clusterIP because Kubernetes assigns it /metadata/annotations that controllers add on their own /status, which is always controller-managed The goal here is to stop the noise so that when something does show as OutOfSync, it means something.\nLayer 2: Detect Without Auto-Fixing Self-heal is the right default, but not everywhere. For the systems where I want a human to look before anything moves, I turn the automation off and keep the detection on:\napiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: critical-app spec: syncPolicy: automated: selfHeal: false # Don't auto-fix prune: false # Don't auto-delete Now ArgoCD still flags OutOfSync and shows you the diff, but it waits for you to decide. This earns its keep for critical production where you want eyes on every change, for debugging the source of recurring drift, and for apps that are deliberately managed partly outside GitOps.\nLayer 2b: Get Notified Instead of Watching Detection only helps if it reaches you. Wire up ArgoCD notifications so drift comes to you rather than you going to the dashboard:\napiVersion: v1 kind: ConfigMap metadata: name: argocd-notifications-cm namespace: argocd data: trigger.on-sync-status-unknown: | - when: app.status.sync.status == 'OutOfSync' send: [app-out-of-sync] template.app-out-of-sync: | message: | Application {{.app.metadata.name}} is OutOfSync. Sync Status: {{.app.status.sync.status}} Health: {{.app.status.health.status}} Repository: {{.app.spec.source.repoURL}} Point that at Slack, PagerDuty, or email. Drift should make a noise.\nReading the Diff When something drifts, ArgoCD will tell you exactly what changed, which is half the battle:\nargocd app diff frontend Or click the OutOfSync app in the UI to see the same thing:\n--- Git (desired) +++ Cluster (actual) @@ -1,4 +1,4 @@ spec: - replicas: 3 + replicas: 5 template: spec: This is where understanding beats guessing. You see the field, the old value, the new value, and you can start asking who or what did it.\nOne Thing Worth Getting Straight: Refresh vs Sync These two operations sound similar and do very different things, and mixing them up causes accidents.\nRefresh compares Git to the cluster and updates the status. It changes nothing.\nargocd app get frontend --refresh Sync applies the Git state to the cluster. It changes things.\nargocd app sync frontend Refresh is safe and runs on its own every three minutes by default. Sync alters live state, so do it deliberately (or let automation do it for you, knowing what that means).\nThe Full Picture: My Drift Strategy Here’s how I actually split this across environments. None of it is exotic; it’s just the layers above, applied with a bit of judgement.\nDevelopment / Staging selfHeal: true to revert all drift prune: true to clean up orphaned resources Fast feedback, no ceremony, pure GitOps Production (most apps) selfHeal: true and prune: true alerts on every OutOfSync event and crucially, investigate why it drifted rather than just letting the heal paper over it Production (critical or special) selfHeal: false, human review required prune: false, manual deletion only strict alerts and an explicit sync approval before anything changes HPA-managed apps ignoreDifferences: - group: apps kind: Deployment jsonPointers: - /spec/replicas syncPolicy: automated: selfHeal: true # For other fields When Drift Shows Up, Chase the Source Healing the drift fixes the symptom. Finding out why it happened fixes the problem:\nCheck audit logs: who ran kubectl?\nkubectl get events --field-selector reason=Update Check controller logs: did an operator do it?\nCheck admission webhooks: is something mutating resources on admission?\nkubectl get mutatingwebhookconfigurations Check the diff: what changed, exactly?\nargocd app diff app-name Better Than Detecting: Don’t Let It Drift Detection is the safety net. The cheaper win is making most drift impossible in the first place:\nRBAC restrictions: limit who can change resources at all\napiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: readonly rules: - apiGroups: [\"*\"] resources: [\"*\"] verbs: [\"get\", \"list\", \"watch\"] # No create/update/delete Policy enforcement: use Kyverno to block manual changes outright\napiVersion: kyverno.io/v1 kind: ClusterPolicy metadata: name: require-gitops spec: rules: - name: block-manual-changes match: resources: kinds: - Deployment exclude: subjects: - kind: ServiceAccount name: argocd-application-controller validate: message: \"Changes must go through GitOps\" deny: {} Training: get the team into the habit of changing Git, not the cluster. Tooling helps, but a shared norm helps more.\nTrack Drift as a Metric If you run Prometheus, treat drift as something you measure over time, not just an instantaneous status:\n# Prometheus query for out-of-sync apps count(argocd_app_info{sync_status=\"OutOfSync\"}) And alert when it stays non-zero for longer than you’d expect a sync to take:\n- alert: GitOpsDriftDetected expr: count(argocd_app_info{sync_status=\"OutOfSync\"}) \u003e 0 for: 10m labels: severity: warning annotations: summary: \"GitOps drift detected\" description: \"One or more applications are OutOfSync with Git\" My Checklist for Drift-Free GitOps [ ] selfHeal enabled for most applications [ ] ignoreDifferences configured for HPA-managed replicas [ ] Notifications set up for OutOfSync events [ ] RBAC restricts direct cluster modifications [ ] Policy enforcement prevents manual changes [ ] Monitoring alerts on drift [ ] Team trained on GitOps workflow Look back at where we started: a dashboard with a green or red light next to each app. Everything since then is just making that signal trustworthy and acting on it before it costs you. Drift is the quiet erosion of the one guarantee GitOps gives you, which is that Git reflects reality. Detect it, heal it automatically where that’s safe, and dig into why it happened the rest of the time. That last part is what keeps the cluster from becoming a black box you built yourself.","permalink":"https://tommeurs.nl/en/posts/argocd-drift-detection/","title":"Drift Detection with ArgoCD: How to Know If Your Cluster Is Still in Sync"},{"content":"My homelab started with one ArgoCD Application. Then a handful. The day I caught myself running kubectl apply -f for the fifteenth time to register yet another Application, I knew I’d built the exact thing I was trying to avoid: manual steps I had to remember, in an order I had to remember, with no record of what should exist.\nThe App-of-Apps pattern fixes that with one idea. You create a single root Application by hand, and it creates everything else. After a cluster wipe I can rebuild the whole thing with one kubectl apply. That property is the entire reason I run it, and it’s the same reason I self-host in the first place: I want the repository to be the truth, not my memory.\nThis is how I structure every GitOps repo now, and the shape holds whether it’s three apps in my homelab or a hundred at work.\nThe Problem: Application Sprawl When you first adopt ArgoCD, you create Applications by hand:\nkubectl apply -f apps/frontend.yaml kubectl apply -f apps/backend.yaml kubectl apply -f apps/database.yaml # repeat for every service This is fine for a couple of services. Past that it starts to hurt:\nManual creation: every new service needs a kubectl apply you have to run and remember No hierarchy: every Application is a peer, with no logical grouping Difficult onboarding: a new teammate (or future me) has no single place that lists what exists Bootstrap problem: rebuild a cluster and you have to recreate every Application by hand, in the right order Here’s the part that always bugged me. You adopted GitOps so your cluster matches a Git repo. Then you went and created the Applications themselves with imperative commands that live nowhere. The deployments are declarative; the thing that defines the deployments is a pile of tribal knowledge.\nThe Solution: App-of-Apps App-of-Apps is one rule: Applications can create Applications.\nRoot Application\n(manages everything)\nInfrastructure App\nPlatform App\nApps App\nnamespaces\nrbac\nnetpols\nmonitoring\nlogging\ningress\nfrontend\nbackend\nworkers\nThe root Application syncs a directory full of Application manifests. Those Applications in turn sync the actual workloads. Once you see it, it’s obvious: an Application is just another Kubernetes resource, so of course ArgoCD can manage Applications the same way it manages Deployments.\nThe Repository Structure Here’s how I lay out a GitOps repo:\ngitops-repo/ ├── clusters/ │ ├── production/ │ │ ├── root.yaml # Root application │ │ ├── infrastructure/ │ │ │ ├── app.yaml # Infrastructure app-of-apps │ │ │ ├── namespaces.yaml │ │ │ ├── rbac.yaml │ │ │ └── network-policies.yaml │ │ ├── platform/ │ │ │ ├── app.yaml # Platform app-of-apps │ │ │ ├── monitoring.yaml │ │ │ ├── logging.yaml │ │ │ └── ingress.yaml │ │ └── apps/ │ │ ├── app.yaml # Workloads app-of-apps │ │ ├── frontend.yaml │ │ ├── backend.yaml │ │ └── workers.yaml │ └── staging/ │ └── (same shape as production) ├── base/ │ ├── monitoring/ │ │ ├── kustomization.yaml │ │ └── (manifests) │ └── (more components) └── apps/ ├── frontend/ │ ├── base/ │ └── overlays/ │ ├── staging/ │ └── production/ └── backend/ └── (same shape) Building the Hierarchy Level 1: The Root Application The root is the only Application you ever touch by hand. Everything below it is downstream of this one file:\n# clusters/production/root.yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: root namespace: argocd finalizers: - resources-finalizer.argocd.argoproj.io spec: project: default source: repoURL: https://github.com/yourorg/gitops-repo.git targetRevision: main path: clusters/production destination: server: https://kubernetes.default.svc namespace: argocd syncPolicy: automated: prune: true selfHeal: true This syncs the clusters/production directory, which holds the next layer of apps.\nLevel 2: Category Applications Group related applications by what they are. I split mine into infrastructure, platform, and workloads, because those layers have different blast radius and different deploy timing:\n# clusters/production/infrastructure/app.yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: infrastructure namespace: argocd spec: project: default source: repoURL: https://github.com/yourorg/gitops-repo.git targetRevision: main path: clusters/production/infrastructure destination: server: https://kubernetes.default.svc namespace: argocd syncPolicy: automated: prune: true selfHeal: true This syncs every YAML file in the infrastructure directory, including the individual Applications inside it.\nLevel 3: Leaf Applications The actual workload applications:\n# clusters/production/apps/frontend.yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: frontend namespace: argocd spec: project: default source: repoURL: https://github.com/yourorg/gitops-repo.git targetRevision: main path: apps/frontend/overlays/production destination: server: https://kubernetes.default.svc namespace: frontend syncPolicy: syncOptions: - CreateNamespace=true automated: prune: true selfHeal: true Sync Waves: Controlling Order Some things genuinely have to come first. A namespace before the pods that live in it. A database before the app that connects to it. cert-manager before anything that wants a certificate. If everything fires at once you get a flurry of errors that all resolve on the next retry, which works but looks alarming and wastes time.\nArgoCD sync waves give you a deploy order without you having to orchestrate it:\napiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: namespaces annotations: argocd.argoproj.io/sync-wave: \"-10\" # Deploy first --- apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: monitoring annotations: argocd.argoproj.io/sync-wave: \"-5\" # After namespaces --- apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: frontend annotations: argocd.argoproj.io/sync-wave: \"0\" # Default, after platform Lower numbers sync first. I use:\n-10: Namespaces, RBAC -5: Platform components (monitoring, logging, ingress) 0: Applications 10: Post-deployment jobs Health Checks and Dependencies Sync waves only matter because ArgoCD waits for one wave to be healthy before it starts the next:\napiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: database spec: # source and destination as above syncPolicy: automated: selfHeal: true syncOptions: - CreateNamespace=true # Health determines when \"ready\" The database Application reports healthy once its pods are running. Anything in a later wave that depends on it sits and waits until that’s true. You get ordering for free, derived from real health rather than a hardcoded sleep.\nMulti-Cluster with App-of-Apps Once you have more than one cluster, the hierarchy just grows another level at the top:\ngitops-repo/ ├── clusters/ │ ├── production-eu/ │ │ └── root.yaml │ ├── production-us/ │ │ └── root.yaml │ └── staging/ │ └── root.yaml Each cluster gets its own root, and they share base config through Kustomize overlays so you’re not copy-pasting the same monitoring stack three times.\nAdding a New Application Here’s the payoff that makes the pattern worth the setup. Adding a new application is one file plus a git push:\n# 1. Create the Application manifest cat \u003e clusters/production/apps/new-service.yaml \u003c\u003cEOF apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: new-service namespace: argocd spec: project: default source: repoURL: https://github.com/yourorg/gitops-repo.git path: apps/new-service/overlays/production targetRevision: main destination: server: https://kubernetes.default.svc namespace: new-service syncPolicy: syncOptions: - CreateNamespace=true automated: prune: true selfHeal: true EOF # 2. Create the application manifests mkdir -p apps/new-service/overlays/production # add your Kustomization here # 3. Commit and push git add . git commit -m \"Add new-service\" git push ArgoCD spots the new Application manifest on the next sync and deploys it. No kubectl, no clicking through the UI, and a reviewable diff in Git that tells the next person exactly what changed and why.\nApplicationSets: The Next Level If you find yourself writing near-identical Application manifests over and over, ApplicationSets generate them from a template:\napiVersion: argoproj.io/v1alpha1 kind: ApplicationSet metadata: name: apps namespace: argocd spec: generators: - git: repoURL: https://github.com/yourorg/gitops-repo.git revision: main directories: - path: apps/* template: metadata: name: '{{path.basename}}' spec: project: default source: repoURL: https://github.com/yourorg/gitops-repo.git targetRevision: main path: '{{path}}/overlays/production' destination: server: https://kubernetes.default.svc namespace: '{{path.basename}}' syncPolicy: automated: prune: true selfHeal: true Drop a directory under apps/ and you get an Application, no manifest to write. I reach for this when the apps are uniform. When they need individual tweaks, explicit Application files stay easier to read, so I mix both depending on the layer.\nCommon Mistakes These are the ones that bit me, in roughly the order they bit me.\nMistake 1: Too Deep Nesting Three levels is usually plenty. Go deeper and you spend more time tracing where a thing is defined than actually changing it:\n✗ root → category → subcategory → team → service → component ✓ root → category → service Mistake 2: Forgetting Finalizers This one cost me an afternoon. Without finalizers, deleting an app-of-apps removes the parent but leaves its children running as orphans, no longer managed by anything:\nmetadata: finalizers: - resources-finalizer.argocd.argoproj.io With the finalizer in place, deleting the parent cascades down and cleans up the children too.\nMistake 3: Circular Dependencies Don’t let Application A depend on Application B that depends back on Application A. ArgoCD won’t untangle it for you, and you’ll watch both sit in a permanent progressing state.\nMistake 4: Everything in One Sync Wave Cram everything into wave 0 and you’re back to race conditions, just with extra YAML. The whole point of the structure is ordered, health-gated rollout, so use the waves.\nMy Standard Structure This is what I’ve landed on after enough iterations to stop second-guessing it:\nclusters/{env}/ ├── root.yaml ├── infrastructure/ # Sync wave -10 │ ├── app.yaml │ ├── namespaces.yaml │ ├── rbac.yaml │ └── network-policies.yaml ├── platform/ # Sync wave -5 │ ├── app.yaml │ ├── argocd.yaml # ArgoCD manages itself │ ├── monitoring.yaml │ ├── logging.yaml │ ├── ingress.yaml │ └── cert-manager.yaml ├── data/ # Sync wave -2 │ ├── app.yaml │ └── databases.yaml └── apps/ # Sync wave 0 ├── app.yaml └── {service}.yaml The ordering guarantees the boring but important things:\nNamespaces exist before anything tries to deploy into them Platform tools are up before the apps that depend on them Databases are running before apps try to connect The Bootstrap Problem Solved This is where it pays off. In GitOps Disaster Recovery I argued that recovery should be a procedure you can run half-asleep. App-of-Apps gets you most of the way there:\n# 1. New cluster # 2. Install ArgoCD # 3. Apply ONE file kubectl apply -f clusters/production/root.yaml # 4. Everything recreates One command, and the cluster rebuilds itself from the repo. I’ve run this for real after deliberately wiping a homelab cluster, and watching a hundred resources come back in the right order from a single kubectl apply is the closest infrastructure gets to satisfying.\nThat’s the property I care about. With App-of-Apps the Git repo stops being a folder of files that happen to deploy things and becomes the actual specification of what should exist. When something drifts, ArgoCD pulls it back. When the cluster dies, the repo brings it back. I don’t have to trust my memory of how it was wired, which is exactly the kind of dependency I’m trying to design out of my systems.","permalink":"https://tommeurs.nl/en/posts/argocd-app-of-apps/","title":"App-of-Apps Pattern in ArgoCD: Scalable GitOps Architecture"},{"content":"For a couple of years my homelab cluster ran without much drama. Nodes came and went, workloads shifted around, and the worst I ever had to deal with was the occasional pod stuck in CrashLoopBackOff. The kind of stable where you stop thinking about what happens if it all goes away.\nThen one evening I ran a terraform destroy against the wrong workspace. By the time I noticed, the control plane was gone. Not degraded. Gone.\nThat is the moment disaster recovery stops being a planning slide and becomes a question you have to answer right now: the cluster is gone, the hardware is dead, the region is down, or you fat-fingered a destroy. What do you do next?\nIf you have been doing GitOps properly, the answer is almost boring. You provision a fresh cluster, point ArgoCD at Git, and wait. Your infrastructure rebuilds itself from the repository. That is the whole promise: Git is your backup.\nI want to walk through why that works, what has to be true for it to work, and where it quietly fails you. Because the first time I tested this for real, I learned that “Git is your backup” is true for infrastructure and a dangerous lie for data.\nWhy GitOps Changes Disaster Recovery Traditional DR is a second copy of a primary thing. You take the cluster, you snapshot etcd, you run Velero , you ship those backups somewhere safe, and you keep a restore procedure you hopefully tested. Recovery time gets measured in hours, and most of those hours go to copying state back into place.\nGitOps flips which thing is primary. The cluster stops being the source of truth and becomes a derived artifact. The real state lives in Git. The cluster is just the current rendering of that state, and you can throw it away and render it again.\nGitOps\nsync\nGit\n(primary)\nCluster\n(derived)\nTraditional\nbackup\nCluster\n(primary)\nBackup\n(secondary)\nOnce you internalise that, recovery changes shape. You are not restoring a cluster. You are bootstrapping a new one and letting it reconcile. Recovery time drops to minutes, because there is no separate backup to copy back. This is resilience by design rather than resilience bolted on: the system that builds your cluster is the same system that rebuilds it.\nIt also fits how I want to understand my infrastructure. When the whole thing is declared in a repo I can read, there are no hidden bits of state that only exist because someone ran kubectl apply at 2am and never wrote it down.\nWhat Has to Be in Git for This to Work Here is the catch, and it is where most GitOps setups quietly fail their own DR test: this only works if everything is actually in Git. Anything that exists only in the running cluster is something you will lose and have to remember to recreate by hand, under pressure, while production is down.\nSo before walking through recovery, walk through what needs to live in the repo.\nApplication manifests All your workload definitions. Deployments, Services, ConfigMaps, Helm charts with their values files, Kustomize overlays. This is the easy part, and most teams already have it. If you are doing GitOps at all, your apps are here.\nInfrastructure configuration The cluster-level resources that are easy to forget because nobody deploys them as part of a release:\nNamespaces RBAC (Roles, RoleBindings, ServiceAccounts) NetworkPolicies ResourceQuotas PodSecurityPolicies / Pod Security Standards These define your security boundaries and isolation. A cluster you rebuilt without its NetworkPolicies is a cluster with the blast doors left open. Treat them as first-class GitOps content, not an afterthought.\nPlatform components The tools your applications assume are already there:\nIngress controllers Cert-manager Monitoring stack (Prometheus, Grafana) Logging (Loki , Fluentd) Service mesh, if you run one If these are not in Git, your apps come back but nothing routes to them, nothing has a certificate, and you are flying blind because your observability stack never deployed.\nSecrets strategy Secrets are the one thing you genuinely cannot commit in plaintext, so they need a strategy that survives losing the cluster:\nSealed Secrets: encrypted in Git, decrypted in-cluster by a controller External Secrets: fetched at runtime from Vault or a cloud secrets manager SOPS : files encrypted with keys you hold The property that matters for DR is reproducibility. A secret that exists only inside the dead cluster is a secret you have lost. Whichever tool you pick, the encrypted or referenced form has to be in Git and the decryption key has to live somewhere you can still reach after the cluster is gone.\nPersistent data Here is the hard truth I learned the unpleasant way: GitOps does not back up your data.\nDatabases, object storage, anything stateful, none of it is in Git and none of it comes back when ArgoCD syncs. GitOps rebuilds the infrastructure that runs your data. The data itself needs a separate, tested backup strategy, and if you skip that step you will rebuild a perfect, empty cluster and feel sick about it.\nThe Recovery Process With all of that in Git, recovery becomes a sequence you can actually run. Here is the path I follow.\nStep 0: Have a recovery playbook Before anything breaks, write down how to come back:\nHow to provision a fresh cluster The bootstrap commands for ArgoCD Where Vault lives and where its backups are External dependencies (DNS, load balancers) Then run it. A playbook you have never executed is fiction, and you do not want to discover the gaps while production is down. The first time I “tested” mine, I found three undocumented manual steps that would have cost me an hour at the worst possible moment.\nStep 1: Provision a new cluster Whatever your method, get a working Kubernetes cluster:\n# Terraform cd terraform/clusters/production terraform apply # Or managed Kubernetes gcloud container clusters create production-recovery \\ --region europe-west1 \\ --num-nodes 3 # Or bare metal talosctl apply-config --nodes 10.0.0.10 --file controlplane.yaml How you get the cluster is deliberately separate from GitOps. GitOps takes over once you have a functioning API server to talk to.\nStep 2: Install ArgoCD kubectl create namespace argocd kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml # Wait for it to be ready kubectl wait --for=condition=Ready pods --all -n argocd --timeout=300s Step 3: Restore secrets access Wire External Secrets back to Vault so the rest of the sync has the credentials it needs:\n# Configure Vault authentication for the new cluster kubectl create secret generic vault-token \\ -n external-secrets \\ --from-literal=token=$VAULT_TOKEN Step 4: Connect ArgoCD to Git # Add your GitOps repository argocd repo add https://github.com/yourorg/gitops-repo.git \\ --username git \\ --password $GIT_TOKEN # Or via Secret kubectl apply -f - \u003c\u003cEOF apiVersion: v1 kind: Secret metadata: name: gitops-repo namespace: argocd labels: argocd.argoproj.io/secret-type: repository stringData: url: https://github.com/yourorg/gitops-repo.git username: git password: ${GIT_TOKEN} EOF Step 5: Apply the root application If you use the App-of-Apps pattern (you should), one manifest pulls in everything else:\nkubectl apply -f - \u003c\u003cEOF apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: root namespace: argocd spec: project: default source: repoURL: https://github.com/yourorg/gitops-repo.git targetRevision: main path: clusters/production destination: server: https://kubernetes.default.svc syncPolicy: automated: prune: true selfHeal: true EOF Step 6: Wait This is the part that still feels slightly magic to me. ArgoCD reconciles everything on its own:\nDiscovers all applications under the root app Creates namespaces and RBAC Deploys platform components Deploys your applications Watch it work:\nargocd app list argocd app get root --refresh Or in the UI: kubectl port-forward svc/argocd-server -n argocd 8080:443\nStep 7: Restore data This is where your separate data backup strategy earns its keep:\n# Restore database pg_restore -d myapp \u003c /backups/myapp-latest.dump # Or Velero restore velero restore create --from-backup production-daily-latest Step 8: Update DNS and load balancers Point external traffic at the new cluster:\nUpdate DNS records Reconfigure CDN origins Update load balancer targets How Fast Can You Actually Recover The honest answer is “it depends, and the dependency is your data.” For the infrastructure half, here is what I typically see:\nPhase Time Cluster provisioning 5-15 min ArgoCD install 2-3 min Secrets setup 1-2 min GitOps sync (small cluster) 5-10 min GitOps sync (large cluster) 15-30 min Data restore Varies wildly DNS propagation 5-60 min Infrastructure total: 20-60 minutes.\nData restore is the wildcard. It scales with your data volume and your backup design, and it is almost always the longest leg of the recovery. The clean, fast part is the GitOps part. The slow, anxious part is the data part, which is exactly the part GitOps does not touch.\nWhere It Bites You I have hit most of these the hard way, so here is what actually goes wrong.\nMissing secrets If Vault is unreachable, External Secrets cannot fetch anything and your workloads never start. You rebuild the whole cluster and then watch every pod wait on a secret that will not arrive.\nMitigation: run Vault in HA on infrastructure separate from the cluster it serves, back it up regularly, and consider replicating it to a second location.\nExternal dependencies Your cluster probably leans on things that are not in Git at all:\nCloud IAM roles DNS zones SSL certificates from Let’s Encrypt External databases Mitigation: document every external dependency, and better still, manage them with Terraform or Crossplane in the same repo so they rebuild alongside everything else.\nCircular dependencies ArgoCD has to be running before it can deploy ArgoCD. The classic bootstrap problem, and it will trip you if your only copy of “how to install ArgoCD” was a managed-by-ArgoCD application.\nMitigation: keep a bootstrap script that works with plain kubectl, no ArgoCD required. Once ArgoCD is up, let it manage itself from there.\nData loss GitOps rebuilds infrastructure, not data. A stale database backup means you recover the cluster and still lose hours or days of data.\nMitigation: a separate, tested data backup strategy with real restore tests, not just backup jobs you assume are working.\nThe Recovery Manifest After my own near-miss, I started keeping a single file that documents everything recovery needs in one place. When things are on fire, you do not want to be grep-ing through a repo for the Vault address.\n# recovery-manifest.yaml recovery: cluster: provisioner: terraform path: terraform/clusters/production secrets: method: external-secrets vault-address: https://vault.internal.company.com vault-backup: s3://backups/vault-snapshots/ gitops: repo: https://github.com/yourorg/gitops-repo.git path: clusters/production root-app: apps/root.yaml data: databases: - name: postgres-main backup-location: s3://backups/postgres/ restore-command: | pg_restore -d main /backups/latest.dump volumes: - name: uploads backup-location: s3://backups/uploads/ external-dependencies: - DNS zone: managed in Route53, terraform/dns/ - Load balancer: created by terraform/clusters/ - Vault: vault.internal.company.com (HA, separate infra) contacts: - oncall: ops-team@company.com - escalation: platform-lead@company.com Commit it to Git. Review it quarterly. Test it at least once a year.\nThe Mindset This Requires GitOps DR asks you to think about your clusters differently, and the shift is worth saying out loud:\nClusters are cattle, not pets. When recovery is cheap, you stop white-knuckling the protection of any single cluster.\nState belongs in Git. Every kubectl apply you run without committing the change is DR debt that comes due at the worst time.\nData is its own problem. GitOps handles infrastructure cleanly and never pretends to handle your data. That boundary is the most important thing to remember.\nPractice recovery. The first time you run a real recovery should not be during a real disaster.\nMy Recovery Checklist ## Pre-Disaster (do now) - [ ] All manifests in Git - [ ] Secrets strategy documented and tested - [ ] Vault backed up and HA configured - [ ] Recovery playbook written - [ ] External dependencies documented - [ ] Data backup strategy implemented - [ ] Recovery tested in last 90 days ## During Recovery - [ ] Provision new cluster - [ ] Install ArgoCD - [ ] Restore secrets decryption capability - [ ] Connect ArgoCD to Git - [ ] Apply root application - [ ] Monitor sync progress - [ ] Restore data from backups - [ ] Update DNS/load balancers - [ ] Verify application health - [ ] Notify stakeholders The best disaster recovery is the one you never need. The second best is the one you have rehearsed often enough that when it finally happens for real, it feels like just another Tuesday.","permalink":"https://tommeurs.nl/en/posts/gitops-disaster-recovery/","title":"GitOps Disaster Recovery: Rebuilding Your Cluster from Git"},{"content":"“Should I use ArgoCD or Flux ?”\nI get this question over coffee, in Slack, in PR comments, probably once a week. People want me to declare a winner so they can stop thinking about it and go build something.\nHere is my bias, up front, so you can weigh everything that follows: I run ArgoCD on my own clusters. I have a few years of it under my belt and I like it. That preference is going to leak through no matter how careful I am, so I am telling you now instead of pretending I am neutral.\nBut ArgoCD winning for me does not mean it wins for you. Both tools are CNCF graduated, both are boring in the good way, and the choice depends almost entirely on your context. So let me lay out the criteria I actually care about, walk both tools through each one, and then tell you where each one fits.\nWhat I’m Comparing On A feature checklist would be useless here. Both tools do Helm , both do Kustomize , both reconcile Git to clusters. If I just listed features you would learn nothing.\nThe things that decide this for me:\nOperating model - how the tool wants to be installed, configured, and upgraded Visibility - what you can see when something goes sideways Failure behaviour - what happens when the GitOps engine itself falls over Multi-cluster shape - how it scales from one cluster to twenty These map directly to the values that drive most of my infrastructure choices. Understanding what I run (visibility), designing for things breaking (failure behaviour), and keeping operational friction low (operating model) are not abstract preferences. They decide which tool I can live with at 2am.\nThe Philosophy Split Before the criteria, you need to understand that these tools disagree about something fundamental.\nArgoCD is application-centric. You define Applications that point at Git sources, and ArgoCD manages them through a central control plane that ships with a UI. The control plane is a thing you run and look after.\nFlux is GitOps-native to the bone. It installs a set of controllers that watch Git repositories, and the entire system, including Flux’s own configuration, lives in Git. There is no central anything. There is no UI.\nThis split is the root of nearly every difference that follows.\nAspect ArgoCD Flux Architecture Centralized Distributed Configuration CRDs + UI Pure GitOps Bootstrapping Manual or GitOps GitOps from start Multi-cluster Hub-spoke Per-cluster ArgoCD: The Application Controller ArgoCD watches Git repos and syncs them to clusters. What you get for running a central control plane is visibility and control, in one place, with a screen you can point at.\nArgoCD Architecture\nWatches Git, Syncs to clusters\nManaged Clusters\nCluster 1\nCluster 2\nCluster 3\nArgoCD Control Plane\nAPI Server\nUI\nApplication Controller\nWhere ArgoCD Earns Its Keep The UI is genuinely useful, and I do not say that lightly. I am a terminal person. I avoid mice. And yet when a sync goes wrong, the ArgoCD resource tree showing me exactly which object is out of sync, with the diff right there, is faster than anything I would assemble from kubectl. For someone new to GitOps it does something even more valuable: it makes the link between a Git commit and cluster state visible, so the whole model clicks into place.\nThe application abstraction matches how teams think. Each Application is a unit you sync, roll back, or delete on its own. That maps cleanly onto how people already reason about services, which means less translation in your head.\nMulti-cluster is a hub-spoke story. One ArgoCD installation manages many clusters. For a central platform team that is a tidy model: one place to look, one place to grant access.\nThe ecosystem is rich. Argo Rollouts for progressive delivery, ApplicationSets for generating Applications dynamically, Notifications for alerting. For most needs there is already a tool that plugs in.\nWhere ArgoCD Costs You It is a system you operate. ArgoCD has its own state in Redis, needs regular upgrades, and can get unhappy at scale. You are not just running your apps now, you are running the thing that runs your apps.\nIt is not pure GitOps by default. ArgoCD’s own config tends to live in the UI or get applied with kubectl, rather than being managed by ArgoCD itself. You can close that gap with App-of-Apps, where ArgoCD manages its own Applications from Git, but that is work you have to do deliberately.\nThe control plane is a single point of failure. If ArgoCD goes down, syncing stops. Your clusters keep serving traffic, the reconciliation engine is what is offline, but you cannot ship updates until it comes back.\nFlux: The GitOps Toolkit Flux comes at the problem from the other side. It is a set of focused controllers, each handling one concern.\nFlux Architecture\nFlux controllers\nEach Cluster\nSource Controller\nKustomize Controller\nHelm Controller\nGit Repository\nclusters/\n├── production/\n│ ├── flux-system/\n│ ├── infrastructure/\n│ └── apps/\n└── staging/\nWhere Flux Earns Its Keep Pure GitOps from the very first command. flux bootstrap writes all of Flux’s configuration into Git. After that, everything goes through pull requests, including Flux’s own upgrades. There is no out-of-band state to drift, because there is no out-of-band anything. If you want a system where Git is the only source of truth with no asterisks, this is it.\nLighter footprint. The controllers are small and focused, you install only the ones you need, and resource usage tends to land below ArgoCD. On small or constrained clusters that difference is real.\nNo central thing to lose. Each cluster runs its own Flux. One cluster’s Flux falling over does not touch the others, because there is no shared control plane to protect in the first place.\nIt nudges you toward GitOps discipline. With no UI to click, the path of least resistance is a pull request. The tool quietly removes the temptation to make a manual change, which is a feature disguised as an absence.\nWhere Flux Costs You No UI. Flux ships no web interface. You work through kubectl, the flux CLI, or a third-party dashboard like Weave GitOps. For a terminal-native team that is fine. For people still learning GitOps, the missing visibility is a genuine barrier, the inverse of what makes ArgoCD so friendly to newcomers.\nSteeper learning curve. The modular design means more CRDs to hold in your head: GitRepository, Kustomization, HelmRelease, HelmRepository. That modularity is power, and it is also more concepts to learn before anything works.\nMulti-cluster is more effort. Flux runs per cluster. Twenty clusters means twenty Flux installations, and coordinating them needs external tooling. The thing that makes Flux resilient (no central plane) is the same thing that makes fleet management harder.\nHead-to-Head Feature ArgoCD Flux Web UI ✓ Built-in ✗ Third-party CLI ✓ Full featured ✓ Full featured Helm support ✓ Native ✓ Native Kustomize support ✓ Native ✓ Native Multi-cluster ✓ Hub-spoke ✗ Per-cluster Progressive delivery ✓ Argo Rollouts ✓ Flagger Self-managing Partial ✓ Full Resource usage Higher Lower CNCF status Graduated Graduated The “It Depends” Part This is the bit that actually decides things, and a table cannot hold it.\nReach for ArgoCD when:\nYou run a platform for other teams. The UI lets developers see what is deployed without deep kubectl knowledge. Self-service with guardrails, which is exactly what a platform team is selling. You want many clusters from one pane of glass. Hub-spoke is simpler for centralized management. ArgoCD on a management cluster, targeting your workload clusters, is a clean shape. Your team is new to GitOps. The visual feedback builds intuition faster than reading reconciliation logs ever will. You need progressive delivery in the box. Argo Rollouts integrates tightly, so canary deployments with automatic analysis are not a separate project. Reach for Flux when:\nGitOps purity is the point. If you want everything in Git with no exceptions and no manual escape hatch, Flux makes that the default rather than something you have to enforce. Footprint matters. Edge deployments, small clusters, or a fleet large enough that ArgoCD’s overhead multiplies into real cost. You run a decentralized model. Each team owns its cluster and its Flux, with no central platform team in the loop. You are already in that ecosystem. If Weave GitOps or similar tools are part of your world, Flux is the natural fit. There are hybrids too. Some shops run ArgoCD for the developer-facing apps and Flux for cluster bootstrap, letting each tool do the part it is best at. That is more moving pieces, but if the trade-offs line up that way for you, it is a real option rather than a hack.\nMy Take I chose ArgoCD, and tying it back to my criteria from the top: I value the UI for understanding state, hub-spoke fits the multi-cluster shape I actually run, and the Application abstraction matches the way I already think about my workloads. Those are personal reasons rooted in my context, not a verdict.\nFlip a couple of those and my answer flips with them. If I ran a single cluster, with a team of experienced Kubernetes people who lived in the terminal and wanted Git as the only source of truth with zero exceptions, I would seriously run Flux. The case for it is strong. It just is not my case.\nSwitching Later If you pick one and want to move, the good news is the expensive part transfers.\nArgoCD to Flux: export your Application definitions and convert them to Flux Kustomizations and HelmReleases. The Git repos stay put.\nFlux to ArgoCD: turn GitRepositories into ArgoCD repo connections and Kustomizations into Applications. Again, the repos do not move.\nBecause Git is the source of truth in both, the hard work (organizing your manifests sensibly) is the part you keep. The tool sitting on top is the cheap part to swap.\nDon’t Overthink It Here is what I tell people who ask:\nTry both on a throwaway cluster. A day with each tells you more than a week of reading comparisons like this one. Notice which one feels right. GitOps is a workflow before it is a tool. Pick the one that matches how your team already works. Start simple. Either tool covers the common 90% out of the box. Remember you can switch. Your investment lives in your Git repos, so the tool is not a lock-in. The principles are what carry weight: declarative, versioned, reconciled. ArgoCD and Flux are two ways of honouring the same idea. The best one is the one your team will actually use, every day, without grumbling. Perfect GitOps with a tool nobody opens beats either of these gathering dust.","permalink":"https://tommeurs.nl/en/posts/argocd-vs-flux/","title":"ArgoCD vs Flux: When to Choose What"},{"content":"Here is the thing I never want to do again: ask a cluster what it is running and not trust the answer. For years that was normal. Someone ran kubectl apply, maybe from a laptop, maybe from a CI job nobody could find anymore, and the live state slowly drifted away from anything written down. When I switched to GitOps, that whole category of uncertainty disappeared. I push to Git, and the cluster converges to match. If I want to know what is deployed, I read a file.\nArgoCD is the tool I reach for to make that happen. This post walks from the smallest possible version up to the config I actually run, so you can stop wherever you have enough.\nWhy Git as the Source of Truth Start with the old way:\nDeveloper → kubectl apply → Cluster The question this can’t answer is the important one: what is actually deployed right now? You query the cluster and hope nobody touched it by hand. Drift creeps in quietly. Rollbacks are manual, and the only audit trail is “someone ran kubectl, probably.”\nGitOps moves the truth one step back:\nDeveloper → Git Push → ArgoCD → Cluster Now Git holds the desired state, and ArgoCD’s job is to make reality match it. Change something by hand and ArgoCD puts it back. Want to know what is running? Read the repo. Want to roll back? Revert the commit. This is the same instinct I described in Sovereign Infrastructure: I want systems I can inspect, not systems I have to interrogate. Imperative commands vanish the moment they finish. Declarative state in Git stays, with a history attached.\nThe Simplest Version Before any of the concepts, let me get ArgoCD running so you have something to click on. One namespace, one manifest.\n# Create namespace kubectl create namespace argocd # Install ArgoCD kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml # Wait for pods to be ready kubectl wait --for=condition=Ready pods --all -n argocd --timeout=300s Done. ArgoCD is up. Now grab the password and open the UI:\n# Get the initial admin password kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath=\"{.data.password}\" | base64 -d # Port-forward the UI kubectl port-forward svc/argocd-server -n argocd 8080:443 # Open https://localhost:8080, login as 'admin' with the password above That is enough to look around. Three concepts make sense of what you are seeing.\nThe Three Concepts You Need Application The Application is the unit you work with. It answers three questions: where do the manifests live, where should they go, and how aggressively should ArgoCD keep them in sync.\napiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: my-app namespace: argocd spec: project: default source: repoURL: https://github.com/yourorg/yourrepo.git targetRevision: main path: manifests/my-app destination: server: https://kubernetes.default.svc namespace: my-app syncPolicy: automated: prune: true selfHeal: true Project A Project groups applications and sets access boundaries. The default project allows everything, which is fine while you are learning. In production you will want to fence things off so one team can’t deploy into another team’s namespace.\nSync Sync is the act of making the cluster match Git. It comes in three flavours:\nManual: you click the button Automated: ArgoCD acts the moment Git changes Self-healing: ArgoCD also reverts changes made directly against the cluster Layer 1: Your First Real Deployment Time to deploy something. I will keep it boring on purpose: an nginx Deployment and Service, managed entirely through Git.\nStep 1: Create a Git Repository Set up a repo (or reuse one) with this layout:\nmy-gitops-repo/ └── apps/ └── nginx/ ├── deployment.yaml ├── service.yaml └── kustomization.yaml deployment.yaml:\napiVersion: apps/v1 kind: Deployment metadata: name: nginx labels: app: nginx spec: replicas: 2 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.25 ports: - containerPort: 80 service.yaml:\napiVersion: v1 kind: Service metadata: name: nginx spec: selector: app: nginx ports: - port: 80 targetPort: 80 kustomization.yaml:\napiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - deployment.yaml - service.yaml Commit and push.\nStep 2: Point ArgoCD at It # argocd-application.yaml apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: nginx namespace: argocd spec: project: default source: repoURL: https://github.com/YOURUSER/my-gitops-repo.git targetRevision: main path: apps/nginx destination: server: https://kubernetes.default.svc namespace: nginx syncPolicy: syncOptions: - CreateNamespace=true automated: prune: true selfHeal: true Apply it:\nkubectl apply -f argocd-application.yaml Step 3: Watch It Converge Open the ArgoCD UI and your application shows up. With automated sync on, it deploys immediately. Otherwise hit “Sync.”\nArgoCD Dashboard - nginx\nDeployment\nnginx\nReplicaSet\nnginx\nPod x2\nnginx\nService\nnginx\nThe UI draws the resource tree with health and sync status on every node. No guessing about what is live.\nThe Workflow That Replaces kubectl This is the part that changed my habits. Want to bump the replica count?\nEdit in Git: change replicas: 2 to replicas: 3 in deployment.yaml Commit and push ArgoCD detects the change (within 3 minutes by default, or instantly with webhooks) Cluster updates on its own No kubectl, no scripts, no nagging worry about whether I actually ran the command. Git is the record.\nRollback Is Just Revert Something broke after a change? Revert the commit, push, and ArgoCD walks the cluster back.\ngit revert HEAD git push # ArgoCD automatically rolls back the cluster The whole history of the cluster is now sitting in git log:\ngit log --oneline apps/nginx/ # a1b2c3d Scale nginx to 3 replicas # d4e5f6g Initial nginx deployment Self-Healing in Action Turn on selfHeal: true and try to outsmart it:\n# Manually scale the deployment kubectl scale deployment nginx -n nginx --replicas=5 # Wait a moment kubectl get deployment nginx -n nginx # NAME READY UP-TO-DATE AVAILABLE # nginx 3/3 3 3 ArgoCD spotted the drift and pulled it back to what Git says. This is the feature that kills the worst failure mode in infrastructure: the setting someone changed last year that nobody remembers and nobody dares touch. If it is not in Git, it does not survive.\nLayer 2: Sync Policies in Detail The sync policy is where you tune how opinionated ArgoCD gets:\nsyncPolicy: automated: prune: true # Delete resources removed from Git selfHeal: true # Revert manual changes allowEmpty: false # Don't sync if source is empty syncOptions: - CreateNamespace=true - PruneLast=true # Prune after other syncs complete - ApplyOutOfSyncOnly=true # Only apply changed resources retry: limit: 5 backoff: duration: 5s factor: 2 maxDuration: 3m My advice: start with automated sync, prune, and selfHeal all on. That gives you the full experience, where Git really is the truth and the cluster has no secrets. You can always relax it for a specific application that genuinely needs hands-on handling, but make that the exception you reach for deliberately.\nPrivate Repositories Real repos are usually private, so ArgoCD needs credentials:\n# Using the CLI argocd repo add https://github.com/yourorg/private-repo.git \\ --username git \\ --password ghp_yourtoken # Or as a Kubernetes secret kubectl create secret generic private-repo \\ -n argocd \\ --from-literal=url=https://github.com/yourorg/private-repo.git \\ --from-literal=username=git \\ --from-literal=password=ghp_yourtoken kubectl label secret private-repo -n argocd \\ argocd.argoproj.io/secret-type=repository For SSH:\nargocd repo add git@github.com:yourorg/private-repo.git \\ --ssh-private-key-path ~/.ssh/id_rsa Helm and Kustomize ArgoCD speaks both Helm and Kustomize natively, so you rarely have to flatten anything by hand.\nHelm Charts apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: prometheus namespace: argocd spec: source: repoURL: https://prometheus-community.github.io/helm-charts chart: prometheus targetRevision: 25.0.0 helm: values: | server: persistentVolume: enabled: true size: 10Gi destination: server: https://kubernetes.default.svc namespace: monitoring Kustomize Overlays apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: my-app-production spec: source: repoURL: https://github.com/yourorg/yourrepo.git path: apps/my-app/overlays/production targetRevision: main destination: server: https://kubernetes.default.svc namespace: production Point it at a directory with a kustomization.yaml and it figures out the rest.\nMistakes I See People Make Forgetting to Enable Prune Without prune: true, deleting a manifest from Git leaves the live resource hanging around. You delete the file, assume it is gone, and weeks later trip over the orphan.\nPutting Secrets in Git Never commit plain secrets. Reach for one of these instead:\nSealed Secrets External Secrets Operator backed by Vault SOPS encrypted secrets The “Just This Once” Manual Change The second you fix something by hand, you have created drift. Either commit the fix to Git or let selfHeal undo it. There is no quiet middle ground that stays clean.\nOne Giant Application for Everything A single Application managing your whole platform turns into a mess fast. Split it by service or team so a sync failure in one place does not block everything else.\nWhere to Go Next You now have a working GitOps loop. The interesting features build on top of it:\nApp-of-Apps pattern: manage applications with applications Drift detection: monitor for unauthorized changes Disaster recovery: rebuild whole clusters from Git ApplicationSets: generate applications dynamically Notifications: Slack or email on sync events My Recommendation Start with one application and get the workflow into your fingers. Enable automated sync plus selfHeal so you feel what Git-as-truth actually does. Use a dedicated gitops repo, separate from your application source code. Set up webhooks for sync that triggers on push instead of polling. Notice when you want manual control and ask what the system is missing instead of reaching for kubectl. Go back to that first feeling I described, asking a cluster what it is and not trusting the answer. Once the answer lives in Git and the cluster keeps itself honest, that uncertainty is gone, and you stop wanting to go back to it.","permalink":"https://tommeurs.nl/en/posts/argocd-beginners/","title":"ArgoCD for Beginners: Your First GitOps Deployment"},{"content":"What happens when your Kubernetes cluster can’t reach the internet? I don’t mean a slow connection. I mean no connection at all. Ships at sea. Remote mining sites. Factory floors with air-gapped networks. Military deployments.\nFor a lot of people that sounds exotic, like a problem someone else has. I treat it as a baseline design requirement, and I’ll explain why it makes my homelab better even though I almost never actually pull the cable.\nWhy This Matters: Beyond the Technical Running Kubernetes offline forces you to answer a question most cloud-native guides skip right past: what are you actually depending on?\nA standard Kubernetes setup has invisible dependencies everywhere:\nContainer registries (Docker Hub, gcr.io, quay.io) Helm chart repositories Certificate authorities NTP servers DNS resolvers Package repositories Telemetry endpoints None of these are optional extras. They’re load-bearing assumptions baked into how most clusters operate, and you usually only notice them the day one goes away. Your cluster “works” until Docker Hub rate-limits you or your DNS forwarder falls over, and then it very much doesn’t.\nThis is a sovereignty problem at heart. If you can’t run your infrastructure without phoning home to systems you don’t control, you don’t really own it. You’re renting capability and hoping the lease holds. Black boxes feel like splinters, and a dependency you can’t see is the worst kind of black box.\nIsland Mode: The Architecture I design all my infrastructure to be “island mode capable”, able to function completely cut off from any external network. The honest reason isn’t that I expect to end up on a submarine. It’s that the constraint forces better architecture. The moment you decide the cluster has to survive with the cable unplugged, every sloppy external dependency turns into a thing you have to deal with on purpose.\nNo connection required\nIsland Mode Cluster\nAll dependencies resolved locally\nWorkloads\nApps run without any external calls\nInternal Services\nRegistry\n(Harbor)\nNTP\n(chrony)\nDNS\n(CoreDNS)\nCA\n(cert-mgr)\nHelm\n(ChartMuseum)\nGitOps\n(ArgoCD)\nInternet\n(optional)\nThe goal is simple to state and annoying to achieve: pull the network cable and everything keeps running. Updates stop. Operations continue. Let me build it up component by component, because each one is its own small dependency you have to bring in-house.\nComponent 1: Local Container Registry Start with the most obvious one. Your cluster needs images. Connected, it pulls from Docker Hub or gcr.io on demand and you never think about it. Offline, every one of those requests fails, and a node that can’t pull an image is a node that can’t schedule a thing.\nFix: mirror everything locally.\nHarbor is what I run for self-hosted registries. It gives you:\nPull-through caching (automatic mirroring when connected) Replication from upstream registries Vulnerability scanning Access control # Harbor replication policy - sync when connected apiVersion: goharbor.io/v1beta1 kind: HarborReplicationPolicy metadata: name: docker-hub-mirror spec: srcRegistry: url: https://registry-1.docker.io destRegistry: url: https://harbor.internal trigger: type: scheduled scheduledTrigger: cron: \"0 2 * * *\" # Sync nightly when connected filters: - type: name value: \"library/**\" # Official images only Connected, Harbor syncs images on a schedule. Disconnected, it serves whatever it already cached. That’s the whole pattern repeated for every component below: be a cache when you can, be the source of truth when you have to.\nThe part people forget: pre-pull your dependencies.\nA pull-through cache only helps if the image went through it at least once. Before going offline, make sure every image your workloads need is actually sitting in the local registry:\n# List all images currently running kubectl get pods -A -o jsonpath=\"{.items[*].spec.containers[*].image}\" | \\ tr ' ' '\\n' | sort | uniq # Ensure they're all mirrored to local registry Component 2: Local DNS Now layer on the next dependency. Kubernetes leans on DNS for service discovery, and CoreDNS handles the cluster-internal names fine. The trouble is everything outside the cluster.\nProblem: any pod trying to resolve api.github.com or registry-1.docker.io falls flat without an upstream resolver, and some of your own tooling will quietly assume those names work.\nFix: teach CoreDNS to answer the external names you care about, pointing them at internal addresses:\napiVersion: v1 kind: ConfigMap metadata: name: coredns namespace: kube-system data: Corefile: | .:53 { errors health # Cluster-internal names kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa } # Known external services - resolve to internal hosts /etc/coredns/custom.hosts { fallthrough } # Forward unknown to local resolver (or fail gracefully) forward . 10.0.0.1 { policy sequential health_check 5s } cache 30 loop reload loadbalance } custom.hosts: | 10.0.10.5 harbor.internal registry.internal 10.0.10.6 git.internal gitlab.internal 10.0.10.7 ntp.internal time.internal Component 3: Local Time Synchronization This one bites people who never saw it coming. Time sync matters far more than it looks. TLS certificates validate against the clock, so a drifted node starts rejecting perfectly good certs. Logs need timestamps you can trust when you’re correlating an incident. Distributed systems need the nodes to agree on what “now” is.\nCut off from public NTP, you need a time source of your own:\n# Chrony as local NTP server apiVersion: apps/v1 kind: DaemonSet metadata: name: chrony namespace: kube-system spec: selector: matchLabels: app: chrony template: spec: hostNetwork: true containers: - name: chrony image: harbor.internal/infra/chrony:4.3 volumeMounts: - name: chrony-conf mountPath: /etc/chrony volumes: - name: chrony-conf configMap: name: chrony-config --- apiVersion: v1 kind: ConfigMap metadata: name: chrony-config data: chrony.conf: | # When connected, use public NTP server time.cloudflare.com iburst server pool.ntp.org iburst # When disconnected, be the time source local stratum 10 allow 10.0.0.0/8 # Drift file for accuracy when disconnected driftfile /var/lib/chrony/drift The line that does the work is local stratum 10. It tells chrony to act as a time source even when there’s no upstream to follow. The nodes sync to each other and stay consistent relative to one another, which is what TLS and your logs actually care about.\nComponent 4: Local Certificate Authority TLS everywhere means certificates everywhere, and certificates have to come from somewhere. Connected, cert-manager chats with Let’s Encrypt and you forget the problem exists. Offline, Let’s Encrypt can’t reach you for the ACME challenge and you’re stuck.\nFix: run your own CA.\n# Self-signed root CA apiVersion: cert-manager.io/v1 kind: ClusterIssuer metadata: name: internal-ca spec: ca: secretName: internal-ca-root --- # Create the root certificate apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: internal-ca-root namespace: cert-manager spec: isCA: true commonName: Internal Root CA secretName: internal-ca-root duration: 87600h # 10 years privateKey: algorithm: ECDSA size: 256 issuerRef: name: selfsigned kind: ClusterIssuer Push the root CA out to every node and client and trust it. From then on cert-manager mints certificates from your own authority with zero external calls. You trade the convenience of public trust for full control over your trust chain, which is the right trade when the public CA can’t reach you anyway.\nComponent 5: GitOps Without Internet If you do GitOps, ArgoCD is probably syncing from GitHub or GitLab.com, and offline that source vanishes. You need the Git server inside the island too.\nOption 1: Embedded GitLab\nRun GitLab in-cluster. It’s heavy on resources, but you get everything in one box.\nOption 2: Gitea\nLightweight Git server, and a much better fit for edge hardware:\napiVersion: apps/v1 kind: Deployment metadata: name: gitea spec: template: spec: containers: - name: gitea image: harbor.internal/gitea/gitea:1.21 env: - name: GITEA__server__OFFLINE_MODE value: \"true\" Point ArgoCD at git.internal instead of github.com. When you have a connection you push your changes upstream as usual, but ArgoCD reconciles against the local copy either way. The cluster’s desired state lives on the island, not on someone else’s server.\nComponent 6: Helm Charts Offline Last dependency in the chain. Helm charts normally come from remote repositories, so offline you need chart storage that lives with you.\nChartMuseum speaks the Helm repository API:\n# Push charts to local museum when connected helm push mychart-1.0.0.tgz http://chartmuseum.internal # ArgoCD uses local charts apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: my-app spec: source: repoURL: http://chartmuseum.internal chart: mychart targetRevision: 1.0.0 The Pre-Deployment Checklist Six components in, here’s where it gets real. The failure mode that hurts most is the one you discover the day after you disconnect, when nobody can fix it without a boat or a long drive. So before you cut the cable, verify the island actually holds together. This is the script I run, and I’d rather it scream at me now than have a pod fail to pull an image at sea:\n#!/bin/bash # offline-readiness-check.sh echo \"=== Checking Offline Readiness ===\" # 1. All images available locally echo \"Checking images...\" for image in $(kubectl get pods -A -o jsonpath=\"{..image}\" | tr ' ' '\\n' | sort -u); do if ! curl -s \"https://harbor.internal/v2/${image}/manifests/latest\" \u003e /dev/null; then echo \"MISSING: $image\" fi done # 2. DNS resolves internally echo \"Checking DNS...\" for name in harbor.internal git.internal ntp.internal; do if ! nslookup $name \u003e /dev/null 2\u003e\u00261; then echo \"DNS FAIL: $name\" fi done # 3. Time is synchronized echo \"Checking NTP...\" chronyc tracking | grep -q \"Leap status.*Normal\" || echo \"TIME SYNC ISSUE\" # 4. Certificates are valid echo \"Checking certificates...\" kubectl get certificates -A -o jsonpath='{range .items[*]}{.metadata.name}: {.status.conditions[0].type}{\"\\n\"}{end}' # 5. GitOps is synced echo \"Checking ArgoCD...\" argocd app list | grep -v \"Synced.*Healthy\" \u0026\u0026 echo \"APPS NOT SYNCED\" echo \"=== Check Complete ===\" Real-World Scenarios The pieces above are abstract until you put them in a place where the cable genuinely doesn’t reach. Three settings make the constraints concrete.\nScenario 1: Ship at Sea A container ship running Kubernetes for cargo management. Satellite connectivity exists, but it’s expensive and it drops out when the weather turns.\nArchitecture:\nK3s for a low resource footprint Harbor with a full mirror of every required image Local GitOps with Gitea Sync updates while in port The catch: updates land in batches during port calls, and that might be once every few weeks. The system has to run rock-solid in between, because nobody is pushing a hotfix mid-Atlantic.\nScenario 2: Factory Floor A manufacturing plant running Kubernetes for automation, where security policy flatly forbids internet connectivity.\nArchitecture:\nFull Kubernetes with an air-gapped installation USB-based updates, signed and verified A separate management network for the rare updates that do happen The catch: updates need someone physically present, so every change has to be right the first time. There’s no rolling back a bad deploy from your laptop at home.\nScenario 3: My Homelab I run my own infrastructure island-mode capable, and I’m not waiting for a disaster to justify it. The discipline is the point. When the design assumes no internet, the architecture comes out cleaner on its own.\nWhat I actually get out of it:\nNo external dependencies, so no external failure modes to inherit Faster image pulls from the local registry Everything keeps working through ISP outages, which around here happen more than I’d like Infrastructure I genuinely own rather than rent The Trade-offs I’m not going to pretend this is free. It isn’t.\nAspect Connected Offline Updates Continuous Batched Vulnerability info Real-time Delayed External integrations Easy Impossible Operational burden Lower Higher Sovereignty Partial Complete There’s no universally correct column here. The right answer depends on what your setup actually demands, and for a homelab I happily pay the higher operational burden to get the complete sovereignty.\nK3s: Built for the Edge K3s earns a section of its own, because it was built for exactly this kind of place:\nSingle binary, ~100MB Embedded SQLite (no external etcd needed) Works on ARM (Raspberry Pi, edge devices) Minimal dependencies # Air-gapped K3s installation curl -sfL https://get.k3s.io \u003e install.sh # On air-gapped machine INSTALL_K3S_SKIP_DOWNLOAD=true \\ INSTALL_K3S_EXEC=\"--private-registry /etc/rancher/k3s/registries.yaml\" \\ ./install.sh Point K3s at a local registry and it runs completely offline. For ships, factories, and Raspberry Pi clusters in a cupboard, that combination is hard to beat.\nMy Recommendation If you take one thing from all of this, take the first point. The rest follows from it.\nDesign for offline, run connected. You probably won’t disconnect often, but building as if you might produces a better cluster every single day.\nMirror everything. Container images, Helm charts, Git repos. If it lives outside your network, get a copy inside it.\nTest disconnection. Actually unplug the cable and watch what breaks. The first time you do this you’ll find a dependency you swore wasn’t there.\nAutomate the sync. While you’re connected, refresh the mirrors on a schedule. A manual process is a process that won’t run the week you needed it most.\nDocument dependencies. Write down exactly what your cluster needs from the outside world, so future-you isn’t reverse-engineering it during an outage.\nIsland mode was never about paranoia. It’s about knowing your dependencies well enough to live without them, and a cluster you can run offline is one you genuinely understand.","permalink":"https://tommeurs.nl/en/posts/kubernetes-offline-edge/","title":"Running Kubernetes Offline: Edge Computing Without Internet"},{"content":"Everyone repeats the line that Kubernetes is self-healing. Pods die, they come back. Nodes drop, workloads reschedule. The system reconciles itself toward the state you declared, and most days you never have to think about it.\nThen one day the thing doing the healing is the thing that broke. The API server is down. etcd won’t respond. The scheduler is wedged. Now what?\nThis is the question I actually care about, because “self-healing” is only useful if I understand its edges. I want to know what degrades gracefully and what takes the whole cluster with it. So I’ve run my clusters through a lot of failures on purpose: planned, unplanned, and a few “hold my beer” experiments on hardware I didn’t mind losing. Here is what actually happens when each piece breaks, and why most of it matters less than people fear.\nThe components that can fail Before the scenarios, here is the map. You can’t reason about blast radius if you don’t know which part owns which job.\nControl Plane:\nkube-apiserver: The API that everything talks to etcd: The database storing all cluster state kube-scheduler: Decides where pods run kube-controller-manager: Runs controllers (ReplicaSet, Deployment, etc.) cloud-controller-manager: Cloud provider integrations (if applicable) Node Components:\nkubelet: Manages pods on each node kube-proxy: Handles network rules for Services Container runtime: Actually runs containers The thing worth internalising up front: most of these are control-plane brains, and your running workloads do not consult the brain to keep serving traffic. That single fact explains almost every scenario below.\nScenario 1: API server down The kube-apiserver is the single front door for every Kubernetes API request. Kill it and you’d assume the cluster falls over. It doesn’t.\nImmediate impact:\nkubectl commands fail No new deployments or updates possible No new pod scheduling Existing pods keep running What keeps working:\nRunning pods continue to run Containers stay alive Network connectivity between pods Services continue to route traffic What breaks:\nNo new pods can be created Failed pods won’t be replaced Horizontal Pod Autoscaler stops working No changes to any resources API Server Down\nno updates\nWorker Nodes\nNode 1\nPods ✓\nNode 2\nPods ✓\nNode 3\nPods ✓\nControl Plane\netcd ✓\nscheduler\nidle\ncontroller-mgr\nidle\n✗ API Server\nPods keep running - they don't need the API\nThis is the whole design philosophy in one picture. Kubernetes expects the API server to be unavailable sometimes, so it degrades gracefully: your workloads keep serving, you just lose the ability to change anything. The control plane is the management layer, not the runtime. That separation is what makes the system survivable.\nFor HA setups you want multiple API servers behind a load balancer. See Kubernetes High Availability: stacked vs external etcd for architecture options.\nScenario 2: etcd down etcd is the brain of Kubernetes. Every bit of cluster state lives here. This is the failure that actually scares me, and the one I back up religiously.\nImmediate impact:\nAPI server can’t read or write state Effectively same as API server down Existing pods keep running What keeps working:\nRunning pods continue to run Containers stay alive Network connectivity Services work What breaks:\nSame as API server down, plus: Risk of state inconsistency on recovery Split-brain scenarios in partial failures etcd Down\nControl Plane\netcd ✗\nAPI ✗\nother components\nWithout etcd, API server cannot function\nNodes continue running - they cache their assignments\nNotice the runtime still holds. Nodes cache their assignments, so the pods that were running stay running even though the brain is offline. The danger isn’t the outage, it’s recovery: bring etcd back inconsistent or from a stale snapshot and you can corrupt your view of the world. That’s why backups here aren’t optional. See etcd Deep Dive for how etcd works and how to back it up properly.\nScenario 3: scheduler down The kube-scheduler decides where pods run. When it’s down:\nImmediate impact:\nNew pods stay in Pending state No scheduling decisions made What keeps working:\nExisting pods keep running API server functions normally You can create resources (they just won’t be scheduled) What breaks:\nNew pods can’t be scheduled Rescheduling after node failure doesn’t happen HPA creates pods that stay Pending Scheduler Down\nPending Queue\nPod: web\nPod: api\nPod: job\n... waiting\nkubectl apply deployment\nAPI accepts\nPod: Pending...\nNodes have capacity but nobody assigns pods to them\nThe tell here is a pile of Pending pods on a cluster with plenty of free capacity. The API accepts your manifests, etcd stores them, the nodes have room, but nothing places the pods on a node. In HA setups you run multiple schedulers with leader election: one active, the rest waiting to take over.\nScenario 4: controller manager down The controller manager runs the controllers that earn Kubernetes the “self-healing” label.\nImmediate impact:\nReplicaSet controller stops Deployment controller stops Node controller stops All reconciliation loops stop What keeps working:\nExisting pods keep running Scheduling still works API server still works What breaks:\nFailed pods don’t get replaced Deployments don’t roll out Node failures aren’t handled Orphaned resources aren’t cleaned up Controller Manager Down\nReplicaSet: 3 desired, 2 running → no action taken\nDeployment: rollout in progress → stuck\nNode marked NotReady → pods not evicted\nThe 'self-healing' part of Kubernetes stops\nThis is the scenario where the word “magic” finally falls away. The self-healing is just software running reconciliation loops, comparing desired state to actual state and acting on the gap. Stop the loops and the gap stops closing. A ReplicaSet that wants three replicas will happily sit at two forever, because nobody is watching. I find this reassuring rather than alarming: it means I can read the source, predict the behaviour, and trust it for the right reasons instead of treating it as a black box.\nScenario 5: kubelet down on a node The kubelet is the Kubernetes agent on each node. When it fails:\nImmediate impact (on that node):\nNode marked as NotReady after timeout (default ~40 seconds) Pods on that node get evicted (after another timeout) No new pods scheduled to that node What keeps working:\nContainers keep running (they don’t need kubelet) Network might still work (depends on CNI ) Other nodes unaffected What breaks:\nNo pod lifecycle management on that node Health checks stop Resource updates stop Eventually pods are rescheduled elsewhere Kubelet Down on Node 2\nNode 3 ✓ Ready\nNode 2 ✗ NotReady\nrescheduled\npod ✓\nrescheduled\npod ?\norphaned\nNode 1 ✓ Ready\npod ✓\nAfter pod-eviction-timeout, pods get rescheduled\nThe part that surprises people: the containers keep running even with the kubelet dead. The container runtime is what keeps them alive, the kubelet just manages their lifecycle. Kill the manager and the workers keep working until something decides to evict and reschedule them somewhere with a healthy kubelet.\nScenario 6: container runtime down If the container runtime (containerd , CRI-O ) fails:\nImmediate impact:\nRunning containers might die New containers can’t start Health checks fail What happens next:\nkubelet detects failures Pods marked as Failed Pods get rescheduled to other nodes This is the one node-level failure that does take your workloads with it, because the runtime is the thing actually running them. It triggers the same eviction-and-reschedule dance as a dead kubelet, just from the other direction.\nScenario 7: network partition Network partitions are the nastiest failures, because nothing has actually crashed. A node loses its path to the control plane while happily continuing to run containers.\nWhat happens:\nNode marked NotReady (can’t reach API server) Pods eventually evicted But they might still be running on the partitioned node Potential for “split brain”: same pod running in two places Network Partition\n✗\nPartitioned Node\nI'm fine, running\nthese pods...\nControl Plane\nNode 2 is NotReady\nEvicting pods...\nPod 'web-abc123' now runs on Node 1 AND Node 2\nBoth think they're the real one\nThe control plane gives up on the unreachable node and reschedules its pods elsewhere. The partitioned node never got the memo and keeps running the originals. Now the same pod exists twice, each convinced it’s the only one. For stateless web pods that’s a shrug. For a stateful database it’s a data-corruption incident waiting to happen, which is why stateful workloads need fencing and quorum, not just “reschedule and hope.”\nFailure timeouts to know These timeouts control how fast Kubernetes reacts to failures:\nTimeout Default What it does node-monitor-grace-period 40s How long before marking node NotReady pod-eviction-timeout 5m How long before evicting pods from NotReady node node-monitor-period 5s How often node status is checked You can tune these down for faster failover, but there’s a real trade-off: tighten them too far and a two-second network blip evicts healthy pods. Fast detection and false positives are the two ends of the same dial.\nThe blast radius principle Every failure has a blast radius. Knowing the radius tells you where to spend your redundancy budget.\nComponent Blast Radius Container Single container Pod All containers in pod Kubelet All pods on node Node All pods on node Scheduler New pod scheduling cluster-wide Controller Manager Self-healing cluster-wide API Server All management operations etcd Everything Read that table bottom to top and your HA priorities write themselves. etcd and the API server have the widest radius, so they get the most redundancy. A single container failing is noise the system already handles.\nWhat this means for you Running workloads are resilient: existing pods survive most control plane failures Management operations aren’t: you need control plane HA for continuous deployment etcd is the critical path: protect it, back it up, monitor it Failures cascade: API server down looks like everything is down, even when workloads are fine Timeouts matter: know your failure detection times before you tune them The reason I trust Kubernetes isn’t that it never fails. It’s that it fails in ways I can predict and reason about. The control plane and the runtime are separated on purpose, so losing the brain doesn’t kill the body. The real question for your cluster is never whether a component will fail. It’s whether you’ve designed for the radius when it does.\nTesting failures Don’t wait for production to teach you how your cluster behaves. Break it yourself, on hardware you don’t mind losing:\n# Simulate API server failure (on a test cluster!) kubectl exec -it -n kube-system kube-apiserver-xxx -- kill 1 # Simulate scheduler failure kubectl scale deployment kube-scheduler -n kube-system --replicas=0 # Simulate kubelet failure on a node ssh node-1 'sudo systemctl stop kubelet' For controlled, repeatable experiments, reach for a chaos engineering tool like Litmus Chaos instead of kill 1 by hand.\nUnderstanding failure modes isn’t pessimism, it’s how you earn the right to call a system resilient. Every system fails. The only choice you get is whether you designed for it or got surprised by it.","permalink":"https://tommeurs.nl/en/posts/kubernetes-graceful-degradation/","title":"Graceful Degradation in Kubernetes: What Happens When Components Fail"},{"content":"Picture this: an attacker pops a single pod in your cluster, maybe through a vulnerable image or a leaked token. From that one foothold, they can reach every database, every internal API, every secret-fetching sidecar you run. Nothing stops them, because by default nothing tries to.\nNetwork Policies are the thing that stops them. They turn “one compromised pod” into “one compromised pod, and that’s it.” Everyone knows they should use them. Almost nobody actually does, because the YAML looks scary and the behaviour is weird until the mental model clicks.\nI’ve burned hours debugging policies that “should work” and didn’t. So this post builds the model up one layer at a time, with a diagram for each step. Start at the top, stop whenever you have enough.\nThe Default: Everything Talks to Everything Here’s the part that surprises people. Out of the box, Kubernetes lets every pod talk to every other pod, across every namespace. No restrictions at all.\nCluster\nPod\nPod\nPod\nPod\nEveryone can talk\nGreat for getting something running on day one. Terrible the moment you care about blast radius. Compromise one pod and the attacker reaches the whole cluster. That’s exactly the flat trust model zero trust tells us to avoid: assume breach, then make sure the breach can’t spread.\nNetwork Policies: The Firewall for Pods A NetworkPolicy describes how pods are allowed to communicate. It’s a firewall rule that targets groups of pods by their labels instead of by IP, which matters because pod IPs are ephemeral and you’d go insane managing them by hand.\nFour concepts carry the whole feature:\nIngress: traffic coming in to a pod Egress: traffic going out from a pod Selectors: how you pick which pods a rule applies to (by labels) Policy Types: whether you’re controlling ingress, egress, or both That’s the vocabulary. Everything below is just combinations of these four.\nThe Most Important Rule: Default Deny The first policy you apply to any namespace is default deny. It blocks all traffic, and then you open up exactly what you need. This is the simplest possible policy and also the most important one.\napiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny-all namespace: production spec: podSelector: {} # Empty = all pods policyTypes: - Ingress - Egress production namespace\nPod\nPod\nPod\nPod\nAll traffic blocked by default\nNow nothing communicates. That feels broken, and that’s the point. You’re starting from zero and building up, which is the only way to know what your apps actually need. Anything you forget to allow will fail loudly during testing instead of quietly leaving a hole open in production.\nAllowing Ingress: Who Can Talk TO This Pod? Say you have a frontend that needs to receive traffic. You select that pod by its label and spell out who’s allowed to reach it.\napiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-frontend-ingress namespace: production spec: podSelector: matchLabels: app: frontend policyTypes: - Ingress ingress: - from: - namespaceSelector: matchLabels: name: ingress-nginx - podSelector: matchLabels: app: ingress-controller ports: - protocol: TCP port: 8080 production ns\ningress-nginx ns\n✓\n✗\ningress controller\nfrontend\napp:frontend\nother pods\nThe frontend now accepts traffic from the ingress controller on port 8080, and nothing else gets in. Other pods in the namespace can knock all they want; the door stays shut.\nAllowing Egress: What Can This Pod Talk TO? Ingress controls who reaches a pod. Egress controls where a pod is allowed to reach. The frontend needs to call the backend API, so we open that path and nothing more.\napiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: frontend-egress namespace: production spec: podSelector: matchLabels: app: frontend policyTypes: - Egress egress: - to: - podSelector: matchLabels: app: backend ports: - protocol: TCP port: 3000 - to: # Allow DNS - namespaceSelector: {} podSelector: matchLabels: k8s-app: kube-dns ports: - protocol: UDP port: 53 production namespace\n✓\n✗\n✓ DNS only\nfrontend\nbackend\ndatabase\nkube-dns\nThe frontend can reach the backend on port 3000 and resolve DNS. It cannot touch the database directly, which is the whole idea. If someone compromises the frontend, they’re stuck talking to the backend and can’t jump straight to your data.\nThe Complete Three-Tier Example Now the layers come together. A realistic web app: ingress to frontend, frontend to backend, backend to database. Each tier gets a policy, each policy allows only its immediate neighbour.\n# Frontend policy apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: frontend-policy spec: podSelector: matchLabels: tier: frontend policyTypes: - Ingress - Egress ingress: - from: - namespaceSelector: matchLabels: name: ingress ports: - port: 80 egress: - to: - podSelector: matchLabels: tier: backend ports: - port: 8080 - to: # DNS - namespaceSelector: {} podSelector: matchLabels: k8s-app: kube-dns ports: - port: 53 protocol: UDP --- # Backend policy apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: backend-policy spec: podSelector: matchLabels: tier: backend policyTypes: - Ingress - Egress ingress: - from: - podSelector: matchLabels: tier: frontend ports: - port: 8080 egress: - to: - podSelector: matchLabels: tier: database ports: - port: 5432 - to: # DNS - namespaceSelector: {} podSelector: matchLabels: k8s-app: kube-dns ports: - port: 53 protocol: UDP --- # Database policy apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: database-policy spec: podSelector: matchLabels: tier: database policyTypes: - Ingress - Egress ingress: - from: - podSelector: matchLabels: tier: backend ports: - port: 5432 egress: [] # Database doesn't need to initiate connections DNS only\nDNS only\nInternet\nIngress\nFrontend\nBackend\nDatabase\nkube-dns\nEach tier talks only to the tier next to it. The frontend can’t reach the database. The database can’t reach the internet at all, which means even a full database compromise can’t exfiltrate data over an outbound connection. That’s least privilege expressed as a graph, and it’s the same single-step model from earlier, just repeated three times.\nCommon Gotchas This is where my debugging hours went. None of these are obvious until they bite you.\nGotcha 1: Don’t Forget DNS Your pods resolve service names through kube-dns. Apply a default-deny egress policy without allowing DNS and suddenly nothing resolves, which looks like a hundred different bugs before you realise it’s just blocked port 53.\negress: - to: - namespaceSelector: {} podSelector: matchLabels: k8s-app: kube-dns ports: - port: 53 protocol: UDP Without this, backend.production.svc.cluster.local never resolves and your app sits there timing out.\nGotcha 2: Policies Are Additive Multiple policies on the same pod combine with OR logic. Policy A allows traffic from X, policy B allows traffic from Y, so the pod accepts both X and Y.\nThere’s no “deny” rule that overrides an “allow.” The only way to deny something is to never allow it in the first place. If you’re used to firewall systems with explicit deny rules and ordering, unlearn that here.\nGotcha 3: Empty Selector Means “All” podSelector: {} # All pods in this namespace namespaceSelector: {} # All namespaces This one catches almost everyone. An empty selector matches everything, not nothing. It reads like “no pods” and behaves like “every pod,” which is exactly why default-deny uses podSelector: {} to grab the whole namespace.\nGotcha 4: Policies Only Affect Selected Pods A NetworkPolicy only touches pods that match its podSelector. Any pod that no policy selects has zero restrictions and talks freely.\nSo you can write beautiful policies for half your namespace and leave the other half wide open without noticing. Default-deny fixes this by guaranteeing every pod is selected by at least one policy.\nTesting Network Policies Don’t trust a policy you haven’t tested. Spin up a throwaway pod and try to reach things that should and shouldn’t work.\n# Create a test pod kubectl run test-pod --image=busybox --rm -it -- sh # Try to reach another service wget -qO- --timeout=2 http://backend.production:8080/health # Try to reach the database directly (should fail) nc -zv database.production 5432 When you need more than busybox, netshoot ships with every networking tool you’d want:\nkubectl run netshoot --image=nicolaka/netshoot -it --rm -- bash The “should fail” tests matter more than the “should work” ones. It’s easy to confirm allowed traffic flows. The whole point is confirming the blocked paths are actually blocked.\nCNI Requirements One catch before you rely on any of this: Network Policies are enforced by your CNI , not by Kubernetes itself. If your CNI doesn’t implement the NetworkPolicy API, the cluster will happily accept your policies and ignore them entirely, which is a nasty way to discover you have no isolation.\nCalico : Full support, plus its own extensions Cilium : Full support, plus L7 policies Weave: Full support Flannel : No support (run Calico on top) Check what you’re running before you assume a policy does anything. On my own clusters I run Cilium, partly for the L7 features below.\nBeyond Basic Policies Once the basics click, the CNI-specific extensions open up:\nCilium Network Policies: L7 (HTTP) filtering, DNS-based policies Calico Network Policies: host protection, global policies Service Mesh: Istio or Linkerd for mutual TLS and L7 policies The vanilla Kubernetes NetworkPolicy API stays deliberately simple, which is a feature. It covers L3/L4, and when you genuinely need L7 you reach for a CNI that gives it to you. Don’t reach for it before you need it.\nMy Recommendation Start small and build up, the same way this post did:\nApply default-deny to every namespace Add policies that allow only the communication you actually need Test thoroughly, especially the paths that should fail, before production Use Cilium’s policy editor to visualise what you’re writing Network Policies are one of the highest-value security improvements available in Kubernetes. No extra infrastructure to run, declarative so they live in Git like everything else, and they cut your blast radius down to almost nothing.\nThe mental model takes a few sessions of trial and error. Once it lands, you’ll feel a little uneasy running any cluster without it.\nNetwork Policies put least privilege on the network. Every connection that doesn’t need to exist shouldn’t exist.","permalink":"https://tommeurs.nl/en/posts/kubernetes-network-policies-visual/","title":"Kubernetes Network Policies: A Visual Guide to Pod Security"},{"content":"When something goes wrong in Kubernetes, the trail usually leads back to etcd. API server timing out? Check etcd. Pods stuck in pending? Might be etcd. Cluster feels sluggish? Probably etcd.\nFor a long time I treated etcd the way most operators do: as a black box that hums along next to the control plane. “The database.” You back it up and otherwise leave it alone. But black boxes feel like splinters to me, and the first time an etcd cluster fell over at 2am I realised I had no idea what I was actually looking at. So I learned. And it turns out the whole thing is built on a handful of ideas that, once they click, make most etcd problems diagnosable instead of terrifying.\nLet me build it up from the bottom.\nWhat is etcd? etcd is a distributed key-value store. A highly reliable dictionary that multiple servers agree on. Kubernetes uses it to hold all cluster state: every pod, deployment, secret, configmap, and custom resource lives in there.\nThe name comes from “/etc distributed”. The Unix /etc directory holds system configuration, and etcd is that idea spread across nodes.\nFour properties define it:\nStrongly consistent: every node sees the same data at the same time Highly available: it survives node failures, as long as you keep quorum Watch-capable: clients can subscribe to changes Ordered: operations happen in a strict sequence Those four words explain almost everything etcd does. The rest of this post is really just unpacking how it pulls them off.\nThe Raft Consensus Algorithm etcd uses Raft to stay consistent across nodes. If you understand Raft , you understand most of etcd’s behaviour, including the parts that look like bugs but aren’t.\nLeader election In any etcd cluster, one node is the leader. The leader handles all writes. Followers replicate the leader’s log.\nreplicates\nreplicates\nheartbeats\nheartbeats\nLeader\n(etcd1)\nFollower\n(etcd2)\nFollower\n(etcd3)\nThe leader sends heartbeats to followers. If a follower stops hearing from the leader for an election timeout (typically 1-2 seconds), it becomes a candidate and starts an election. This is why a flaky network shows up as constant leader churn: the followers keep thinking the leader died.\n# See who's the leader etcdctl endpoint status --cluster -w table Log replication When a write comes in, this is the path it takes:\nClient sends the write to the leader Leader appends it to its log Leader replicates to followers Once a majority confirms, the leader commits Leader responds to the client Step 4 is the whole reason etcd needs a majority (quorum) to write. With 3 nodes you need 2. With 5 you need 3. Lose quorum and etcd goes read-only rather than risk diverging. That refusal to guess is exactly what you want from your source of truth.\nTerm numbers Each election bumps the term number. Treat it as a logical clock. If a node sees a higher term, it knows a newer leader was elected and falls in line. A term number that keeps climbing fast is a tell-tale sign of an unstable cluster.\n# Check cluster health and terms etcdctl endpoint health --cluster -w table Data Model So far we have a consistent, replicated log. Now look at what actually gets stored in it.\netcd keeps data as key-value pairs in a flat namespace. Kubernetes layers a hierarchical convention on top:\n/registry/pods/default/nginx-abc123 /registry/pods/kube-system/coredns-xyz789 /registry/deployments/default/my-app /registry/secrets/default/my-secret Everything sits under /registry/. The Kubernetes API server is mostly a translation layer between the Kubernetes API and etcd’s key-value model. When you kubectl get pods, you are reading a slice of that tree.\nRevisions and MVCC etcd uses Multi-Version Concurrency Control (MVCC). Every modification creates a new revision, and old revisions stick around for a while. That single design choice buys you three things:\nWatch from revision: “tell me everything that changed since revision 12345” Historical reads: “what was the value at revision 12345?” Optimistic locking: “update only if still at revision 12345” # Get current revision etcdctl get / --prefix --limit=1 -w json | jq '.header.revision' # Read a key with its revision etcdctl get /registry/pods/default/nginx -w json | jq '.kvs[0].mod_revision' That first capability, watch-from-revision, is the one Kubernetes leans on hardest. Hold that thought.\nWhy etcd Performance Matters Kubernetes is chatty with etcd. Every API call, every controller reconciliation, every watch notification touches it. A slow etcd is a slow cluster, full stop. This is where the abstract Raft theory turns into very concrete operational pain.\nCritical metrics fsync latency: time to persist to disk. Aim for under 10ms. Past 20ms you start seeing problems, because remember step 4 of log replication waits on the disk.\n# Check disk latency etcdctl check perf --load=\"s\" Raft proposal latency: time to commit a write through Raft. Should be under 50ms.\nWatch count: number of active watches. Kubernetes runs thousands of them.\nCommon performance problems Slow disks: etcd writes to disk synchronously. Use SSDs, not spinning rust. NVMe if you can get it. This is the single most common cause of a sad cluster.\nNetwork latency: Raft heartbeats and log replication want low latency between nodes. Keep etcd nodes on the same network segment.\nToo many objects: a cluster with 100k pods leans on etcd far harder than one with 1k. Watch cardinality explodes.\nLarge objects: secrets stuffed with huge certificates, ConfigMaps holding megabytes of data, all of it flows through etcd.\nBacking Up etcd Here is the blunt version: if etcd data is gone, your cluster is gone. Everything else is derived state that can be rebuilt. This is the one thing you cannot afford to lose, so back it up on a schedule.\n# Snapshot backup etcdctl snapshot save /backup/etcd-$(date +%Y%m%d-%H%M%S).db # Verify the backup etcdctl snapshot status /backup/etcd-*.db -w table For automated backups:\n#!/bin/bash # /etc/cron.daily/etcd-backup BACKUP_DIR=/var/backups/etcd RETENTION_DAYS=7 etcdctl snapshot save $BACKUP_DIR/etcd-$(date +%Y%m%d-%H%M%S).db # Cleanup old backups find $BACKUP_DIR -name \"etcd-*.db\" -mtime +$RETENTION_DAYS -delete Restoring from Backup Restoring etcd is serious business. You are resetting cluster state to a point in time, with all the consequences that implies.\n# Stop etcd on all nodes first # Restore on each node (different data-dir) etcdctl snapshot restore /backup/etcd-snapshot.db \\ --name etcd1 \\ --initial-cluster etcd1=https://10.0.0.1:2380,etcd2=https://10.0.0.2:2380,etcd3=https://10.0.0.3:2380 \\ --initial-cluster-token etcd-cluster-restored \\ --initial-advertise-peer-urls https://10.0.0.1:2380 \\ --data-dir /var/lib/etcd-restored # Start etcd with new data-dir After restore, everything that happened after the snapshot is gone. Pods might reference deleted deployments. Services might point at endpoints that no longer exist. The cluster reconciles its way out of it, but expect some chaos on the way. Test this on a throwaway cluster before you ever need it for real.\nCompaction and Defragmentation Remember those old revisions MVCC keeps around? They do not vanish on their own. etcd holds them until compaction runs, and without it the database grows until it falls over.\n# Check current database size etcdctl endpoint status --cluster -w table # Compact to revision etcdctl compact 123456 # Defragment to reclaim space etcdctl defrag --cluster Kubernetes auto-compacts etcd via --etcd-compaction-interval, but defragmentation stays manual. Compaction drops the old revisions; defrag actually hands the freed disk space back to the filesystem.\nWatching etcd This is where that watch-from-revision feature pays off. Kubernetes controllers do their work by watching etcd. Something changes, etcd notifies the watchers immediately, no polling required.\n# Watch all pod changes etcdctl watch /registry/pods --prefix # Watch specific namespace etcdctl watch /registry/pods/default --prefix That mechanism is how the scheduler learns about new pods the instant they appear, how the kubelet learns about pod assignments, how the whole control plane stays in sync without a central clock. The entire reconciliation model of Kubernetes rides on this one etcd primitive.\nDebugging etcd Issues Now that you know how the pieces fit, debugging stops being guesswork. Each symptom maps to a layer you already understand.\nHigh latency # Check if disk is the problem etcdctl check perf --load=\"s\" # Check cluster health etcdctl endpoint health --cluster -w table # Check leader changes (too many = instability) etcdctl endpoint status --cluster -w table Disk space issues # Check database size etcdctl endpoint status --cluster -w table # Force compaction and defrag REV=$(etcdctl endpoint status -w json | jq '.[0].Status.header.revision') etcdctl compact $REV etcdctl defrag --cluster Network issues # Check if nodes can reach each other for ep in https://10.0.0.1:2379 https://10.0.0.2:2379 https://10.0.0.3:2379; do etcdctl --endpoints=$ep endpoint health done etcd in Production What this looks like on my own clusters:\nDedicated SSDs: etcd wants fast, consistent disk I/O. Don’t share spindles with other workloads.\nMonitor relentlessly: etcd_disk_backend_commit_duration_seconds, etcd_network_peer_round_trip_time_seconds, etcd_server_leader_changes_seen_total. Those three tell you about disk, network, and stability respectively.\nRegular backups: daily at minimum, and test the restore now and then so you trust it.\nSeparate network: where possible, put etcd peer traffic on its own network to dodge contention.\nRight-size the cluster: 3 nodes for most workloads, 5 for the critical ones. Beyond 5 you are mostly paying replication cost for very little extra resilience.\nWrapping Up etcd is simple in concept and unforgiving in execution. It is the foundation everything else builds on, so when etcd is healthy Kubernetes is healthy, and when etcd struggles the whole cluster struggles with it.\nThe payoff for learning the internals is that the splinter stops being a splinter. Once you can see Raft consensus, quorum, and disk latency for what they are, etcd is a well-designed distributed system doing exactly what distributed systems do. No magic, just mechanism, and mechanism you can reason about at 2am.\nGive it fast disks, back it up, watch the metrics, and it will quietly keep your cluster running.\netcd is the source of truth. Everything else in Kubernetes is derived state. Protect your source of truth.","permalink":"https://tommeurs.nl/en/posts/etcd-deep-dive/","title":"How etcd Actually Works: The Heart of Your Kubernetes Cluster"},{"content":"The first “production” Kubernetes cluster I ran had a single control plane node. It hummed along happily for weeks, right up until a disk failed and took the whole thing with it. Every pod, every service, gone. That outage taught me what “single point of failure” actually feels like, and it pushed me toward a question that I keep seeing trip people up: when you build a cluster that survives node loss, do you run etcd on your control plane nodes, or on dedicated nodes of its own?\nThat choice has a name. Stacked etcd versus external etcd. Both are valid, both ship in real production clusters, and the right answer depends entirely on your context. I want to walk through them the way I wish someone had walked me through it, with the trade-offs laid bare instead of a vendor diagram telling me which is “best practice.”\nA quick bias check before we start: I self-host everything, my clusters live on refurbished mini PCs in a homelab, and I value understanding what I run over chasing the most scalable architecture on paper. Keep that in mind. Your constraints will tilt the balance differently.\nWhat etcd Actually Does etcd is the part of Kubernetes that remembers things. It’s a distributed key-value store holding all cluster state: pod definitions, secrets, configmaps, service accounts, the lot. Every time you kubectl apply something, the desired state lands in etcd. The control plane components then read from it and work to make reality match.\nSo when etcd is down, your cluster is effectively brain-dead. The API server has nothing to serve, controllers can’t reconcile anything, and your running workloads drift with no one steering. etcd availability and cluster availability are the same number.\netcd stays consistent using the Raft consensus algorithm, which needs a quorum, a strict majority of members, to accept writes. Three members tolerate one failure. Five members tolerate two. Drop below quorum and etcd goes read-only until enough members come back. That quorum math is the thing every HA decision below revolves around, so it’s worth holding onto.\nThe Two Topologies The whole comparison comes down to where etcd lives relative to the rest of the control plane.\nStacked means etcd runs on each control plane node, sitting right next to the API server, controller manager, and scheduler.\nStacked etcd Topology\nControl Plane Node 3\nAPI Server\nController\nScheduler\netcd\nControl Plane Node 2\nAPI Server\nController\nScheduler\netcd\nControl Plane Node 1\nAPI Server\nController\nScheduler\netcd\nExternal means etcd runs on its own dedicated nodes, fully separated from the control plane.\nExternal etcd Topology\netcd Cluster\netcd Node 1\netcd Node 2\netcd Node 3\nControl Plane Nodes\nControl Plane 3\nAPI Server\nController\nScheduler\nControl Plane 2\nAPI Server\nController\nScheduler\nControl Plane 1\nAPI Server\nController\nScheduler\nSame components, different blast radius. That difference is the whole story. Let me compare them on the criteria that actually decide the question: operational complexity, failure isolation, scaling flexibility, and resource behaviour.\nOperational Complexity Stacked wins here without much of a fight. Three nodes give you a fully HA control plane with etcd quorum baked in, and the lifecycle is coupled in a helpful way: add a control plane node and you automatically get an etcd member to go with it. kubeadm , K3s , and most installers default to stacked because it’s the path with the fewest moving parts. You run kubeadm init with --control-plane-endpoint, join two more nodes, and you’re done.\nExternal etcd asks more of you up front. You bootstrap and certificate an etcd cluster yourself before kubeadm will even initialise the control plane, and from then on you’re maintaining two tiers with separate upgrade and backup procedures. More nodes, more network paths to secure, more things to reason about at 2am.\nIf your team hasn’t operated etcd before, that complexity is a real cost, not an abstract one. I’d rather understand three nodes deeply than half-understand six.\nFailure Isolation This is where external earns its keep. With stacked etcd, a dead node costs you two things at once: a control plane component and an etcd member. Lose a node and your quorum margin shrinks at the same moment your API capacity does. With external etcd those failures are independent. A control plane node dying leaves etcd quorum untouched, and an etcd node dying leaves the API servers serving.\nFor most clusters that coupling is fine, because three nodes still tolerate one failure either way. The isolation matters when you’re running tight SLAs and can’t afford a single node loss taking out two systems at once.\nScaling Flexibility Stacked forces your etcd member count and your API server count to move together. That’s usually what you want, but not always. etcd actually performs worse past a certain member count because every write has to reach a majority, so you rarely want more than five members. Your API server tier, meanwhile, might genuinely benefit from scaling out further under heavy read load.\nExternal decouples the two. Run three etcd members for stable consensus and five API servers for throughput, sized independently. On large clusters, past roughly a hundred nodes where API and etcd load climbs, that independence stops being theoretical.\nResource Behaviour etcd is unusually sensitive to disk latency. It writes every committed entry to disk and expects that write to be fast, so when something else on the box hammers the disk, etcd starts missing heartbeats and can trigger an unnecessary leader election. On a stacked node, that “something else” is your own API server doing a large list operation. The two compete for the same disk.\nExternal sidesteps this by giving etcd dedicated hardware. Fast NVMe for the etcd tier, more RAM for the API tier, each component tuned for what it actually does. You also get to back up, restore, and upgrade etcd without touching the control plane at all.\nThe Trade-offs Side by Side Criterion Stacked etcd External etcd Node count 3 for full HA 6+ (3 etcd + 3 control plane) Setup effort Low, installer defaults High, manual etcd bootstrap Failure isolation Coupled (node loss hits both) Independent Scaling etcd and API scale together Scale each tier separately Disk contention etcd shares disk with API server etcd gets dedicated disk Best fit Homelab, small/medium clusters Large clusters, strict SLAs The table is the quick reference. The honest answer lives in the next section.\nSo Which Should You Pick Reach for stacked when you’re running a homelab or development cluster, when you’re under roughly a hundred nodes with sane workloads, when your team is newer to etcd operations, or when you’re on managed Kubernetes (EKS, GKE, AKS already handle this and the choice isn’t yours to make). The lower node count means less power draw, less cost, and fewer things to understand, which for a self-hoster is the whole point.\nReach for external when you’re past a hundred nodes and the API and etcd load justify dedicated hardware, when your SLAs genuinely can’t tolerate a node loss hitting two systems, when you need to scale the control plane and etcd independently, or when your team already operates etcd clusters comfortably and the added complexity is cheap for you.\nThere’s a middle path worth knowing about too. K3s defaults to embedded etcd (stacked, with a much lighter setup) but also supports an external SQL database (MySQL, PostgreSQL) in place of etcd entirely, which is genuinely handy if you already run and back up one of those.\nSetting Up Stacked HA with kubeadm # On first control plane node kubeadm init \\ --control-plane-endpoint \"loadbalancer.example.com:6443\" \\ --upload-certs # Save the join commands for other control planes # Join additional control plane nodes kubeadm join loadbalancer.example.com:6443 \\ --token \u003ctoken\u003e \\ --discovery-token-ca-cert-hash sha256:\u003chash\u003e \\ --control-plane \\ --certificate-key \u003ccert-key\u003e The load-bearing flag is --control-plane-endpoint. It should point at a load balancer in front of your API servers, never a single node IP, otherwise you’ve quietly reintroduced the single point of failure you were trying to remove.\nSetting Up External etcd For external etcd you bring up the etcd cluster first:\n# On each etcd node, configure etcd cat \u003e /etc/etcd/etcd.conf.yaml \u003c\u003c EOF name: etcd-1 data-dir: /var/lib/etcd initial-cluster-state: new initial-cluster: etcd-1=https://10.0.0.1:2380,etcd-2=https://10.0.0.2:2380,etcd-3=https://10.0.0.3:2380 listen-peer-urls: https://10.0.0.1:2380 listen-client-urls: https://10.0.0.1:2379,https://127.0.0.1:2379 advertise-client-urls: https://10.0.0.1:2379 initial-advertise-peer-urls: https://10.0.0.1:2380 client-transport-security: cert-file: /etc/etcd/pki/server.crt key-file: /etc/etcd/pki/server.key trusted-ca-file: /etc/etcd/pki/ca.crt peer-transport-security: cert-file: /etc/etcd/pki/peer.crt key-file: /etc/etcd/pki/peer.key trusted-ca-file: /etc/etcd/pki/ca.crt EOF Then point kubeadm at that external cluster:\n# kubeadm-config.yaml apiVersion: kubeadm.k8s.io/v1beta3 kind: ClusterConfiguration etcd: external: endpoints: - https://10.0.0.1:2379 - https://10.0.0.2:2379 - https://10.0.0.3:2379 caFile: /etc/kubernetes/pki/etcd/ca.crt certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key What I Actually Run For me, stacked wins, and it isn’t close. I run stacked etcd in my homelab on three control plane nodes. It has survived disk failures, network blips, and the occasional “oops, I rebooted the wrong server” without losing the cluster. Stacked doesn’t mean fragile. It means coupled, and coupling three nodes I understand fully beats spreading state across six I half-understand.\nBut tie that back to my bias: I’m a self-hoster optimising for understanding and low node count, not a platform team running a thousand-node fleet under a five-nines SLA. If that’s your world, external etcd’s isolation is worth every extra node. Your context decides, not mine.\nThe thing that matters more than the topology is the node count underneath it. One node is not HA, it’s a single point of failure with extra steps. Two nodes is worse than one, because quorum needs a majority and 2/2 - 1 = 1, so you can’t actually tolerate any failure at all. Three is the floor for real HA, with either topology.\nKubernetes HA is about surviving failures, not preventing them. Disks die, networks flap, and someone reboots the wrong box eventually. The only question worth asking is whether the cluster shrugs and keeps going, and three control plane nodes with healthy etcd quorum is what lets it.","permalink":"https://tommeurs.nl/en/posts/kubernetes-high-availability-etcd/","title":"Kubernetes High Availability: Stacked vs External etcd Explained"},{"content":"I wrote most of what is here the week the problem turned up, so the archive runs in date order, and date order helps nobody who is starting out.\nThree paths below. Each one begins at nothing and ends at something running. Pick the one that sounds like your week and ignore the other two.\nI want off Windows Ends with Linux as your everyday desktop, and a setup that survives the next reinstall.\nLinux desktop, 5 posts. Why Linux on the machine you use all day, the defaults before the customisation, and the distro question last. Four of the five assume a terminal and nothing else. Arch against NixOS is last on purpose, and it is the one you will want to skip ahead to.\nLinux Is for Everyone: Why This Blog Now Welcomes Beginners Too Learn the Defaults: Why Portability Beats Customization Terminal Multiplexing: tmux vs Zellij vs Screen Dotfile Management with mise and chezmoi: The Perfect Combo Arch vs NixOS as a Workstation: Professional and Personal Use I want to run my own services Ends with a cluster on your own shelf and a backup you have restored from at least once.\nHomelab from scratch, 6 posts. Why you would run your own, what to buy, how to reach it from outside, what to run on it, and how to get it back when it dies. Post 2 is where this gets expensive. Hardware is the part nobody prices before buying it, so it comes before the network and before the hypervisor.\nSelf-Hosting for Everyone: A Vision for Digital Agency The Hardware Ownership Paradox: When Sovereignty Gets Expensive Tailscale for Homelab: Secure Remote Access Without Port Forwarding NixOS as a Hypervisor: KVM and QEMU Can Do Everything K3s Cluster Setup on Refurbished Mini-PCs Backup Strategy for Your Homelab: The 3-2-1 Rule in Practice I have to run Kubernetes at work Ends with knowing which part falls over first, and what that looks like at 2am.\nKubernetes in production, 9 posts. From what a container actually is to running a cluster with no internet, with the failure modes that bite in between. The longest path here, and post 2 argues you may not need a cluster at all. Leaving at that point with that answer counts as finishing.\nSimulation, Emulation, Virtualization, and Containers: The Restaurant Metaphor When not to use Kubernetes Data-Driven Kubernetes Migration: Why You Need Metrics Before You Move Kubernetes High Availability: Stacked vs External etcd Explained How etcd Actually Works: The Heart of Your Kubernetes Cluster Graceful Degradation in Kubernetes: What Happens When Components Fail Unbreakable - My Fascination. Chaos Engineering: Breaking Your Cluster to Make It Stronger Running Kubernetes Offline: Edge Computing Without Internet If you want the why first Why I self-host everything and Working with an AuDHD brain are the reasoning the three paths rest on. Neither is a path of its own, so read them when you want to know why the choices above look the way they do.\nIf none of those fit Series holds four more reading paths, for when you already know what you came for. Categories is the subject index. Beginner collects every post that assumes a terminal and nothing else, including the ones that sit outside a path. Everything here is something I run or have run. Where it cost me a weekend, the post says so.","permalink":"https://tommeurs.nl/en/start-here/","title":"Start here"}]