What happens when your Kubernetes cluster can’t reach the internet? I don’t mean a slow connection. I mean no connection at all. Ships at sea. Remote mining sites. Factory floors with air-gapped networks. Military deployments.
For a lot of people that sounds exotic, like a problem someone else has. I treat it as a baseline design requirement, and I’ll explain why it makes my homelab better even though I almost never actually pull the cable.
Why This Matters: Beyond the Technical
Running Kubernetes offline forces you to answer a question most cloud-native guides skip right past: what are you actually depending on?
A standard Kubernetes setup has invisible dependencies everywhere:
- Container registries (Docker Hub, gcr.io, quay.io)
- Helm chart repositories
- Certificate authorities
- NTP servers
- DNS resolvers
- Package repositories
- Telemetry endpoints
None of these are optional extras. They’re load-bearing assumptions baked into how most clusters operate, and you usually only notice them the day one goes away. Your cluster “works” until Docker Hub rate-limits you or your DNS forwarder falls over, and then it very much doesn’t.
This is a sovereignty problem at heart. If you can’t run your infrastructure without phoning home to systems you don’t control, you don’t really own it. You’re renting capability and hoping the lease holds. Black boxes feel like splinters, and a dependency you can’t see is the worst kind of black box.
Island Mode: The Architecture
I design all my infrastructure to be “island mode capable”, able to function completely cut off from any external network. The honest reason isn’t that I expect to end up on a submarine. It’s that the constraint forces better architecture. The moment you decide the cluster has to survive with the cable unplugged, every sloppy external dependency turns into a thing you have to deal with on purpose.
flowchart TD
subgraph island["Island Mode Cluster"]
subgraph internal["Internal Services"]
Registry["Registry<br/>(Harbor)"]
NTP["NTP<br/>(chrony)"]
DNS["DNS<br/>(CoreDNS)"]
CA["CA<br/>(cert-mgr)"]
Helm["Helm<br/>(ChartMuseum)"]
GitOps["GitOps<br/>(ArgoCD)"]
end
internal -->|"All dependencies resolved locally"| workloads
subgraph workloads["Workloads"]
Apps["Apps run without any external calls"]
end
end
island x--x|"No connection required"| Internet["Internet<br/>(optional)"]
The goal is simple to state and annoying to achieve: pull the network cable and everything keeps running. Updates stop. Operations continue. Let me build it up component by component, because each one is its own small dependency you have to bring in-house.
Component 1: Local Container Registry
Start with the most obvious one. Your cluster needs images. Connected, it pulls from Docker Hub or gcr.io on demand and you never think about it. Offline, every one of those requests fails, and a node that can’t pull an image is a node that can’t schedule a thing.
Fix: mirror everything locally.
Harbor is what I run for self-hosted registries. It gives you:
- Pull-through caching (automatic mirroring when connected)
- Replication from upstream registries
- Vulnerability scanning
- Access control
# Harbor replication policy - sync when connected
apiVersion: goharbor.io/v1beta1
kind: HarborReplicationPolicy
metadata:
name: docker-hub-mirror
spec:
srcRegistry:
url: https://registry-1.docker.io
destRegistry:
url: https://harbor.internal
trigger:
type: scheduled
scheduledTrigger:
cron: "0 2 * * *" # Sync nightly when connected
filters:
- type: name
value: "library/**" # Official images only
Connected, Harbor syncs images on a schedule. Disconnected, it serves whatever it already cached. That’s the whole pattern repeated for every component below: be a cache when you can, be the source of truth when you have to.
The part people forget: pre-pull your dependencies.
A pull-through cache only helps if the image went through it at least once. Before going offline, make sure every image your workloads need is actually sitting in the local registry:
# List all images currently running
kubectl get pods -A -o jsonpath="{.items[*].spec.containers[*].image}" | \
tr ' ' '\n' | sort | uniq
# Ensure they're all mirrored to local registry
Component 2: Local DNS
Now layer on the next dependency. Kubernetes leans on DNS for service discovery, and CoreDNS handles the cluster-internal names fine. The trouble is everything outside the cluster.
Problem: any pod trying to resolve api.github.com or registry-1.docker.io falls flat without an upstream resolver, and some of your own tooling will quietly assume those names work.
Fix: teach CoreDNS to answer the external names you care about, pointing them at internal addresses:
apiVersion: v1
kind: ConfigMap
metadata:
name: coredns
namespace: kube-system
data:
Corefile: |
.:53 {
errors
health
# Cluster-internal names
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
}
# Known external services - resolve to internal
hosts /etc/coredns/custom.hosts {
fallthrough
}
# Forward unknown to local resolver (or fail gracefully)
forward . 10.0.0.1 {
policy sequential
health_check 5s
}
cache 30
loop
reload
loadbalance
}
custom.hosts: |
10.0.10.5 harbor.internal registry.internal
10.0.10.6 git.internal gitlab.internal
10.0.10.7 ntp.internal time.internal
Component 3: Local Time Synchronization
This one bites people who never saw it coming. Time sync matters far more than it looks. TLS certificates validate against the clock, so a drifted node starts rejecting perfectly good certs. Logs need timestamps you can trust when you’re correlating an incident. Distributed systems need the nodes to agree on what “now” is.
Cut off from public NTP, you need a time source of your own:
# Chrony as local NTP server
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: chrony
namespace: kube-system
spec:
selector:
matchLabels:
app: chrony
template:
spec:
hostNetwork: true
containers:
- name: chrony
image: harbor.internal/infra/chrony:4.3
volumeMounts:
- name: chrony-conf
mountPath: /etc/chrony
volumes:
- name: chrony-conf
configMap:
name: chrony-config
---
apiVersion: v1
kind: ConfigMap
metadata:
name: chrony-config
data:
chrony.conf: |
# When connected, use public NTP
server time.cloudflare.com iburst
server pool.ntp.org iburst
# When disconnected, be the time source
local stratum 10
allow 10.0.0.0/8
# Drift file for accuracy when disconnected
driftfile /var/lib/chrony/drift
The line that does the work is local stratum 10. It tells chrony to act as a time source even when there’s no upstream to follow. The nodes sync to each other and stay consistent relative to one another, which is what TLS and your logs actually care about.
Component 4: Local Certificate Authority
TLS everywhere means certificates everywhere, and certificates have to come from somewhere. Connected, cert-manager chats with Let’s Encrypt and you forget the problem exists. Offline, Let’s Encrypt can’t reach you for the ACME challenge and you’re stuck.
Fix: run your own CA.
# Self-signed root CA
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: internal-ca
spec:
ca:
secretName: internal-ca-root
---
# Create the root certificate
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: internal-ca-root
namespace: cert-manager
spec:
isCA: true
commonName: Internal Root CA
secretName: internal-ca-root
duration: 87600h # 10 years
privateKey:
algorithm: ECDSA
size: 256
issuerRef:
name: selfsigned
kind: ClusterIssuer
Push the root CA out to every node and client and trust it. From then on cert-manager mints certificates from your own authority with zero external calls. You trade the convenience of public trust for full control over your trust chain, which is the right trade when the public CA can’t reach you anyway.
Component 5: GitOps Without Internet
If you do GitOps, ArgoCD is probably syncing from GitHub or GitLab.com, and offline that source vanishes. You need the Git server inside the island too.
Option 1: Embedded GitLab
Run GitLab in-cluster. It’s heavy on resources, but you get everything in one box.
Option 2: Gitea
Lightweight Git server, and a much better fit for edge hardware:
apiVersion: apps/v1
kind: Deployment
metadata:
name: gitea
spec:
template:
spec:
containers:
- name: gitea
image: harbor.internal/gitea/gitea:1.21
env:
- name: GITEA__server__OFFLINE_MODE
value: "true"
Point ArgoCD at git.internal instead of github.com. When you have a connection you push your changes upstream as usual, but ArgoCD reconciles against the local copy either way. The cluster’s desired state lives on the island, not on someone else’s server.
Component 6: Helm Charts Offline
Last dependency in the chain. Helm charts normally come from remote repositories, so offline you need chart storage that lives with you.
ChartMuseum speaks the Helm repository API:
# Push charts to local museum when connected
helm push mychart-1.0.0.tgz http://chartmuseum.internal
# ArgoCD uses local charts
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: my-app
spec:
source:
repoURL: http://chartmuseum.internal
chart: mychart
targetRevision: 1.0.0
The Pre-Deployment Checklist
Six components in, here’s where it gets real. The failure mode that hurts most is the one you discover the day after you disconnect, when nobody can fix it without a boat or a long drive. So before you cut the cable, verify the island actually holds together. This is the script I run, and I’d rather it scream at me now than have a pod fail to pull an image at sea:
#!/bin/bash
# offline-readiness-check.sh
echo "=== Checking Offline Readiness ==="
# 1. All images available locally
echo "Checking images..."
for image in $(kubectl get pods -A -o jsonpath="{..image}" | tr ' ' '\n' | sort -u); do
if ! curl -s "https://harbor.internal/v2/${image}/manifests/latest" > /dev/null; then
echo "MISSING: $image"
fi
done
# 2. DNS resolves internally
echo "Checking DNS..."
for name in harbor.internal git.internal ntp.internal; do
if ! nslookup $name > /dev/null 2>&1; then
echo "DNS FAIL: $name"
fi
done
# 3. Time is synchronized
echo "Checking NTP..."
chronyc tracking | grep -q "Leap status.*Normal" || echo "TIME SYNC ISSUE"
# 4. Certificates are valid
echo "Checking certificates..."
kubectl get certificates -A -o jsonpath='{range .items[*]}{.metadata.name}: {.status.conditions[0].type}{"\n"}{end}'
# 5. GitOps is synced
echo "Checking ArgoCD..."
argocd app list | grep -v "Synced.*Healthy" && echo "APPS NOT SYNCED"
echo "=== Check Complete ==="
Real-World Scenarios
The pieces above are abstract until you put them in a place where the cable genuinely doesn’t reach. Three settings make the constraints concrete.
Scenario 1: Ship at Sea
A container ship running Kubernetes for cargo management. Satellite connectivity exists, but it’s expensive and it drops out when the weather turns.
Architecture:
- K3s for a low resource footprint
- Harbor with a full mirror of every required image
- Local GitOps with Gitea
- Sync updates while in port
The catch: updates land in batches during port calls, and that might be once every few weeks. The system has to run rock-solid in between, because nobody is pushing a hotfix mid-Atlantic.
Scenario 2: Factory Floor
A manufacturing plant running Kubernetes for automation, where security policy flatly forbids internet connectivity.
Architecture:
- Full Kubernetes with an air-gapped installation
- USB-based updates, signed and verified
- A separate management network for the rare updates that do happen
The catch: updates need someone physically present, so every change has to be right the first time. There’s no rolling back a bad deploy from your laptop at home.
Scenario 3: My Homelab
I run my own infrastructure island-mode capable, and I’m not waiting for a disaster to justify it. The discipline is the point. When the design assumes no internet, the architecture comes out cleaner on its own.
What I actually get out of it:
- No external dependencies, so no external failure modes to inherit
- Faster image pulls from the local registry
- Everything keeps working through ISP outages, which around here happen more than I’d like
- Infrastructure I genuinely own rather than rent
The Trade-offs
I’m not going to pretend this is free. It isn’t.
| Aspect | Connected | Offline |
|---|---|---|
| Updates | Continuous | Batched |
| Vulnerability info | Real-time | Delayed |
| External integrations | Easy | Impossible |
| Operational burden | Lower | Higher |
| Sovereignty | Partial | Complete |
There’s no universally correct column here. The right answer depends on what your setup actually demands, and for a homelab I happily pay the higher operational burden to get the complete sovereignty.
K3s: Built for the Edge
K3s earns a section of its own, because it was built for exactly this kind of place:
- Single binary, ~100MB
- Embedded SQLite (no external etcd needed)
- Works on ARM (Raspberry Pi, edge devices)
- Minimal dependencies
# Air-gapped K3s installation
curl -sfL https://get.k3s.io > install.sh
# On air-gapped machine
INSTALL_K3S_SKIP_DOWNLOAD=true \
INSTALL_K3S_EXEC="--private-registry /etc/rancher/k3s/registries.yaml" \
./install.sh
Point K3s at a local registry and it runs completely offline. For ships, factories, and Raspberry Pi clusters in a cupboard, that combination is hard to beat.
My Recommendation
If you take one thing from all of this, take the first point. The rest follows from it.
Design for offline, run connected. You probably won’t disconnect often, but building as if you might produces a better cluster every single day.
Mirror everything. Container images, Helm charts, Git repos. If it lives outside your network, get a copy inside it.
Test disconnection. Actually unplug the cable and watch what breaks. The first time you do this you’ll find a dependency you swore wasn’t there.
Automate the sync. While you’re connected, refresh the mirrors on a schedule. A manual process is a process that won’t run the week you needed it most.
Document dependencies. Write down exactly what your cluster needs from the outside world, so future-you isn’t reverse-engineering it during an outage.
Island mode was never about paranoia. It’s about knowing your dependencies well enough to live without them, and a cluster you can run offline is one you genuinely understand.
