What happens when your Kubernetes cluster can’t reach the internet? Not “slow connection” — no connection at all. Ships at sea. Remote mining sites. Factory floors with air-gapped networks. Military deployments.
This isn’t an edge case. It’s a design requirement for anyone who takes sovereignty seriously.
Why This Matters: Beyond the Technical
Running Kubernetes offline forces you to confront a question most cloud-native guides ignore: what are you actually depending on?
A standard Kubernetes setup has invisible dependencies everywhere:
- Container registries (Docker Hub, gcr.io, quay.io)
- Helm chart repositories
- Certificate authorities
- NTP servers
- DNS resolvers
- Package repositories
- Telemetry endpoints
These aren’t optional extras. They’re load-bearing assumptions baked into how most clusters operate. Your cluster “works” until any of these disappear.
This matters for sovereignty. If you can’t run your infrastructure without external dependencies, you don’t truly own it. You’re renting capability from systems you don’t control.
Island Mode: The Architecture
I design all my infrastructure to be “island mode capable” — able to function completely disconnected from any external network. Even if I never actually run offline, this constraint forces better architecture:
flowchart TD
subgraph island["Island Mode Cluster"]
subgraph internal["Internal Services"]
Registry["Registry<br/>(Harbor)"]
NTP["NTP<br/>(chrony)"]
DNS["DNS<br/>(CoreDNS)"]
CA["CA<br/>(cert-mgr)"]
Helm["Helm<br/>(ChartMuseum)"]
GitOps["GitOps<br/>(ArgoCD)"]
end
internal -->|"All dependencies resolved locally"| workloads
subgraph workloads["Workloads"]
Apps["Apps run without any external calls"]
end
end
island x--x|"No connection required"| Internet["Internet<br/>(optional)"]
The goal: if you unplug the network cable, everything keeps running. Updates stop, but operations continue.
Component 1: Local Container Registry
Your cluster needs images. In a connected environment, it pulls from Docker Hub or gcr.io on demand. Offline, those requests fail.
Solution: Mirror everything locally.
Harbor is my choice for self-hosted registries. It supports:
- Pull-through caching (automatic mirroring when connected)
- Replication from upstream registries
- Vulnerability scanning
- Access control
# Harbor replication policy - sync when connected
apiVersion: goharbor.io/v1beta1
kind: HarborReplicationPolicy
metadata:
name: docker-hub-mirror
spec:
srcRegistry:
url: https://registry-1.docker.io
destRegistry:
url: https://harbor.internal
trigger:
type: scheduled
scheduledTrigger:
cron: "0 2 * * *" # Sync nightly when connected
filters:
- type: name
value: "library/**" # Official images only
When connected, Harbor syncs images. When disconnected, it serves from cache.
Critical: Pre-pull your dependencies.
Before going offline, ensure all images your workloads need are in the local registry:
# List all images currently running
kubectl get pods -A -o jsonpath="{.items[*].spec.containers[*].image}" | \
tr ' ' '\n' | sort | uniq
# Ensure they're all mirrored to local registry
Component 2: Local DNS
Kubernetes uses DNS for service discovery. CoreDNS handles cluster-internal names, but what about external names?
Problem: Pods trying to resolve api.github.com or registry-1.docker.io will fail without upstream DNS.
Solution: Configure CoreDNS to handle known external names internally:
apiVersion: v1
kind: ConfigMap
metadata:
name: coredns
namespace: kube-system
data:
Corefile: |
.:53 {
errors
health
# Cluster-internal names
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
}
# Known external services - resolve to internal
hosts /etc/coredns/custom.hosts {
fallthrough
}
# Forward unknown to local resolver (or fail gracefully)
forward . 10.0.0.1 {
policy sequential
health_check 5s
}
cache 30
loop
reload
loadbalance
}
custom.hosts: |
10.0.10.5 harbor.internal registry.internal
10.0.10.6 git.internal gitlab.internal
10.0.10.7 ntp.internal time.internal
Component 3: Local Time Synchronization
Time synchronization matters more than you think. TLS certificates validate against time. Logs need accurate timestamps. Distributed systems need clock agreement.
Without internet NTP servers, you need a local time source:
# Chrony as local NTP server
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: chrony
namespace: kube-system
spec:
selector:
matchLabels:
app: chrony
template:
spec:
hostNetwork: true
containers:
- name: chrony
image: harbor.internal/infra/chrony:4.3
volumeMounts:
- name: chrony-conf
mountPath: /etc/chrony
volumes:
- name: chrony-conf
configMap:
name: chrony-config
---
apiVersion: v1
kind: ConfigMap
metadata:
name: chrony-config
data:
chrony.conf: |
# When connected, use public NTP
server time.cloudflare.com iburst
server pool.ntp.org iburst
# When disconnected, be the time source
local stratum 10
allow 10.0.0.0/8
# Drift file for accuracy when disconnected
driftfile /var/lib/chrony/drift
The key: local stratum 10 makes the server act as a time source even without upstream. Nodes sync to each other, maintaining relative consistency.
Component 4: Local Certificate Authority
TLS everywhere means certificates everywhere. In connected environments, cert-manager talks to Let’s Encrypt. Offline, that’s impossible.
Solution: Run your own CA.
# Self-signed root CA
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: internal-ca
spec:
ca:
secretName: internal-ca-root
---
# Create the root certificate
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: internal-ca-root
namespace: cert-manager
spec:
isCA: true
commonName: Internal Root CA
secretName: internal-ca-root
duration: 87600h # 10 years
privateKey:
algorithm: ECDSA
size: 256
issuerRef:
name: selfsigned
kind: ClusterIssuer
Distribute the root CA to all nodes and clients. Now cert-manager issues certificates from your internal CA without any external calls.
Component 5: GitOps Without Internet
ArgoCD typically syncs from GitHub or GitLab. Offline, you need a local Git server.
Option 1: Embedded GitLab
Run GitLab in-cluster. Heavy, but full-featured.
Option 2: Gitea
Lightweight Git server, perfect for edge:
apiVersion: apps/v1
kind: Deployment
metadata:
name: gitea
spec:
template:
spec:
containers:
- name: gitea
image: harbor.internal/gitea/gitea:1.21
env:
- name: GITEA__server__OFFLINE_MODE
value: "true"
ArgoCD points to git.internal instead of github.com. When connected, you push changes. ArgoCD syncs locally whether connected or not.
Component 6: Helm Charts Offline
Helm charts typically pull from remote repositories. Offline, you need local chart storage.
ChartMuseum provides a Helm repository API:
# Push charts to local museum when connected
helm push mychart-1.0.0.tgz http://chartmuseum.internal
# ArgoCD uses local charts
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: my-app
spec:
source:
repoURL: http://chartmuseum.internal
chart: mychart
targetRevision: 1.0.0
The Pre-Deployment Checklist
Before disconnecting, verify:
#!/bin/bash
# offline-readiness-check.sh
echo "=== Checking Offline Readiness ==="
# 1. All images available locally
echo "Checking images..."
for image in $(kubectl get pods -A -o jsonpath="{..image}" | tr ' ' '\n' | sort -u); do
if ! curl -s "https://harbor.internal/v2/${image}/manifests/latest" > /dev/null; then
echo "MISSING: $image"
fi
done
# 2. DNS resolves internally
echo "Checking DNS..."
for name in harbor.internal git.internal ntp.internal; do
if ! nslookup $name > /dev/null 2>&1; then
echo "DNS FAIL: $name"
fi
done
# 3. Time is synchronized
echo "Checking NTP..."
chronyc tracking | grep -q "Leap status.*Normal" || echo "TIME SYNC ISSUE"
# 4. Certificates are valid
echo "Checking certificates..."
kubectl get certificates -A -o jsonpath='{range .items[*]}{.metadata.name}: {.status.conditions[0].type}{"\n"}{end}'
# 5. GitOps is synced
echo "Checking ArgoCD..."
argocd app list | grep -v "Synced.*Healthy" && echo "APPS NOT SYNCED"
echo "=== Check Complete ==="
Real-World Scenarios
Scenario 1: Ship at Sea
A container ship runs Kubernetes for cargo management. Satellite connectivity is expensive and unreliable.
Architecture:
- K3s for low resource footprint
- Harbor with full mirror of required images
- Local GitOps with Gitea
- Sync updates when in port
The catch: Updates happen in batches during port calls. The system must run stable for weeks between syncs.
Scenario 2: Factory Floor
A manufacturing plant runs Kubernetes for automation. Security policy prohibits internet connectivity.
Architecture:
- Full Kubernetes with air-gapped installation
- USB-based updates (signed and verified)
- Separate management network for rare updates
The catch: Updates require physical presence. Every change must be bulletproof.
Scenario 3: My Homelab
I run my infrastructure to be island-mode capable. Not because I expect disconnection, but because it forces good design.
Benefits:
- No external dependencies = no external failure modes
- Faster pulls from local registry
- Works during ISP outages
- Truly sovereign infrastructure
The Trade-offs
Offline Kubernetes isn’t free:
| Aspect | Connected | Offline |
|---|---|---|
| Updates | Continuous | Batched |
| Vulnerability info | Real-time | Delayed |
| External integrations | Easy | Impossible |
| Operational burden | Lower | Higher |
| Sovereignty | Partial | Complete |
The question isn’t “which is better” — it’s “what do you need?”
K3s: Built for the Edge
K3s deserves special mention. It’s designed for exactly this use case:
- Single binary, ~100MB
- Embedded SQLite (no external etcd needed)
- Works on ARM (Raspberry Pi, edge devices)
- Minimal dependencies
# Air-gapped K3s installation
curl -sfL https://get.k3s.io > install.sh
# On air-gapped machine
INSTALL_K3S_SKIP_DOWNLOAD=true \
INSTALL_K3S_EXEC="--private-registry /etc/rancher/k3s/registries.yaml" \
./install.sh
With a local registry configured, K3s runs completely offline.
My Recommendation
Design for offline, run connected. Even if you never disconnect, the architecture is better.
Mirror everything. Container images, Helm charts, Git repos. If it’s external, mirror it.
Test disconnection. Actually unplug the network and see what breaks. You’ll be surprised.
Automate the sync. When connected, automatically update mirrors. Don’t rely on manual processes.
Document dependencies. Know exactly what your cluster needs from the outside world.
Island mode isn’t about paranoia. It’s about understanding your dependencies. A system you can run offline is a system you truly understand.
