What happens when your Kubernetes cluster can’t reach the internet? I don’t mean a slow connection. I mean no connection at all. Ships at sea. Remote mining sites. Factory floors with air-gapped networks. Military deployments.

For a lot of people that sounds exotic, like a problem someone else has. I treat it as a baseline design requirement, and I’ll explain why it makes my homelab better even though I almost never actually pull the cable.

Why This Matters: Beyond the Technical

Running Kubernetes offline forces you to answer a question most cloud-native guides skip right past: what are you actually depending on?

A standard Kubernetes setup has invisible dependencies everywhere:

  • Container registries (Docker Hub, gcr.io, quay.io)
  • Helm chart repositories
  • Certificate authorities
  • NTP servers
  • DNS resolvers
  • Package repositories
  • Telemetry endpoints

None of these are optional extras. They’re load-bearing assumptions baked into how most clusters operate, and you usually only notice them the day one goes away. Your cluster “works” until Docker Hub rate-limits you or your DNS forwarder falls over, and then it very much doesn’t.

This is a sovereignty problem at heart. If you can’t run your infrastructure without phoning home to systems you don’t control, you don’t really own it. You’re renting capability and hoping the lease holds. Black boxes feel like splinters, and a dependency you can’t see is the worst kind of black box.

Island Mode: The Architecture

I design all my infrastructure to be “island mode capable”, able to function completely cut off from any external network. The honest reason isn’t that I expect to end up on a submarine. It’s that the constraint forces better architecture. The moment you decide the cluster has to survive with the cable unplugged, every sloppy external dependency turns into a thing you have to deal with on purpose.

flowchart TD
    subgraph island["Island Mode Cluster"]
        subgraph internal["Internal Services"]
            Registry["Registry<br/>(Harbor)"]
            NTP["NTP<br/>(chrony)"]
            DNS["DNS<br/>(CoreDNS)"]
            CA["CA<br/>(cert-mgr)"]
            Helm["Helm<br/>(ChartMuseum)"]
            GitOps["GitOps<br/>(ArgoCD)"]
        end
        internal -->|"All dependencies resolved locally"| workloads
        subgraph workloads["Workloads"]
            Apps["Apps run without any external calls"]
        end
    end
    island x--x|"No connection required"| Internet["Internet<br/>(optional)"]

The goal is simple to state and annoying to achieve: pull the network cable and everything keeps running. Updates stop. Operations continue. Let me build it up component by component, because each one is its own small dependency you have to bring in-house.

Component 1: Local Container Registry

Start with the most obvious one. Your cluster needs images. Connected, it pulls from Docker Hub or gcr.io on demand and you never think about it. Offline, every one of those requests fails, and a node that can’t pull an image is a node that can’t schedule a thing.

Fix: mirror everything locally.

Harbor is what I run for self-hosted registries. It gives you:

  • Pull-through caching (automatic mirroring when connected)
  • Replication from upstream registries
  • Vulnerability scanning
  • Access control
# Harbor replication policy - sync when connected
apiVersion: goharbor.io/v1beta1
kind: HarborReplicationPolicy
metadata:
  name: docker-hub-mirror
spec:
  srcRegistry:
    url: https://registry-1.docker.io
  destRegistry:
    url: https://harbor.internal
  trigger:
    type: scheduled
    scheduledTrigger:
      cron: "0 2 * * *"  # Sync nightly when connected
  filters:
    - type: name
      value: "library/**"  # Official images only

Connected, Harbor syncs images on a schedule. Disconnected, it serves whatever it already cached. That’s the whole pattern repeated for every component below: be a cache when you can, be the source of truth when you have to.

The part people forget: pre-pull your dependencies.

A pull-through cache only helps if the image went through it at least once. Before going offline, make sure every image your workloads need is actually sitting in the local registry:

# List all images currently running
kubectl get pods -A -o jsonpath="{.items[*].spec.containers[*].image}" | \
  tr ' ' '\n' | sort | uniq

# Ensure they're all mirrored to local registry

Component 2: Local DNS

Now layer on the next dependency. Kubernetes leans on DNS for service discovery, and CoreDNS handles the cluster-internal names fine. The trouble is everything outside the cluster.

Problem: any pod trying to resolve api.github.com or registry-1.docker.io falls flat without an upstream resolver, and some of your own tooling will quietly assume those names work.

Fix: teach CoreDNS to answer the external names you care about, pointing them at internal addresses:

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health

        # Cluster-internal names
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }

        # Known external services - resolve to internal
        hosts /etc/coredns/custom.hosts {
          fallthrough
        }

        # Forward unknown to local resolver (or fail gracefully)
        forward . 10.0.0.1 {
          policy sequential
          health_check 5s
        }

        cache 30
        loop
        reload
        loadbalance
    }
  custom.hosts: |
    10.0.10.5  harbor.internal registry.internal
    10.0.10.6  git.internal gitlab.internal
    10.0.10.7  ntp.internal time.internal

Component 3: Local Time Synchronization

This one bites people who never saw it coming. Time sync matters far more than it looks. TLS certificates validate against the clock, so a drifted node starts rejecting perfectly good certs. Logs need timestamps you can trust when you’re correlating an incident. Distributed systems need the nodes to agree on what “now” is.

Cut off from public NTP, you need a time source of your own:

# Chrony as local NTP server
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: chrony
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: chrony
  template:
    spec:
      hostNetwork: true
      containers:
      - name: chrony
        image: harbor.internal/infra/chrony:4.3
        volumeMounts:
        - name: chrony-conf
          mountPath: /etc/chrony
      volumes:
      - name: chrony-conf
        configMap:
          name: chrony-config
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: chrony-config
data:
  chrony.conf: |
    # When connected, use public NTP
    server time.cloudflare.com iburst
    server pool.ntp.org iburst

    # When disconnected, be the time source
    local stratum 10
    allow 10.0.0.0/8

    # Drift file for accuracy when disconnected
    driftfile /var/lib/chrony/drift

The line that does the work is local stratum 10. It tells chrony to act as a time source even when there’s no upstream to follow. The nodes sync to each other and stay consistent relative to one another, which is what TLS and your logs actually care about.

Component 4: Local Certificate Authority

TLS everywhere means certificates everywhere, and certificates have to come from somewhere. Connected, cert-manager chats with Let’s Encrypt and you forget the problem exists. Offline, Let’s Encrypt can’t reach you for the ACME challenge and you’re stuck.

Fix: run your own CA.

# Self-signed root CA
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: internal-ca
spec:
  ca:
    secretName: internal-ca-root

---
# Create the root certificate
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: internal-ca-root
  namespace: cert-manager
spec:
  isCA: true
  commonName: Internal Root CA
  secretName: internal-ca-root
  duration: 87600h  # 10 years
  privateKey:
    algorithm: ECDSA
    size: 256
  issuerRef:
    name: selfsigned
    kind: ClusterIssuer

Push the root CA out to every node and client and trust it. From then on cert-manager mints certificates from your own authority with zero external calls. You trade the convenience of public trust for full control over your trust chain, which is the right trade when the public CA can’t reach you anyway.

Component 5: GitOps Without Internet

If you do GitOps, ArgoCD is probably syncing from GitHub or GitLab.com, and offline that source vanishes. You need the Git server inside the island too.

Option 1: Embedded GitLab

Run GitLab in-cluster. It’s heavy on resources, but you get everything in one box.

Option 2: Gitea

Lightweight Git server, and a much better fit for edge hardware:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: gitea
spec:
  template:
    spec:
      containers:
      - name: gitea
        image: harbor.internal/gitea/gitea:1.21
        env:
        - name: GITEA__server__OFFLINE_MODE
          value: "true"

Point ArgoCD at git.internal instead of github.com. When you have a connection you push your changes upstream as usual, but ArgoCD reconciles against the local copy either way. The cluster’s desired state lives on the island, not on someone else’s server.

Component 6: Helm Charts Offline

Last dependency in the chain. Helm charts normally come from remote repositories, so offline you need chart storage that lives with you.

ChartMuseum speaks the Helm repository API:

# Push charts to local museum when connected
helm push mychart-1.0.0.tgz http://chartmuseum.internal

# ArgoCD uses local charts
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
spec:
  source:
    repoURL: http://chartmuseum.internal
    chart: mychart
    targetRevision: 1.0.0

The Pre-Deployment Checklist

Six components in, here’s where it gets real. The failure mode that hurts most is the one you discover the day after you disconnect, when nobody can fix it without a boat or a long drive. So before you cut the cable, verify the island actually holds together. This is the script I run, and I’d rather it scream at me now than have a pod fail to pull an image at sea:

#!/bin/bash
# offline-readiness-check.sh

echo "=== Checking Offline Readiness ==="

# 1. All images available locally
echo "Checking images..."
for image in $(kubectl get pods -A -o jsonpath="{..image}" | tr ' ' '\n' | sort -u); do
  if ! curl -s "https://harbor.internal/v2/${image}/manifests/latest" > /dev/null; then
    echo "MISSING: $image"
  fi
done

# 2. DNS resolves internally
echo "Checking DNS..."
for name in harbor.internal git.internal ntp.internal; do
  if ! nslookup $name > /dev/null 2>&1; then
    echo "DNS FAIL: $name"
  fi
done

# 3. Time is synchronized
echo "Checking NTP..."
chronyc tracking | grep -q "Leap status.*Normal" || echo "TIME SYNC ISSUE"

# 4. Certificates are valid
echo "Checking certificates..."
kubectl get certificates -A -o jsonpath='{range .items[*]}{.metadata.name}: {.status.conditions[0].type}{"\n"}{end}'

# 5. GitOps is synced
echo "Checking ArgoCD..."
argocd app list | grep -v "Synced.*Healthy" && echo "APPS NOT SYNCED"

echo "=== Check Complete ==="

Real-World Scenarios

The pieces above are abstract until you put them in a place where the cable genuinely doesn’t reach. Three settings make the constraints concrete.

Scenario 1: Ship at Sea

A container ship running Kubernetes for cargo management. Satellite connectivity exists, but it’s expensive and it drops out when the weather turns.

Architecture:

  • K3s for a low resource footprint
  • Harbor with a full mirror of every required image
  • Local GitOps with Gitea
  • Sync updates while in port

The catch: updates land in batches during port calls, and that might be once every few weeks. The system has to run rock-solid in between, because nobody is pushing a hotfix mid-Atlantic.

Scenario 2: Factory Floor

A manufacturing plant running Kubernetes for automation, where security policy flatly forbids internet connectivity.

Architecture:

  • Full Kubernetes with an air-gapped installation
  • USB-based updates, signed and verified
  • A separate management network for the rare updates that do happen

The catch: updates need someone physically present, so every change has to be right the first time. There’s no rolling back a bad deploy from your laptop at home.

Scenario 3: My Homelab

I run my own infrastructure island-mode capable, and I’m not waiting for a disaster to justify it. The discipline is the point. When the design assumes no internet, the architecture comes out cleaner on its own.

What I actually get out of it:

  • No external dependencies, so no external failure modes to inherit
  • Faster image pulls from the local registry
  • Everything keeps working through ISP outages, which around here happen more than I’d like
  • Infrastructure I genuinely own rather than rent

The Trade-offs

I’m not going to pretend this is free. It isn’t.

AspectConnectedOffline
UpdatesContinuousBatched
Vulnerability infoReal-timeDelayed
External integrationsEasyImpossible
Operational burdenLowerHigher
SovereigntyPartialComplete

There’s no universally correct column here. The right answer depends on what your setup actually demands, and for a homelab I happily pay the higher operational burden to get the complete sovereignty.

K3s: Built for the Edge

K3s earns a section of its own, because it was built for exactly this kind of place:

  • Single binary, ~100MB
  • Embedded SQLite (no external etcd needed)
  • Works on ARM (Raspberry Pi, edge devices)
  • Minimal dependencies
# Air-gapped K3s installation
curl -sfL https://get.k3s.io > install.sh

# On air-gapped machine
INSTALL_K3S_SKIP_DOWNLOAD=true \
INSTALL_K3S_EXEC="--private-registry /etc/rancher/k3s/registries.yaml" \
./install.sh

Point K3s at a local registry and it runs completely offline. For ships, factories, and Raspberry Pi clusters in a cupboard, that combination is hard to beat.

My Recommendation

If you take one thing from all of this, take the first point. The rest follows from it.

  1. Design for offline, run connected. You probably won’t disconnect often, but building as if you might produces a better cluster every single day.

  2. Mirror everything. Container images, Helm charts, Git repos. If it lives outside your network, get a copy inside it.

  3. Test disconnection. Actually unplug the cable and watch what breaks. The first time you do this you’ll find a dependency you swore wasn’t there.

  4. Automate the sync. While you’re connected, refresh the mirrors on a schedule. A manual process is a process that won’t run the week you needed it most.

  5. Document dependencies. Write down exactly what your cluster needs from the outside world, so future-you isn’t reverse-engineering it during an outage.


Island mode was never about paranoia. It’s about knowing your dependencies well enough to live without them, and a cluster you can run offline is one you genuinely understand.