K8sGPT with a Local 70B Model on Apple Silicon

TL;DR I pointed K8sGPT at a Llama 3.3 70B model running on a Mac Studio in my own office, so no cluster metadata leaves the network, and fed it three broken workloads. It found all three and wrote fixes I’d accept from a colleague, except for the resource numbers, which it made up. Assisted diagnosis works, at 25 seconds a query. Autonomous cluster management is still marketing, and the security section is why I wouldn’t let a model change anything on its own.

Every vendor pitch deck right now has the same slide. “Autonomous cluster management.” An AI watches your Kubernetes cluster, spots the problem, diagnoses it and fixes it while you sleep. Platform engineers stop firefighting and the cluster heals itself.

I run a homelab because I want to know what’s going on inside it. So when I see a slide like that, I want to run it before I believe it.

The catch with most of these tools is what “AI” means once you read the fine print. It means your cluster metadata gets shipped to OpenAI. Pod names, labels, error messages, the lot. I self-host precisely so that data stays on my network, and an AIOps story that only works once I hand it to a third party has nothing to offer me.

So here’s the version I wanted to try. K8sGPT pointed at a Llama 3.3 70B model running locally on Apple Silicon. No cloud API, no data leaving the building. Is a local model good enough to be useful for cluster diagnosis? Or does sovereignty here cost more than it’s worth? That’s the question this post answers.

One thing before the commands. This is a homelab experiment. I’m reporting what I tested and what I found, and the security section further down is the reason you shouldn’t run any of it in production. Read it that way.

The hardware and the software

The hardware

A Mac Studio M3 Ultra with 512GB of unified memory. The M3 Ultra has 80 GPU cores you can use for inference, and unified memory means nothing gets copied between CPU RAM and GPU RAM.

At around €10,000 it’s an expensive box, and I’m not pretending otherwise. It’s also about the only consumer hardware that runs a 70B model at Q8 quantization at a speed you’d sit and wait for. The unified memory is the whole trick. On a normal PC with a discrete card you pay a copy tax between CPU and GPU RAM on every pass, and that tax is what kills you.

The software stack

Installing vLLM with the Metal backend

vLLM has experimental Metal support for Apple Silicon. The install is a conda environment and one pip:

# Create a dedicated conda environment
conda create -n vllm python=3.11
conda activate vllm

# Install vLLM with Metal support
pip install vllm

# Verify Metal backend
python -c "import vllm; print(vllm.__version__)"

Experimental is the right word. At the time of writing the Metal support in vLLM is rough, and you notice. llama.cpp with its Metal backend is the stable option, but K8sGPT wants an OpenAI-compatible API and vLLM gives me one out of the box. So I took the rougher path rather than write a shim.

Downloading the model

# Download the model (Q8 quantization, ~75GB)
huggingface-cli download meta-llama/Llama-3.3-70B-Instruct \
  --local-dir ./models/llama-3.3-70b-instruct

You need a Hugging Face account and you have to accept the Llama license.

Starting the vLLM server

# Start the inference server
vllm serve ./models/llama-3.3-70b-instruct \
  --host 0.0.0.0 \
  --port 8000 \
  --dtype float16 \
  --max-model-len 8192 \
  --device mps

The --device mps flag forces Metal Performance Shaders. Leave it off and vLLM falls back to the CPU.

Check that it’s up:

curl http://localhost:8000/v1/models

Deploying the K8sGPT operator

The operator goes into the cluster with Helm :

# Add the Helm repo
helm repo add k8sgpt https://charts.k8sgpt.ai/
helm repo update

# Install the operator
helm install k8sgpt-operator k8sgpt/k8sgpt-operator \
  -n k8sgpt-operator-system \
  --create-namespace

Then a K8sGPT resource that points the backend at the vLLM server on the Mac:

# k8sgpt-config.yaml
apiVersion: core.k8sgpt.ai/v1alpha1
kind: K8sGPT
metadata:
  name: k8sgpt-local
  namespace: k8sgpt-operator-system
spec:
  ai:
    enabled: true
    model: llama-3.3-70b-instruct
    backend: localai
    baseUrl: http://192.168.1.100:8000/v1  # IP of your Mac Studio
  noCache: false
  version: v0.3.40
  analyzers:
    - Pod
    - Deployment
    - Service
    - ReplicaSet
    - PersistentVolumeClaim
    - Ingress
    - StatefulSet
    - CronJob

kubectl apply -f k8sgpt-config.yaml

Three broken things

The setup runs. Now the part that matters. Is it useful, or is it a very expensive way to get a wrong answer with confidence? I threw three deliberately broken things at it and graded the output the way I’d grade a junior engineer’s first pass.

Scenario A: a CrashLoopBackOff

A deployment that references a ConfigMap which doesn’t exist:

# broken-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: broken-app
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: broken-app
  template:
    metadata:
      labels:
        app: broken-app
    spec:
      containers:
      - name: app
        image: nginx:1.25
        envFrom:
        - configMapRef:
            name: app-config  # This ConfigMap doesn't exist

kubectl apply -f broken-deployment.yaml

After a minute the pod is in CrashLoopBackOff. The operator writes its analysis to a results resource:

kubectl get results -n k8sgpt-operator-system -o yaml

Paraphrased, the output was this:

Analysis: Pod broken-app-xxxx is in CreateContainerConfigError state.
The pod is referencing a ConfigMap 'app-config' that does not exist
in the namespace.

Suggested remediation:
1. Create the missing ConfigMap:
   kubectl create configmap app-config --from-literal=KEY=value
2. Or remove the configMapRef from the deployment spec
3. Verify the ConfigMap name spelling matches exactly

Usable. The diagnosis is right and the suggestions are practical. A junior engineer could work from this and not get led astray. What it missed is the one check I’d have done first, whether the ConfigMap lives in another namespace. That’s the mistake I make about once a month.

Scenario B: no resource limits

A deployment with three replicas and no resources block at all:

# no-limits.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: no-limits-app
  namespace: default
spec:
  replicas: 3
  selector:
    matchLabels:
      app: no-limits
  template:
    metadata:
      labels:
        app: no-limits
    spec:
      containers:
      - name: app
        image: nginx:1.25
        # No resources defined

K8sGPT’s analysis:

Analysis: Deployment no-limits-app has containers without resource
requests or limits defined.

Issues identified:
- Container 'app' has no CPU requests/limits
- Container 'app' has no memory requests/limits

Suggested remediation:
Add resource specifications to ensure predictable scheduling and
prevent resource starvation:

resources:
  requests:
    memory: "64Mi"
    cpu: "100m"
  limits:
    memory: "128Mi"
    cpu: "200m"

Partially usable. It spots the problem. The numbers it hands you are made up. The model never saw a metric, so nothing in those requests and limits is based on what this workload does. An experienced engineer pulls real data first, kubectl top pods or Prometheus, and sizes from there. The LLM guesses, and nothing in the output says so.

It also doesn’t mention the Vertical Pod Autoscaler, doesn’t warn that limits without requests can bite you, and picks the 1:2 ratio between requests and limits out of thin air.

Scenario C: a privileged pod

A pod that does most of the things a security policy exists to stop:

# insecure-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: insecure-pod
  namespace: default
spec:
  hostNetwork: true
  containers:
  - name: app
    image: nginx:1.25
    securityContext:
      privileged: true
      runAsRoot: true
    volumeMounts:
    - name: host-root
      mountPath: /host
  volumes:
  - name: host-root
    hostPath:
      path: /

The analysis, with the Trivy integration switched on:

Analysis: Pod insecure-pod has multiple security issues:

Critical:
- Container running in privileged mode (privileged: true)
- Container has access to host network (hostNetwork: true)
- Container mounts host root filesystem

High:
- No securityContext.readOnlyRootFilesystem
- No securityContext.allowPrivilegeEscalation: false

Suggested remediation:
1. Remove privileged: true unless absolutely necessary
2. Use a dedicated network namespace (remove hostNetwork: true)
3. Avoid mounting host paths, especially root filesystem
4. Add security hardening:

   securityContext:
     privileged: false
     readOnlyRootFilesystem: true
     allowPrivilegeEscalation: false
     runAsNonRoot: true
     capabilities:
       drop:
         - ALL

Usable. This one surprised me. The issues are all there and the remediation reads like what a security engineer would tell you. Trivy earns its place here by flagging the image vulnerabilities as well, so you get the misconfiguration and the CVE story in one pass.

What’s missing is the policy layer. No mention of Pod Security Standards and the restricted profile, and no word that a Kyverno or OPA policy should have blocked this pod before it ever scheduled.

Performance

Inference speed

End-to-end latency

From the moment a problem exists to the moment the report lands:

Resource usage

During inference:

Against the OpenAI API

The OpenAI API is faster and the output is a bit better. The price is that your cluster metadata leaves your network. For me that trade is the whole point of the exercise. I’ll wait 25 seconds instead of 5 to keep the data at home, and I don’t find that a hard call. Whether you make the same one depends on what’s in your pod labels.

Air-gapped

Can it run with no internet at all? Yes, with some preparation. This is where the local model pays for itself, because the air-gapped case is off the table the moment you depend on a cloud API.

What to download beforehand

# 1. Model weights (~75GB)
huggingface-cli download meta-llama/Llama-3.3-70B-Instruct \
  --local-dir ./airgap-bundle/models/

# 2. vLLM Python packages
pip download vllm -d ./airgap-bundle/packages/

# 3. K8sGPT container images
docker pull ghcr.io/k8sgpt-ai/k8sgpt-operator:latest
docker save ghcr.io/k8sgpt-ai/k8sgpt-operator:latest > ./airgap-bundle/images/k8sgpt-operator.tar

# 4. Helm charts
helm pull k8sgpt/k8sgpt-operator --destination ./airgap-bundle/charts/

Transport and install

# On the air-gapped machine:

# 1. Install Python packages offline
pip install --no-index --find-links=./airgap-bundle/packages/ vllm

# 2. Load container images
docker load < ./airgap-bundle/images/k8sgpt-operator.tar
# Or push to your local registry

# 3. Install Helm chart
helm install k8sgpt-operator ./airgap-bundle/charts/k8sgpt-operator-*.tgz \
  --set image.repository=your-local-registry/k8sgpt-operator

What works offline

The Trivy vulnerability database is the exception. It needs periodic updates, and you’ll have to transport those separately. Without a recent database, K8sGPT misses new CVEs.

Security and the threat model

This is where I have to be honest. A local model gives you data sovereignty and nothing else on the security side. The setup is as secure as the box it runs on, and a Mac Studio on your network as an inference server brings its own pile of problems. The LLM brings a second pile that’s specific to it.

Platform issues

A Mac Studio as an inference server has limits you can’t configure away:

So for anything with real compliance requirements, ISO27001 Annex A, NIS2 , SOC2, a Mac Studio is the wrong machine. For a homelab or a development environment it’s fine.

Risks that come with the model

The threat model

What I’d do with it

Usable, under conditions

Yes, a local model is usable for Kubernetes diagnosis, under conditions. It identifies standard issues correctly, gives remediation you can act on, catches security problems, and does all of that without a byte leaving the network.

What it can’t do matters as much. It can’t debug a problem that spans several components. It can’t do auto-remediation you’d trust. It doesn’t know your setup, and it can’t guarantee it’s right.

Per use case

Homelab and learning

Go for it. You’ll learn about LLM inference infrastructure, about Kubernetes troubleshooting patterns and about where AI-assisted operations stop being useful, and the blast radius is your own cluster.

Development and staging

Usable with guardrails. Someone reviews the output before anything gets applied, every LLM interaction is logged, and there’s no auto-remediation. Diagnosis only.

Production, connected

Use a cloud API. The models are better, GPT-4 or Claude, the latency is lower, there’s no hardware to buy and someone else signs the SLA . For most organisations the privacy trade-off is acceptable, as long as there’s no PII in the cluster metadata.

Production, air-gapped or sovereign

Only as a last resort. If the data truly can’t leave, look at smaller, dedicated models first and put defence in depth around the inference server. Treat every line the model produces as untrusted, log and audit everything, and keep it an assistant rather than an authority.

Back to the slide

Autonomous cluster management with LLMs is marketing today. What I tested is assisted diagnosis, a model that points at problems and offers suggestions, with me reading every one of them. The human in the loop is a requirement, and it doesn’t come off once you trust the model enough.

And I’m impressed anyway. A 70B model on hardware in my own office produced analyses I’d accept from a colleague. The gap between good enough to read and good enough to act on a production cluster without me is still wide. Scenario B shows how wide. The model invented resource limits and presented them as advice.

I got the answer I came for. A local model is good enough to be a useful diagnostic assistant, and running it here means I never had to choose between AIOps and the reasons in Sovereign Infrastructure. Why Privacy Matters is the other half of why the data stays home. So it stays an assistant. I read every suggestion before I type it, the way I’d read one from a sharp colleague who’s never seen my cluster.

• AI
• Sovereignty